Record · corpus.blog/posts/01a0fdc8-5107-735a-b197-ff3be080f992

Inside the Matrix: Visualizing Matrix Multiplication, Attention and Beyond

PyTorch Blog · published 25 September 2023

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at pytorch.org Wayback capture

2blogs citing

First cited: 30 Jun 2024
Most recent: 4 Jul 2024
Rank this month: not ranked
External links: 27
Words captured: 6,460

Cited by

Blog

In the post

Date

jackyoustra.com

Some tools I found useful Read at the source ↗

4 Jul 2024

alessiodevoto.github.io

Efficiency Metrics in Machine Learning Read at the source ↗

30 Jun 2024

Links from this post

Link

Host

https://bhosmer.github.io/mm/ref.html

bhosmer.github.io

https://bhosmer.github.io/mm/

bhosmer.github.io

https://github.com/karpathy/nanoGPT

github.com

https://arxiv.org/pdf/2305.19370.pdf

arxiv.org

https://bhosmer.github.io/mm/index.html?params=%7B%22expr%22%3A%22L%20%40%20R%22%2C%22name%22%3A%22L%20%40%20R%22%2C%22epilog%22%3A%22none%22%2C%22left%22%3A%7B%22name%22%3A%22L%22%2C%22matmul%22%3Afalse%2C%22h%22%3A32%2C%22w%22%3A24%2C%22init%22%3A%22expr%22%2C%22url%22%3A%22%22%2C%22min%22%3A-1%2C%22max%22%3A1%2C%22dropout%22%3A0%2C%22expr%22%3A%22-2%20*%20(Math.trunc(i%20%2F%208

bhosmer.github.io

https://bhosmer.github.io/mm/index.html?params=%7B%22expr%22%3A%22L%20%40%20R%22%2C%22name%22%3A%22L%20%40%20R%22%2C%22epilog%22%3A%22none%22%2C%22left%22%3A%7B%22name%22%3A%22L%22%2C%22matmul%22%3Afalse%2C%22h%22%3A32%2C%22w%22%3A24%2C%22init%22%3A%22gaussian%22%2C%22url%22%3A%22%22%2C%22min%22%3A-1%2C%22max%22%3A1%2C%22dropout%22%3A0%2C%22expr%22%3A%22(-Math.trunc(i%20%2F%208

bhosmer.github.io

https://bhosmer.github.io/mm/index.html?params=%7B%22expr%22%3A%22L%20%40%20R%22%2C%22name%22%3A%22L%20%40%20R%22%2C%22epilog%22%3A%22none%22%2C%22left%22%3A%7B%22name%22%3A%22L%22%2C%22matmul%22%3Afalse%2C%22h%22%3A24%2C%22w%22%3A24%2C%22init%22%3A%22eye%22%2C%22url%22%3A%22%22%2C%22min%22%3A-1%2C%22max%22%3A1%2C%22dropout%22%3A0%2C%22expr%22%3A%22(-Math.trunc(i%20%2F%208

bhosmer.github.io

https://arxiv.org/pdf/2106.09685.pdf

arxiv.org

https://bhosmer.github.io/mm/index.html?0=A%20%40%20B%20%40%20C&1=A%20%40%20B%20%40%20C&2=none&12=closed&64=true&3.1=A%20%40%20B&3.4=true&3.5=32&3.6=32&3.7=row%20major&3.8=&3.9=-1&3.10=1&3.11=0&3.12=open&3.2=none&13.1=A&13.4=false&13.5=64&13.6=32&13.7=expr&13.8=&13.0=-2%20*%20(Math.trunc(i%20%2F%208

bhosmer.github.io

https://bhosmer.github.io/mm/index.html?0=A%20%40%20B%20%40%20C&1=A%20%40%20B%20%40%20C&2=none&12=closed&64=true&3.1=A%20%40%20B&3.4=true&3.5=32&3.6=32&3.7=row%20major&3.8=&3.9=-1&3.10=1&3.11=0&3.12=open&3.2=none&13.1=A&13.4=false&13.5=64&13.6=96&13.7=expr&13.8=&13.0=-2%20*%20(Math.trunc(i%20%2F%208

bhosmer.github.io

https://bhosmer.github.io/mm/index.html?0=A%20%40%20B%20%40%20C%20%40%20D%20%40%20E&1=A%20%40%20B%20%40%20C%20%40%20D%20%40%20E&2=none&23=closed&63=true&3.2=none&4.5=inherit&6.7=1&8.9=positive&8.10=left&8.11=bottom&8.12=back&13.0=A%20%40%20B%20%40%20C%20%40%20D%20%40%20E&13.1=A%20%40%20B%20%40%20C&13.2=none&14.1=A%20%40%20B&14.15=true&14.16=32&14.17=32&14.18=row%20major&14.19=&14.20=-1&14.21=1&14.22=0&14.23=open&14.2=none&24.1=A&24.15=false&24.16=64&24.17=96&24.18=expr&24.19=&24.0=-2%20*%20(Math.trunc(i%20%2F%208

bhosmer.github.io

https://bhosmer.github.io/mm/index.html?0=A%20%40%20(B%20%40%20C

bhosmer.github.io

https://bhosmer.github.io/mm/index.html?0=A%20%40%20B%20%40%20(C%20%40%20D

bhosmer.github.io

https://bhosmer.github.io/mm/index.html?0=A%20%40%20B%20%40%20C%20%40%20D%20%40%20E&1=A%20%40%20B%20%40%20C%20%40%20D%20%40%20E&2=none&23=closed&63=true&3.1=A%20%40%20B%20%40%20C%20%40%20D&3.4=true&3.5=32&3.6=32&3.7=row%20major&3.8=&3.9=-1&3.10=1&3.11=0&3.2=none&12.13=inherit&14.15=1&16.17=positive&16.18=left&16.19=bottom&16.20=back&21.0=A%20%40%20B%20%40%20C%20%40%20D%20%40%20E&21.1=A%20%40%20B%20%40%20C&21.2=none&22.1=A%20%40%20B&22.4=true&22.5=32&22.6=32&22.7=row%20major&22.8=&22.9=-1&22.10=1&22.11=0&22.23=open&22.2=none&24.1=A&24.4=false&24.5=64&24.6=96&24.7=expr&24.8=&24.0=-2%20*%20(Math.trunc(i%20%2F%208

bhosmer.github.io

https://bhosmer.github.io/mm/index.html?0=out%20%3D%20(attn%20%3D%20(Q%20%3D%20input%20%40%20wQ

bhosmer.github.io

https://arxiv.org/pdf/1909.08053.pdf

arxiv.org

https://bhosmer.github.io/mm/index.html?0=out%20%3D%20(attn%20%3D%20(Q%20%3D%20input_0_5%20%40%20wQ_5_4

bhosmer.github.io

https://bhosmer.github.io/mm/index.html?0=out%20%3D%20(attn%20%3D%20(Q%20%3D%20input_0_0%20%40%20wQ_0_2

bhosmer.github.io

https://arxiv.org/pdf/2205.14135.pdf

arxiv.org

https://github.com/NVIDIA/cutlass/blob/main/media/docs/efficient_gemm.md

github.com

https://github.com/karpathy/nanoGPT/blob/master/model.py

github.com

https://bhosmer.github.io/mm/index.html?0=layer_out%20%3D%20(attn_out%20%3D%20(attn%20%3D%20(Q%20%3D%20input%20%40%20wQ

bhosmer.github.io

https://bhosmer.github.io/mm/?0=L%20%40%20R&1=L%20%40%20R&2=none&12=closed&59=true&3.1=L&3.4=false&3.5=128&3.6=4&3.7=gaussian&3.8=&3.9=-1&3.10=1&3.11=0&3.0=&3.12=open&13.1=R&13.4=false&13.5=4&13.6=128&13.7=gaussian&13.8=&13.9=-1&13.10=1&13.11=0&13.0=&14.15=none&14.16=1&14.17=false&14.18=none&14.19=0&14.20=1&14.21=1&14.22=1&23.20=1&23.22=1&23.21=1&24.25=blocks&24.26=11.214&24.27=0&24.28=1&24.29=0&24.30=negative&24.31=left&24.32=top&24.33=front&24.12=open&34.35=10&34.36=true&34.37=2&34.38=1&34.39=0&34.40=0.5&34.41=10&34.42=false&34.43=false&34.12=open&44.45=local&44.46=0.3&44.47=0.5&44.48=0.7&44.49=1.25&44.50=0.77&44.51=0.74&44.52=0.04&44.12=open&53.8=&54.55=-147.50937470998977&54.56=141.1312665550063&54.57=104.24779022699425&58.55=-7.97607936427614&58.56=3.391570600844088&58.57=-11.685048965074932&expr=0&name=1&epilog=2&left=3&matmul=4&h=5&w=6&init=7&url=8&min=9&max=10&dropout=11&folder=12&right=13&anim=14&fuse=15&speed=16&hide%20inputs=17&alg=18&spin=19&i%20blocks=20&k%20blocks=21&j%20blocks=22&block=23&layout=24&scheme=25&gap=26&scatter=27&molecule=28&blast=29&polarity=30&left%20placement=31&right%20placement=32&result%20placement=33&deco=34&legends=35&shape=36&spotlight=37&row%20guides=38&flow%20guides=39&lens%20size=40&magnification=41&interior%20spotlight=42&axes=43&viz=44&sensitivity=45&min%20size=46&min%20light=47&max%20light=48&elem%20scale=49&zero%20hue=50&hue%20gap=51&hue%20spread=52&diag=53&cam=54&x=55&y=56&z=57&cam.target=58&compress=59

bhosmer.github.io

https://bhosmer.github.io/mm/index.html?0=out%20%3D%20(attn%20%3D%20(Q%20%3D%20input%20%40%20(wQ%20%3D%20wQ_A%20%40%20wQ_B

bhosmer.github.io

https://github.com/bhosmer/mm

github.com

https://bhosmer.github.io/mm/examples/attngpt2/index.html

bhosmer.github.io

https://colab.research.google.com/drive/1wZIoU20eRWKtRNCW7e5Iugm3MhfaE1f7?usp=sharing

colab.research.google.com