corpus.blog
56,966 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a08724-e7fd-72f9-a94d-b370e54bda4a

A Meticulous Guide to Advances in Deep Learning Efficiency over the Years

alexzhang13.github.io · published 30 October 2024

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at alexzhang13.github.io Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
Links from this post
418
Words captured
26,028

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://lilianweng.github.io/
lilianweng.github.io
https://www.anandtech.com/show/21310/nvidia-blackwell-architecture-and-b200b100-accelerators-announced-going-bigger-with-smaller-data
anandtech.com
https://ai.meta.com/blog/meta-llama-3-1/
ai.meta.com
https://people.idsia.ch/~juergen/deep-learning-history.html
people.idsia.ch
http://yann.lecun.com/exdb/publis/pdf/lecun-89e.pdf
yann.lecun.com
https://en.m.wikipedia.org/wiki/Sun-4
en.m.wikipedia.org
https://iclr-blog-track.github.io/2022/03/26/lecun1989/
iclr-blog-track.github.io
https://en.wikipedia.org/wiki/GeForce_256
en.wikipedia.org
https://www.nextplatform.com/2015/10/28/inside-the-programming-evolution-of-gpu-computing/
nextplatform.com
https://www.researchgate.net/figure/Some-operations-of-each-level-of-BLAS_tbl1_232641623
researchgate.net
Advanced Matrix Multiplication Optimization on Modern Multi-Core Processors Read at the source ↗
salykova.github.io
https://fabiensanglard.net/cuda/
fabiensanglard.net
https://blogs.nvidia.com/blog/what-is-cuda-2/
blogs.nvidia.com
https://en.wikipedia.org/wiki/CUDA
en.wikipedia.org
https://github.com/NVIDIA/cutlass/blob/main/media/docs/cute/01_layout.md
github.com
https://docs.nvidia.com/nvpl/_static/sparse/storage_format/sparse_matrix.html
docs.nvidia.com
https://www.ibm.com/topics/support-vector-machine
ibm.com
https://towardsdatascience.com/boosting-algorithms-explained-d38f56ef3f30
towardsdatascience.com
https://www.coursera.org/articles/decision-tree-machine-learning
coursera.org
https://people.eecs.berkeley.edu/~jordan/kernels/0521813972c02_p25-46.pdf
people.eecs.berkeley.edu
https://www.reddit.com/r/MachineLearning/comments/hoo6m8/d_ml_oldtimers_when_did_deep_learning_really_take/?captcha=1
reddit.com
https://www.cs.princeton.edu/courses/archive/fall17/cos429/notes/cos429_fall2017_lecture4_interest_points.pdf
cs.princeton.edu
https://www.cs.princeton.edu/courses/archive/fall11/cos429/notes/cos429_f11_lecture03_filtering.pdf
cs.princeton.edu
https://www.image-net.org/static_files/papers/imagenet_cvpr09.pdf?ref=blog.roboflow.com
image-net.org
https://ieeexplore.ieee.org/document/5206848
ieeexplore.ieee.org
https://image-net.org/challenges/LSVRC/2012/results
image-net.org
https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
proceedings.neurips.cc
https://ieeexplore.ieee.org/document/5995504
ieeexplore.ieee.org
https://papers.nips.cc/paper_files/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
papers.nips.cc
https://www.techpowerup.com/gpu-specs/geforce-gtx-580.c270
techpowerup.com
https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/
machinelearningmastery.com
https://arxiv.org/abs/1207.0580
arxiv.org
https://code.google.com/archive/p/cuda-convnet/
code.google.com
https://github.com/albanie/convnet-burden
github.com
https://arxiv.org/abs/1202.2745
arxiv.org
https://people.idsia.ch/~juergen/DanNet-triggers-deep-CNN-revolution-2011.html
people.idsia.ch
https://sebastianraschka.com/faq/docs/first-cnn-gpu.html
sebastianraschka.com
https://www.askpython.com/python-modules/tensorflow-vs-pytorch-vs-jax
askpython.com
https://arxiv.org/pdf/1211.5590
arxiv.org
https://caffe.berkeleyvision.org/
caffe.berkeleyvision.org
https://www.tensorflow.org/api_docs/python/tf/compat/v1
tensorflow.org
https://en.wikipedia.org/wiki/Torch_(machine_learning
en.wikipedia.org
https://pytorch.org/
pytorch.org
https://www.tensorflow.org/tutorials/quickstart/beginner
tensorflow.org
https://keras.io/
keras.io
https://jax.readthedocs.io/en/latest/
jax.readthedocs.io
https://en.wikipedia.org/wiki/Just-in-time_compilation
en.wikipedia.org
https://ai.stackexchange.com/questions/8190/where-can-i-find-the-original-paper-that-introduced-rnns
ai.stackexchange.com
https://en.wikipedia.org/wiki/Gated_recurrent_unit
en.wikipedia.org
https://en.wikipedia.org/wiki/Long_short-term_memory
en.wikipedia.org
https://en.wikipedia.org/wiki/Linear_time-invariant_system
en.wikipedia.org
https://hazyresearch.stanford.edu/blog/2023-02-15-long-convs
hazyresearch.stanford.edu
https://en.wikipedia.org/wiki/Fast_Fourier_transform
en.wikipedia.org
https://arxiv.org/abs/1312.5602
arxiv.org
https://arxiv.org/abs/1707.06347
arxiv.org
https://arxiv.org/abs/1406.2661
arxiv.org
https://arxiv.org/abs/1906.04493
arxiv.org
https://arxiv.org/abs/1503.03585
arxiv.org
https://arxiv.org/abs/2011.13456
arxiv.org
https://arxiv.org/abs/1706.03762
arxiv.org
https://arxiv.org/abs/2004.05150
arxiv.org
https://arxiv.org/abs/2104.09864
arxiv.org
https://arxiv.org/abs/2109.08668
arxiv.org
https://arxiv.org/abs/1512.03385
arxiv.org
https://arxiv.org/abs/1905.11946
arxiv.org
https://arxiv.org/abs/1502.05477
arxiv.org
https://arxiv.org/abs/2108.12409
arxiv.org
https://www.dremio.com/wiki/grid-search/
dremio.com
https://machinelearningmastery.com/k-fold-cross-validation/
machinelearningmastery.com
Neural Architecture Search Read at the source ↗
lilianweng.github.io
https://sakana.ai/evolutionary-model-merge/
sakana.ai
https://pytorch.org/docs/stable/generated/torch.optim.Adam.html
pytorch.org
https://en.wikipedia.org/wiki/Stochastic_gradient_descent
en.wikipedia.org
https://www.stat.cmu.edu/~ryantibs/convexopt-F13/scribes/lec6.pdf
stat.cmu.edu
https://towardsdatascience.com/stochastic-gradient-descent-with-momentum-a84097641a5d
towardsdatascience.com
https://arxiv.org/abs/1609.04747
arxiv.org
https://arxiv.org/abs/1412.6980
arxiv.org
https://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf
jmlr.org
https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf
cs.toronto.edu
https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html
pytorch.org
https://www.mit.edu/~gfarina/2024/67220s24_L12_newton/L12.pdf
mit.edu
https://math.stackexchange.com/questions/2201384/what-is-the-definition-of-a-first-order-method
math.stackexchange.com
https://math.stackexchange.com/questions/2957673/second-order-taylor-series-terms-in-gradient-descent
math.stackexchange.com
https://arxiv.org/abs/1802.09568
arxiv.org
https://arxiv.org/pdf/2306.07179
arxiv.org
https://en.cppreference.com/w/cpp/standard_library
en.cppreference.com
https://towardsdatascience.com/saga-of-the-lottery-ticket-hypothesis-af30091f5cb
towardsdatascience.com
https://proceedings.neurips.cc/paper_files/paper/2015/file/ae0eb3eed39d2bcef4622b2499a05fe6-Paper.pdf
proceedings.neurips.cc
https://arxiv.org/abs/1409.1556
arxiv.org
https://arxiv.org/pdf/1803.03635
arxiv.org
https://arxiv.org/abs/1810.02340
arxiv.org
https://arxiv.org/abs/2006.05467
arxiv.org
https://www.cs.toronto.edu/~kriz/cifar.html
cs.toronto.edu
https://docs.nvidia.com/cuda/cusparse/
docs.nvidia.com
https://developer.nvidia.com/blog/cutlass-linear-algebra-cuda/
developer.nvidia.com
https://arxiv.org/pdf/2308.06767
arxiv.org
https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt/
developer.nvidia.com
https://www.nvidia.com/en-us/glossary/recommendation-system/
nvidia.com
https://princeton-introml.github.io/files/ch20.pdf
princeton-introml.github.io
https://mbernste.github.io/posts/normed_vector_space/
mbernste.github.io
https://towardsdatascience.com/introduction-to-embedding-clustering-and-similarity-11dd80b00061
towardsdatascience.com
https://arxiv.org/pdf/2007.14523
arxiv.org
https://dl.acm.org/doi/10.5555/645925.671516
dl.acm.org
https://arxiv.org/pdf/2010.10784
arxiv.org
https://medium.com/@lmpo/understanding-model-quantization-for-llms-1573490d44ad
medium.com
https://en.wikipedia.org/wiki/Single-precision_floating-point_format
en.wikipedia.org
https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html
docs.nvidia.com
https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf
nvidia.com
https://aman.ai/primers/ai/quantization/
aman.ai
https://speechprocessingbook.aalto.fi/Modelling/Vector_quantization_VQ.html
speechprocessingbook.aalto.fi
https://steve.hollasch.net/cgindex/coding/ieeefloat.html
steve.hollasch.net
https://arxiv.org/pdf/1905.12322
arxiv.org
https://www.nvidia.com/en-us/data-center/h100/
nvidia.com
https://github.com/NVIDIA/apex
github.com
https://pytorch.org/docs/stable/amp.html
pytorch.org
https://residentmario.github.io/pytorch-training-performance-guide/mixed-precision.html
residentmario.github.io
https://arxiv.org/pdf/1510.00149
arxiv.org
https://arxiv.org/abs/1510.00149
arxiv.org
https://www.linkedin.com/posts/haiyongw_the-1000-times-increase-in-nvidia-gpu-performance-activity-7110666027297869826-LIMv
linkedin.com
https://www.amazon.com/Programming-Massively-Parallel-Processors-Hands/dp/0124159923
amazon.com
https://discord.com/invite/Wu4pdW8QqM
discord.com
https://blog.cloudflare.com/branch-predictor/
blog.cloudflare.com
https://superuser.com/questions/324284/what-is-meant-by-the-terms-cpu-core-die-and-package
superuser.com
https://en.wikipedia.org/wiki/Nvidia_Tesla
en.wikipedia.org
https://cvw.cac.cornell.edu/gpu-architecture/gpu-characteristics/simt_warp
cvw.cac.cornell.edu
https://developer.nvidia.com/blog/cuda-refresher-cuda-programming-model/
developer.nvidia.com
https://carpentries-incubator.github.io/lesson-gpu-programming/global_local_memory.html
carpentries-incubator.github.io
https://www.tomshardware.com/news/glossary-dram-ram-graphics-cards-gddr-definition,38002.html
tomshardware.com
https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html
docs.nvidia.com
https://en.wikipedia.org/wiki/Volta_(microarchitecture
en.wikipedia.org
https://personal.utdallas.edu/~dodge/EE2310/lec13.pdf
personal.utdallas.edu
https://www.youtube.com/watch?v=jVzSBkbfdiw&ab_channel=JacobSorber
youtube.com
https://en.wikipedia.org/wiki/Thread_block_(CUDA_programming
en.wikipedia.org
https://developer.nvidia.com/blog/using-shared-memory-cuda-cc/
developer.nvidia.com
https://www.mathworks.com/help/parallel-computing/stencil-operations-on-a-gpu.html
mathworks.com
https://ulhpc-tutorials.readthedocs.io/en/latest/cuda/exercises/convolution/
ulhpc-tutorials.readthedocs.io
https://docs.nvidia.com/gameworks/content/developertools/desktop/analysis/report/cudaexperiments/kernellevel/achievedoccupancy.htm
docs.nvidia.com
https://nanxiao.gitbooks.io/cuda-little-book/content/posts/compute-bound-and-memory-bound-kernels.html
nanxiao.gitbooks.io
https://dublog.net/blog/all-the-activations/
dublog.net
https://docs.nvidia.com/deeplearning/performance/dl-performance-gpu-background/index.html
docs.nvidia.com
https://www.cudocompute.com/blog/comparative-analysis-of-nvidia-a100-vs-h100-gpus
cudocompute.com
https://www.intel.com/content/dam/develop/external/us/en/documents/31848-compilerautovectorizationguide.pdf
intel.com
https://en.wikipedia.org/wiki/PCI_Express
en.wikipedia.org
https://en.wikipedia.org/wiki/Pascal_(microarchitecture
en.wikipedia.org
https://www.arccompute.io/arc-blog/nvidia-h100-pcie-vs-sxm5-form-factors-which-gpu-is-right-for-your-company
arccompute.io
https://stackoverflow.com/questions/5736968/why-is-cuda-pinned-memory-so-fast
stackoverflow.com
https://lumetta.web.engr.illinois.edu/408-S20/slide-copies/ece408-lecture16-S20.pdf
lumetta.web.engr.illinois.edu
Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning Read at the source ↗
timdettmers.com
https://arxiv.org/pdf/1704.04760
arxiv.org
https://en.wikipedia.org/wiki/Tensor_Processing_Unit
en.wikipedia.org
https://en.wikipedia.org/wiki/Application-specific_integrated_circuit
en.wikipedia.org
https://www.amd.com/en/products/graphics/desktops/radeon.html
amd.com
https://fileadmin.cs.lth.se/cs/Personal/Michael_Doggett/pubs/doggett12-tc.pdf
fileadmin.cs.lth.se
https://cloud.google.com/tpu/docs/system-architecture-tpu-vm
cloud.google.com
https://cloud.google.com/tpu/docs/training-on-tpu-pods
cloud.google.com
https://www.arm.com/glossary/fpga
arm.com
https://halverscience.net/fpgas_for_sci_and_eng/
halverscience.net
https://www.imc.com/us/articles/how-are-fpgas-used-in-trading
imc.com
https://en.wikipedia.org/wiki/Spiking_neural_network
en.wikipedia.org
https://research.ibm.com/publications/truenorth-design-and-tool-flow-of-a-65-mw-1-million-neuron-programmable-neurosynaptic-chip
research.ibm.com
https://www.etched.com/
etched.com
https://arxiv.org/abs/2005.14165
arxiv.org
Introducing ChatGPT Read at the source ↗
openai.com
https://arxiv.org/pdf/2109.01652
arxiv.org
https://arxiv.org/abs/2203.02155
arxiv.org
https://arxiv.org/abs/2306.11644
arxiv.org
https://en.wikipedia.org/wiki/Foundation_model
en.wikipedia.org
https://arxiv.org/abs/2404.19756
arxiv.org
https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/
developer.nvidia.com
https://developer.download.nvidia.com/CUDA/training/StreamsAndConcurrencyWebinar.pdf
developer.download.nvidia.com
https://docs.nvidia.com/cuda/pdf/Hopper_Tuning_Guide.pdf
docs.nvidia.com
https://hazyresearch.stanford.edu/blog/2024-05-12-tk
hazyresearch.stanford.edu
https://github.com/HazyResearch/ThunderKittens
github.com
https://arxiv.org/abs/1604.06174
arxiv.org
https://arxiv.org/abs/1910.02054
arxiv.org
https://peterchng.com/blog/2024/06/11/what-is-the-transformer-kv-cache/
peterchng.com
https://arxiv.org/pdf/2004.05150
arxiv.org
https://arxiv.org/pdf/1904.10509
arxiv.org
https://github.com/openai/sparse_attention
github.com
https://arxiv.org/abs/2001.04451
arxiv.org
https://github.com/lucidrains/reformer-pytorch
github.com
https://arxiv.org/abs/2006.04768
arxiv.org
https://github.com/lucidrains/linformer
github.com
https://en.wikipedia.org/wiki/Johnson%E2%80%93Lindenstrauss_lemma
en.wikipedia.org
https://github.com/allenai/longformer
github.com
https://arxiv.org/abs/2009.14794
arxiv.org
https://github.com/lucidrains/performer-pytorch
github.com
https://arxiv.org/abs/2404.07143
arxiv.org
https://github.com/alexzhang13/InfiniAttention
github.com
https://www.ethanepperly.com/index.php/2021/10/26/big-ideas-in-applied-math-low-rank-matrices/
ethanepperly.com
https://arxiv.org/pdf/2305.13245
arxiv.org
https://arxiv.org/abs/2305.13245
arxiv.org
https://arxiv.org/abs/1911.02150
arxiv.org
https://arxiv.org/abs/2101.03961
arxiv.org
https://arxiv.org/pdf/1701.06538
arxiv.org
https://arxiv.org/abs/1902.00751
arxiv.org
https://arxiv.org/abs/2106.09685
arxiv.org
https://arxiv.org/abs/2305.14314
arxiv.org
https://en.wikipedia.org/wiki/Gaussian_measure
en.wikipedia.org
https://developer.nvidia.com/blog/unified-memory-cuda-beginners/
developer.nvidia.com
https://github.com/huggingface/peft
github.com
https://arxiv.org/abs/2101.00190
arxiv.org
https://arxiv.org/abs/2104.08691
arxiv.org
https://arxiv.org/abs/2205.05638
arxiv.org
https://arxiv.org/abs/2403.14608
arxiv.org
https://arxiv.org/abs/2404.03592
arxiv.org
https://www.beren.io/2023-04-04-DL-models-are-secretly-linear/
beren.io
https://arxiv.org/pdf/2303.02536
arxiv.org
https://stackoverflow.com/questions/7335920/what-specifically-are-wall-clock-time-user-cpu-time-and-system-cpu-time-in-uni
stackoverflow.com
https://stackoverflow.com/questions/56601075/what-is-a-fused-kernel-or-fused-layer-in-deep-learning
stackoverflow.com
Introducing Triton: Open-source GPU programming for neural networks Read at the source ↗
openai.com
https://arxiv.org/pdf/2402.13499v1
arxiv.org
https://arxiv.org/pdf/2205.14135
arxiv.org
https://arxiv.org/abs/2205.14135
arxiv.org
https://arxiv.org/abs/1805.02867
arxiv.org
https://arxiv.org/abs/2307.08691
arxiv.org
https://www.youtube.com/watch?v=g5ZKBH6UQvE&list=PLRRuQYjFhpmubuwx-w8X964ofVkW1T8O4&index=21&ab_channel=ProgrammingMassivelyParallelProcessors
youtube.com
https://pytorch.org/blog/flexattention/
pytorch.org
https://github.com/alexzhang13/flashattention2-custom-mask
github.com
https://arxiv.org/abs/2407.08608
arxiv.org
https://docs.nvidia.com/cuda/parallel-thread-execution/
docs.nvidia.com
https://github.com/NVIDIA/cutlass/blob/main/media/docs/efficient_gemm.md
github.com
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision Read at the source ↗
tridao.me
https://github.com/facebookresearch/xformers/releases
github.com
https://arxiv.org/abs/2410.10989
arxiv.org
https://github.com/linkedin/Liger-Kernel/
github.com
https://arxiv.org/abs/2311.05908
arxiv.org
https://arxiv.org/abs/2312.00752
arxiv.org
https://arxiv.org/abs/2111.00396
arxiv.org
https://nvlabs.github.io/instant-ngp/
nvlabs.github.io
https://github.com/Ligo-Biosciences/AlphaFold3
github.com
https://www.nature.com/articles/s41586-024-07487-w
nature.com
https://www.nobelprize.org/prizes/chemistry/2024/summary/
nobelprize.org
https://www.ligo.bio/
ligo.bio
https://github.com/lucidrains/triangle-multiplicative-module
github.com
https://github.com/alexzhang13/msa
github.com
https://arxiv.org/pdf/1802.04799
arxiv.org
A Gentle Introduction to LLVM IR Read at the source ↗
mcyoung.xyz
https://www.codecademy.com/resources/docs/general/machine-code
codecademy.com
https://uditagarwal.in/ml-compilers-part-2-graph-optimizations/
uditagarwal.in
https://onnx.ai/get-started.html
onnx.ai
https://onnxruntime.ai/
onnxruntime.ai
https://arxiv.org/pdf/2002.03794
arxiv.org
https://arxiv.org/abs/1802.04799
arxiv.org
https://www.cs.princeton.edu/~wayne/kleinberg-tardos/pearson/03Graphs.pdf
cs.princeton.edu
https://docs.calyxir.org/frontends/tvm-relay.html
docs.calyxir.org
https://www.cs.princeton.edu/courses/archive/spring19/cos320/lectures/lecture4.pdf
cs.princeton.edu
https://en.wikipedia.org/wiki/Dead-code_elimination
en.wikipedia.org
https://tvm.apache.org/docs/tutorial/tensor_expr_get_started.html
tvm.apache.org
https://mlir.llvm.org/getting_started/
mlir.llvm.org
https://github.com/NVIDIA/TensorRT
github.com
https://github.com/NVIDIA/TensorRT-LLM
github.com
https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html
pytorch.org
https://github.com/pytorch/glow
github.com
https://pytorch.org/blog/introducing-nvfuser-a-deep-learning-compiler-for-pytorch/
pytorch.org
https://pytorch.org/docs/stable/torch.compiler_dynamo_overview.html
pytorch.org
https://dev-discuss.pytorch.org/t/torchinductor-a-pytorch-native-compiler-with-define-by-run-ir-and-symbolic-shapes/747
dev-discuss.pytorch.org
https://github.com/srush/Triton-Puzzles
github.com
https://github.com/alexzhang13/Triton-Puzzles-Solutions
github.com
https://github.com/linkedin/Liger-Kernel
github.com
https://medium.com/@minhanh.dongnguyen/megatron-lm-how-model-parallelism-is-pushing-language-models-to-new-heights-c21a5343e06a
medium.com
https://huggingface.co/datasets/allenai/c4
huggingface.co
https://arxiv.org/pdf/2407.21783
arxiv.org
https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html
docs.nvidia.com
https://arxiv.org/pdf/1609.04836
arxiv.org
https://arxiv.org/abs/1609.04836
arxiv.org
https://www.kaggle.com/datasets/hojjatk/mnist-dataset
kaggle.com
https://arxiv.org/abs/1811.02084
arxiv.org
https://arxiv.org/abs/1909.08053
arxiv.org
https://arxiv.org/pdf/1811.06965
arxiv.org
https://arxiv.org/abs/1811.06965
arxiv.org
https://people.eecs.berkeley.edu/~matei/papers/2019/sosp_pipedream.pdf
people.eecs.berkeley.edu
https://arxiv.org/pdf/2006.09503
arxiv.org
https://openreview.net/pdf?id=cw-EmNq5zfD
openreview.net
https://github.com/NVIDIA/Megatron-LM?tab=readme-ov-file
github.com
https://images.nvidia.com/aem-dam/Solutions/Data-Center/nvidia-dgx-a100-datasheet.pdf
images.nvidia.com
https://arxiv.org/pdf/1910.02054
arxiv.org
https://www.deepspeed.ai/tutorials/
deepspeed.ai
https://arxiv.org/pdf/2311.09431
arxiv.org
https://arxiv.org/abs/2310.01889
arxiv.org
https://github.com/lucidrains/ring-attention-pytorch
github.com
https://arxiv.org/abs/2305.19370
arxiv.org
https://arxiv.org/abs/2311.09431
arxiv.org
https://pytorch.org/tutorials/intermediate/dist_tuto.html
pytorch.org
https://aws.amazon.com/what-is/cli/
aws.amazon.com
https://pytorch.org/tutorials/intermediate/ddp_tutorial.html
pytorch.org
https://github.com/NVIDIA/nccl
github.com
https://github.com/facebookincubator/gloo
github.com
https://en.wikipedia.org/wiki/Message_Passing_Interface
en.wikipedia.org
https://arxiv.org/abs/1712.00409
arxiv.org
https://arxiv.org/pdf/2001.08361
arxiv.org
https://arxiv.org/abs/2001.08361
arxiv.org
https://openwebtext2.readthedocs.io/en/latest/
openwebtext2.readthedocs.io
https://statproofbook.github.io/D/prob-marg.html
statproofbook.github.io
https://arxiv.org/abs/2203.15556
arxiv.org
https://arxiv.org/abs/2206.14486
arxiv.org
https://arxiv.org/abs/2210.11399
arxiv.org
https://proceedings.neurips.cc/paper_files/paper/2023/hash/9d89448b63ce1e2e8dc7af72c984c196-Abstract-Conference.html
proceedings.neurips.cc
https://github.com/openai/human-eval
github.com
https://arxiv.org/abs/2108.07732
arxiv.org
https://arxiv.org/abs/2309.05463
arxiv.org
https://arxiv.org/abs/2404.14219
arxiv.org
Mixtral of experts Read at the source ↗
mistral.ai
GPT-4o mini: advancing cost-efficient intelligence Read at the source ↗
openai.com
https://blog.google/technology/ai/google-gemini-update-flash-ai-assistant-io-2024/
blog.google
https://arxiv.org/pdf/2310.06694
arxiv.org
https://arxiv.org/abs/2310.06694
arxiv.org
https://hanj.cs.illinois.edu/cs412/bk3/KL-divergence.pdf
hanj.cs.illinois.edu
https://openreview.net/forum?id=5h0qf7IBZZ
openreview.net
https://arxiv.org/abs/2308.02019
arxiv.org
https://arxiv.org/abs/2310.16944
arxiv.org
https://huggingface.co/docs/trl/main/en/sft_trainer
huggingface.co
https://arxiv.org/abs/2305.18290
arxiv.org
https://arxiv.org/abs/2208.07339
arxiv.org
https://github.com/bitsandbytes-foundation/bitsandbytes
github.com
https://arxiv.org/abs/2210.17323
arxiv.org
https://arxiv.org/abs/2208.11580
arxiv.org
https://proceedings.neurips.cc/paper/1992/hash/303ed4c69846ab36c2904d3ba8573050-Abstract.html
proceedings.neurips.cc
https://arxiv.org/abs/2306.00978
arxiv.org
https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/
nvidia.com
https://arxiv.org/abs/2310.11453
arxiv.org
https://pytorch.org/blog/quantization-aware-training/
pytorch.org
https://arxiv.org/abs/2402.17764
arxiv.org
https://arxiv.org/abs/2211.10438
arxiv.org
https://hanlab.mit.edu/blog/block-sparse-attention
hanlab.mit.edu
https://arxiv.org/abs/2309.17453
arxiv.org
https://paperswithcode.com/method/sliding-window-attention
paperswithcode.com
https://arxiv.org/abs/1903.05895
arxiv.org
https://en.wikipedia.org/wiki/Discrete_Fourier_transform
en.wikipedia.org
https://linear.axler.net/BlockDiagonal.pdf
linear.axler.net
https://proceedings.mlr.press/v162/dao22a.html
proceedings.mlr.press
https://quic.github.io/cloud-ai-sdk-pages/1.12/Getting-Started/Model-Architecture-Support/Large-Language-Models/llm/
quic.github.io
https://github.com/ggerganov/llama.cpp
github.com
https://github.com/ggerganov
github.com
https://en.wikipedia.org/wiki/Llama_(language_model
en.wikipedia.org
https://www.usenix.org/system/files/osdi22-yu.pdf
usenix.org
https://www.usenix.org/conference/osdi22/presentation/yu
usenix.org
https://arxiv.org/pdf/2309.06180
arxiv.org
https://arxiv.org/abs/2309.06180
arxiv.org
https://arxiv.org/abs/2403.02310
arxiv.org
https://arxiv.org/html/2407.07000v1
arxiv.org
https://arxiv.org/abs/2211.17192
arxiv.org
https://arxiv.org/pdf/2401.10774
arxiv.org
https://arxiv.org/abs/2401.10774
arxiv.org
https://medium.com/image-processing-with-python/the-feedforward-network-ffn-in-the-transformer-model-6bb6e0ff18db
medium.com
https://arxiv.org/abs/2210.15191
arxiv.org
https://minerl.readthedocs.io/en/latest/
minerl.readthedocs.io
https://www.swebench.com/multimodal.html
swebench.com
https://www.youtube.com/c/DwarkeshPatel
youtube.com
https://www.semianalysis.com/p/100000-h100-clusters-power-network
semianalysis.com
https://www.reuters.com/technology/microsoft-openai-planning-100-billion-data-center-project-information-reports-2024-03-29/
reuters.com
https://ai.princeton.edu/news/2024/princeton-invests-new-300-gpu-cluster-academic-ai-research
ai.princeton.edu
https://baxtel.com/news/university-of-texas-to-host-cluster-of-600-nvidia-h100-gpus
baxtel.com
https://arxiv.org/abs/2407.21783
arxiv.org
https://www.factorialfunds.com/blog/thoughts-on-llama-3
factorialfunds.com
https://github.com/KellerJordan/modded-nanogpt
github.com
https://github.com/KellerJordan/modded-nanogpt?tab=readme-ov-file
github.com
https://github.com/karpathy/llm.c
github.com
https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
huggingface.co
https://x.com/kellerjordan0/status/1798863559243513937
x.com
https://www.databricks.com/blog/mosaicbert
databricks.com
https://arxiv.org/abs/1907.11692
arxiv.org
https://arxiv.org/abs/1810.04805
arxiv.org
https://github.com/Dao-AILab/flash-attention
github.com
https://arxiv.org/pdf/2208.08124
arxiv.org
https://github.com/HazyResearch/flash-fft-conv
github.com
https://www.ai21.com/jamba
ai21.com
https://github.com/togethercomputer/stripedhyena
github.com
https://arxiv.org/abs/2305.13048
arxiv.org
https://x.com/blinkdl_ai
x.com
https://arxiv.org/abs/2307.08621
arxiv.org
https://arxiv.org/abs/2408.15237
arxiv.org
https://hazyresearch.stanford.edu/blog/2024-10-14-lolcats-p1
hazyresearch.stanford.edu
https://www.nvidia.com/en-us/data-center/resources/mlperf-benchmarks/
nvidia.com
https://cerebras.ai/
cerebras.ai
https://www.together.ai/
together.ai
https://groq.com/
groq.com
https://tenstorrent.com/
tenstorrent.com
https://colab.google/
colab.google
https://www.nvidia.com/en-us/data-center/tesla-t4/
nvidia.com
https://studiolab.sagemaker.aws/
studiolab.sagemaker.aws
https://lightning.ai/
lightning.ai
https://www.kaggle.com/
kaggle.com
https://www.nvidia.com/en-gb/data-center/tesla-k80/
nvidia.com
https://www.kaggle.com/discussions/general/108481
kaggle.com
https://pytorch.org/torchtune/stable/index.html
pytorch.org
https://github.com/huggingface/accelerate
github.com
https://github.com/microsoft/DeepSpeed
github.com
https://github.com/axolotl-ai-cloud/axolotl
github.com
https://pytorch.org/docs/stable/quantization-support.html
pytorch.org
https://github.com/NVIDIA/TensorRT-Model-Optimizer
github.com
https://www.youtube.com/watch?v=LuhJEEJQgUM&ab_channel=GPUMODE
youtube.com
https://nvitop.readthedocs.io/en/latest/
nvitop.readthedocs.io
https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html
pytorch.org
https://developer.nvidia.com/nsight-compute
developer.nvidia.com
https://docs.nvidia.com/nsight-compute/NsightComputeCli/index.html
docs.nvidia.com
https://developer.nvidia.com/nsight-systems
developer.nvidia.com
https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ
youtube.com
https://www.youtube.com/watch?v=4pkbXmE4POc&list=PLRRuQYjFhpmubuwx-w8X964ofVkW1T8O4
youtube.com
PyTorch internals Read at the source ↗
blog.ezyang.com
How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog Read at the source ↗
siboehm.com
CUTLASS Tutorial: Fast Matrix-Multiplication with WGMMA on NVIDIA® Hopper™ GPUs Read at the source ↗
research.colfax-intl.com
https://www.youtube.com/watch?v=RgUl6BlyaF4&list=PL80kAHvQbh-qGtNc54A6KW4i4bkTPjiRF
youtube.com
https://residentmario.github.io/pytorch-training-performance-guide/intro.html
residentmario.github.io
https://www.nvidia.com/en-gb/data-center/dgx-systems/
nvidia.com
https://www.arccompute.io/solutions/hardware/gpu-servers
arccompute.io
https://lambdalabs.com/deep-learning/servers/hyperplane
lambdalabs.com
https://x.com/a1zhang/status/1851963904491950132
x.com
https://github.com/merrymercy/awesome-tensor-compilers
github.com
http://lastweek.io/notes/MLIR/
lastweek.io
https://arxiv.org/abs/2104.04473
arxiv.org
https://arxiv.org/abs/2205.05198
arxiv.org
https://arxiv.org/abs/2009.06732
arxiv.org
https://arxiv.org/abs/2106.08962
arxiv.org
https://arxiv.org/abs/2306.15552
arxiv.org
https://arxiv.org/abs/2308.06767
arxiv.org
https://arxiv.org/abs/2312.03863
arxiv.org
https://ieeexplore.ieee.org/document/10177729
ieeexplore.ieee.org