corpus.blog
41,554 blogs · 5,149,205 posts Claim your blog
Record · corpus.blog/posts/01a07df3-76c6-7166-af76-c1299b32bf9e

GPT in 60 Lines of NumPy

Jay Mody · published 30 January 2023

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at jaykmody.com Wayback capture
6blogs citing
First cited
12 Feb 2023
Most recent
12 May 2024
Rank this month
not ranked
External links
89
Words captured
9,713

Cited by

Blog
In the post
Date
g-w1.github.io
Building intuition with spaced repetition systems Read at the source ↗
12 May 2024
asun9.com
Machine Learning Jargon Read at the source ↗
4 Mar 2024
explainextended.com
Happy New Year: GPT in 500 lines of SQL Read at the source ↗
31 Dec 2023
jtolio.com
Just update rules between neurons Read at the source ↗
17 Mar 2023
Zero to Mastery
Python Monthly Newsletter 💻🐍 Read at the source ↗
1 Mar 2023
dipkumar.dev
KV cache in GPT: how it speeds up transformer inference Read at the source ↗
12 Feb 2023

Links from this post

Link
Host
https://github.com/jaymody/picoGPT/blob/29e78cc52b58ed2c1c483ffea2eb46ff6bdec785/gpt2_pico.py
github.com
https://github.com/jaymody/picoGPT
github.com
https://news.ycombinator.com/item?id=34726115
news.ycombinator.com
https://jiqihumanr.github.io/2023/04/13/gpt-from-scratch/
jiqihumanr.github.io
https://mlwizardry.netlify.app/nlp/gpt-from-scratch/
mlwizardry.netlify.app
https://arxiv.org/pdf/1706.03762.pdf
arxiv.org
How GPT3 Works - Visualizations and Animations Read at the source ↗
jalammar.github.io
https://en.wikipedia.org/wiki/GPT-3
en.wikipedia.org
https://machinelearningknowledge.ai/ezoimgfmt/b2611031.smushcdn.com/2611031/wp-content/uploads/2022/12/ChatGPT-Demo-of-Drafting-an-Email.png?lossy=0&strip=1&webp=1&ezimgfmt=ng:webp/ngcb1
machinelearningknowledge.ai
https://machinelearningknowledge.ai/ezoimgfmt/b2611031.smushcdn.com/2611031/wp-content/uploads/2022/12/ChatGPT-Example-Book-Summarization.png?lossy=0&strip=1&webp=1&ezimgfmt=ng:webp/ngcb1
machinelearningknowledge.ai
https://khrisdigital.com/wp-content/uploads/2022/12/image-1.png
khrisdigital.com
https://machinelearningknowledge.ai/ezoimgfmt/b2611031.smushcdn.com/2611031/wp-content/uploads/2022/12/ChatGPT-Examples-Explaining-Black-Holes.png?lossy=0&strip=1&webp=1&ezimgfmt=ng:webp/ngcb1
machinelearningknowledge.ai
https://machinelearningknowledge.ai/ezoimgfmt/b2611031.smushcdn.com/2611031/wp-content/uploads/2022/12/ChatGPT-Demo-of-Writing-SQL-Queries.png?lossy=0&strip=1&webp=1&ezimgfmt=ng:webp/ngcb1
machinelearningknowledge.ai
https://machinelearningknowledge.ai/ezoimgfmt/b2611031.smushcdn.com/2611031/wp-content/uploads/2022/12/Chat-GPT-Example-Writing-a-Will.png?lossy=0&strip=1&webp=1&ezimgfmt=ng:webp/ngcb1
machinelearningknowledge.ai
https://huggingface.co/course/chapter6/5?fw=pt
huggingface.co
https://huggingface.co/course/chapter6/6?fw=pt
huggingface.co
https://docs.cohere.ai/docs/controlling-generation-with-top-k-top-p
docs.cohere.ai
https://docs.cohere.ai/docs/temperature
docs.cohere.ai
https://arxiv.org/pdf/1609.04747.pdf
arxiv.org
https://www.youtube.com/watch?v=ErfnhcEV1O8
youtube.com
https://en.wikipedia.org/wiki/Self-supervised_learning
en.wikipedia.org
https://twitter.com/eturner303/status/1266264358771757057
twitter.com
https://en.wikipedia.org/wiki/Transfer_learning
en.wikipedia.org
https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf
s3-us-west-2.amazonaws.com
https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
d4mucfpksywv.cloudfront.net
https://arxiv.org/abs/2005.14165
arxiv.org
ChatGPT, OpenAI’s conversational AI model Read at the source ↗
openai.com
https://imgur.com/a/AbDFcgk
imgur.com
https://twitter.com/zswitten/status/1598380220943593472
twitter.com
https://github.com/openai/gpt-2/blob/master/src/encoder.py
github.com
https://www.google.com/search?q=tqdm
google.com
https://github.com/google/python-fire
github.com
https://github.com/jaymody/picoGPT/blob/a750c145ba4d09d5764806a6c78c71ffaff88e64/utils.py
github.com
https://github.com/karpathy/minGPT/blob/37baab71b9abea1b76ab957409a1cc2fbfba8a26/mingpt/bpe.py
github.com
https://github.com/jaymody/picoGPT/blob/29e78cc52b58ed2c1c483ffea2eb46ff6bdec785/utils.py
github.com
https://arxiv.org/pdf/1606.08415.pdf
arxiv.org
https://en.wikipedia.org/wiki/Softmax_function
en.wikipedia.org
https://arxiv.org/pdf/1607.06450.pdf
arxiv.org
https://arxiv.org/pdf/1502.03167.pdf
arxiv.org
https://stats.stackexchange.com/questions/474440/why-do-transformers-use-layer-norm-instead-of-batch-norm
stats.stackexchange.com
https://tungmphung.com/deep-learning-normalization-methods/
tungmphung.com
https://numpy.org/doc/stable/user/basics.indexing.html
numpy.org
https://preview.redd.it/n9fgba8b0qr01.png?auto=webp&s=e86d2d3447c777d3222016e81a0adfaec1a95592
preview.redd.it
https://developers.google.com/machine-learning/glossary/
developers.google.com
https://en.wikipedia.org/wiki/Monotonic_function
en.wikipedia.org
https://en.wikipedia.org/wiki/Lernaean_Hydra
en.wikipedia.org
https://arxiv.org/pdf/2002.04745.pdf
arxiv.org
https://arxiv.org/pdf/1512.03385.pdf
arxiv.org
https://programmathically.com/understanding-the-exploding-and-vanishing-gradients-problem/
programmathically.com
Attention? Attention! Read at the source ↗
lilianweng.github.io
https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/
jalammar.github.io
https://github.com/jaymody/picoGPT/blob/main/gpt2.py
github.com
https://github.com/jaymody/picoGPT/blob/a750c145ba4d09d5764806a6c78c71ffaff88e64/gpt2_pico.py
github.com
https://github.com/openai/gpt-2
github.com
https://gist.github.com/jaymody/9054ca64eeea7fad1b58a185696bb518
gist.github.com
https://github.com/google/jax
github.com
https://cloud.google.com/tpu/docs/system-architecture-tpu-vm
cloud.google.com
https://kipp.ly/blog/transformer-inference-arithmetic/
kipp.ly
Large Transformer Model Inference Optimization Read at the source ↗
lilianweng.github.io
https://pile.eleuther.ai/
pile.eleuther.ai
https://github.com/NVIDIA/Megatron-LM
github.com
How to Train Really Large Models on Many GPUs? Read at the source ↗
lilianweng.github.io
https://arxiv.org/pdf/1909.08053.pdf
arxiv.org
https://arxiv.org/pdf/2204.06514.pdf
arxiv.org
https://arxiv.org/pdf/2204.02311.pdf
arxiv.org
https://github.com/kingoflolz/mesh-transformer-jax
github.com
https://arxiv.org/pdf/2203.15556.pdf
arxiv.org
https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/
microsoft.com
https://arxiv.org/pdf/2005.14165.pdf
arxiv.org
https://arxiv.org/abs/2211.09110
arxiv.org
https://en.wikipedia.org/wiki/Goodhart%27s_law
en.wikipedia.org
https://github.com/lucidrains/x-transformers
github.com
https://arxiv.org/pdf/2102.11972.pdf
arxiv.org
https://arxiv.org/pdf/2302.13971.pdf
arxiv.org
https://huggingface.co/datasets/imdb
huggingface.co
https://arxiv.org/pdf/2109.01652.pdf
arxiv.org
https://arxiv.org/pdf/2203.02155.pdf
arxiv.org
https://aclanthology.org/2021.emnlp-main.243.pdf
aclanthology.org
https://arxiv.org/pdf/2110.07602.pdf
arxiv.org
https://arxiv.org/pdf/2101.00190.pdf
arxiv.org
https://arxiv.org/pdf/2103.10385.pdf
arxiv.org
https://arxiv.org/pdf/2106.09685.pdf
arxiv.org
https://arxiv.org/pdf/1902.00751.pdf
arxiv.org
https://arxiv.org/abs/2205.05638
arxiv.org
https://arxiv.org/pdf/2210.11416.pdf
arxiv.org
https://nlp.seas.harvard.edu/2018/04/03/attention.html
nlp.seas.harvard.edu
https://arxiv.org/pdf/2108.12409.pdf
arxiv.org
https://arxiv.org/pdf/2104.09864v4.pdf
arxiv.org
https://twitter.com/stephenroller/status/1579993017234382849
twitter.com
Elsewhere on jaykmody.com (2)