corpus.blog
56,966 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a0df50-42a2-7315-a56c-0dc7c4231ca3

PPO for LLMs: A Guide for Normal People

cameronrwolfe.substack.com · published 27 October 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at cameronrwolfe.substack.com Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
Links from this post
97
Words captured
12,098

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://substackcdn.com/image/fetch/$s_!PJsw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8db8bc5-f39d-4d1a-be16-26e0c0eb01a7_2502x1398.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!lQCe!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7117e42-c6ab-43c4-8878-5a88cb99c9ae_2203x870.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!cjh1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbee11fdb-dee8-4d4e-8819-b97642a17129_2008x338.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!YCeT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52061751-cc8a-4f3e-a889-5d4e542b21bf_2092x770.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!8D_n!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbfd6da8-2406-4197-b9d0-d3a1ec301b39_1496x876.png
substackcdn.com
https://en.wikipedia.org/wiki/Expected_value
en.wikipedia.org
https://substackcdn.com/image/fetch/$s_!45io!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F523baab0-10b4-438e-85d7-e7c5c0681209_1692x884.png
substackcdn.com
https://spinningup.openai.com/en/latest/spinningup/rl_intro.html
spinningup.openai.com
https://substackcdn.com/image/fetch/$s_!RBDE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd4b8b6b8-fe96-4b70-87d2-038a3b3511cf_1346x1134.png
substackcdn.com
https://cameronrwolfe.substack.com/i/173306894/markov-decision-process-mdp-versus-bandit-formulation
cameronrwolfe.substack.com
https://substackcdn.com/image/fetch/$s_!QUg4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b1a8412-5cfb-481f-bd50-473f0a6fd9b5_1992x1037.png
substackcdn.com
https://en.wikipedia.org/wiki/Markov_decision_process
en.wikipedia.org
https://substackcdn.com/image/fetch/$s_!KWz-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52f4f8de-4456-4cbd-935c-a945968b704d_1466x916.png
substackcdn.com
https://en.wikipedia.org/wiki/Gradient_descent
en.wikipedia.org
https://substackcdn.com/image/fetch/$s_!slrY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3072897-d905-42be-b385-6186c24ae059_2390x302.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!yR8D!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20b7b374-8bee-45fb-b7ee-a26008aa7259_1267x843.png
substackcdn.com
Policy Gradient Algorithms Read at the source ↗
lilianweng.github.io
https://arxiv.org/abs/2402.03300
arxiv.org
REINFORCE: Easy Online RL for LLMs Read at the source ↗
cameronrwolfe.substack.com
https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html
spinningup.openai.com
https://natolambert.com/
natolambert.com
https://rlhfbook.com/c/11-policy-gradients.html
rlhfbook.com
https://lilianweng.github.io/
lilianweng.github.io
https://cameronrwolfe.substack.com/p/policy-gradients-the-foundation-of
cameronrwolfe.substack.com
https://substackcdn.com/image/fetch/$s_!GetI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1685ea69-1b2c-438c-87ed-dba51c4bee65_2406x1065.png
substackcdn.com
https://andrewcharlesjones.github.io/journal/log-derivative.html
andrewcharlesjones.github.io
https://substackcdn.com/image/fetch/$s_!Rkmm!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0f526be-55f2-4eae-abd8-fa4382d8335a_1564x432.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!PYzF!,w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e4bdafe-cd71-48b7-8a10-abdc895432f7_1920x1076.gif
substackcdn.com
https://en.wikipedia.org/wiki/Automatic_differentiation
en.wikipedia.org
https://substackcdn.com/image/fetch/$s_!TwP0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4bb2d85-fdea-4cfc-a46b-e6c5f78ff4f4_1613x593.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!Ymws!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6b14bade-8617-4bfa-9e4a-59811bbe8de7_1374x218.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!s3m9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92c4ac85-74ac-4c12-8d51-c6c9b3bf22ba_2216x460.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!QhBq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd4801db8-b3f3-4ec3-9d3f-624b8ffbd550_1774x344.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!Vl-C!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58aa8bae-6778-4ec0-ac53-3f8b8550390f_2137x836.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!1PL6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3dbd6ad6-4d9e-4085-b4a7-849b29789350_1662x470.png
substackcdn.com
https://arxiv.org/abs/1502.05477
arxiv.org
https://substackcdn.com/image/fetch/$s_!x5A5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a9c1514-c3dd-4692-bb7a-d63644987d5e_1784x940.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!IXsZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a7d1530-a2cc-48c6-9e95-8571b781ba35_1994x792.png
substackcdn.com
https://en.wikipedia.org/wiki/Conjugate_gradient_method
en.wikipedia.org
https://substackcdn.com/image/fetch/$s_!fFIz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F301f1d55-7e7c-4c2f-8138-67a3bc162338_1872x388.png
substackcdn.com
https://arxiv.org/abs/1707.06347
arxiv.org
https://substackcdn.com/image/fetch/$s_!S1nc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc38f9ea3-d07f-4240-898e-de3c75e66878_2264x786.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!fqSm!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80447ac5-6fd2-4cbb-b33c-a4e385e7fc2c_1390x478.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!oHJG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f6be9f2-f165-4e48-be0c-e63074454d2a_2003x338.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!ovlv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38769a7f-6549-4fed-ab3e-f829185b5069_1544x642.png
substackcdn.com
https://cameronrwolfe.substack.com/i/167254905/kullback-leibler-kl-divergence
cameronrwolfe.substack.com
https://substackcdn.com/image/fetch/$s_!MMrI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc3d5004-2390-489f-995a-e0245c174535_2534x530.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!kyeM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7464e10-d669-4f6b-ab83-f1980b8918d4_2416x436.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!noKQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F55141cda-9010-48ea-ba62-5cd56e9bd814_1772x629.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!fXOv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb8133ba-f772-44f5-bfbc-19e800a842cc_1732x570.png
substackcdn.com
Reward Models Read at the source ↗
cameronrwolfe.substack.com
https://en.wikipedia.org/wiki/Mean_squared_error
en.wikipedia.org
https://en.wikipedia.org/wiki/Multilayer_perceptron
en.wikipedia.org
https://github.com/Farama-Foundation/Gymnasium
github.com
https://substackcdn.com/image/fetch/$s_!CHQh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1cc9a21-11e9-4c34-8d72-0576cde83e94_2086x894.png
substackcdn.com
https://arxiv.org/abs/1207.4708
arxiv.org
https://arxiv.org/abs/1602.01783
arxiv.org
https://arxiv.org/abs/1611.01224
arxiv.org
https://substackcdn.com/image/fetch/$s_!SgN4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc79fdf5d-6d9e-4f9c-b87e-885fe063de66_1814x499.png
substackcdn.com
https://arxiv.org/abs/1506.02438
arxiv.org
https://substackcdn.com/image/fetch/$s_!A4K-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c1e98c7-da70-4da6-a365-3b2fe9cd2230_1723x896.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!_U8s!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F18ae75ed-997b-4654-b383-dda56a8d9b2e_2298x716.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!v3wn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff11ed641-c3be-442a-ad17-b41072a721a8_2015x843.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!DRfY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdc295ca-a904-4885-85b2-59968c744cc0_2872x674.png
substackcdn.com
https://github.com/allenai/open-instruct
github.com
https://github.com/allenai/open-instruct/blob/main/open_instruct/ppo2.py
github.com
https://substackcdn.com/image/fetch/$s_!CJn6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0fd3791-df29-4a92-b185-21f6be4f2ddc_2176x642.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!uPv8!,w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56eba05c-359c-400d-920f-38a36dd4690a_1920x1078.gif
substackcdn.com
Direct Preference Optimization (DPO) Read at the source ↗
cameronrwolfe.substack.com
https://arxiv.org/abs/2009.01325
arxiv.org
https://cameronrwolfe.substack.com/i/138218863/evaluating-language-models-and-the-rouge-score
cameronrwolfe.substack.com
https://cameronrwolfe.substack.com/i/166169560/the-bradley-terry-model-of-preference
cameronrwolfe.substack.com
https://substackcdn.com/image/fetch/$s_!bjdU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F377524f4-cff7-44f9-b717-ed1e842b50bb_1612x970.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!oeIY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc713702e-ca1c-4759-bff4-b1dedfdf1bbf_1650x1016.png
substackcdn.com
https://arxiv.org/abs/1907.00456
arxiv.org
https://substackcdn.com/image/fetch/$s_!ZjlA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc088796c-52eb-45e5-afbc-195116ec5d1f_1612x764.png
substackcdn.com
https://huggingface.co/datasets/openai/summarize_from_feedback
huggingface.co
https://substackcdn.com/image/fetch/$s_!HYOl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda0d4ac2-cee0-464b-ba5d-3b278f1b1b9c_1628x846.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!d5Qe!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd1f3213a-8fd2-4703-8987-b2cfcbc5880a_662x672.png
substackcdn.com
https://arxiv.org/abs/2403.17031
arxiv.org
https://substackcdn.com/image/fetch/$s_!Om25!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbdf3dce4-738f-47c5-a5e3-f12c75887538_1864x1216.png
substackcdn.com
https://huggingface.co/datasets/CarperAI/openai_summarize_tldr
huggingface.co
https://substackcdn.com/image/fetch/$s_!fBTb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd4cd7447-83f7-4f34-921a-41672d4c391c_1866x536.png
substackcdn.com
https://joelouismarino.github.io/posts/2017/08/statistical_whitening/
joelouismarino.github.io
https://substackcdn.com/image/fetch/$s_!XoxA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9646db42-a84e-4dca-99a2-e585c053143c_1722x336.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!y_F0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F63af44b0-f8ab-4b8a-9872-276a6d78726f_2462x820.png
substackcdn.com
https://arxiv.org/abs/2203.02155
arxiv.org
https://cameronrwolfe.substack.com/p/the-history-of-open-source-llms-imitation
cameronrwolfe.substack.com
https://substackcdn.com/image/fetch/$s_!ZdHw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45180b88-a11e-42e8-8910-ceca2c3b447a_1618x980.png
substackcdn.com
https://cameronrwolfe.substack.com/i/88082618/language-models-are-few-shot-learners
cameronrwolfe.substack.com
https://cameronrwolfe.substack.com/i/117151147/few-shot-learning
cameronrwolfe.substack.com
https://substackcdn.com/image/fetch/$s_!ddkD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ee233ce-ea11-4928-bcbc-131c5fdc2f2f_1732x930.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!xMFU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9b979ad-bd64-47c4-bfe7-64890b661ba9_1660x724.png
substackcdn.com
https://cameronrwolfe.substack.com/p/language-model-scaling-laws-and-gpt
cameronrwolfe.substack.com
https://substackcdn.com/image/fetch/$s_!BTzq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F08415ad7-db55-4f46-8415-2fb3da1c9ab6_1350x1348.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!JB4X!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc9280f9-a159-4e81-ab17-86faf28f47ba_1876x882.png
substackcdn.com
https://en.wikipedia.org/wiki/Geometric_series
en.wikipedia.org