Record · corpus.blog/posts/01a0b5e7-c77a-73a0-ae81-24a6cb02ae22

Pretraining progress is mostly coming from data

dwarkesh.com · published 8 September 2026

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at dwarkesh.com Wayback capture

2blogs citing

First cited: 11 Sept 2026
Most recent: 16 Sept 2026
Rank this month: not ranked
Links from this post: 15
Words captured: 2,835

Cited by

Blog

In the post

Date

beren.io

Algorithmic Progress is not Exogenous Read at the source ↗

16 Sept 2026

transformernews.ai

Congress must not waste the AI policy window Read at the source ↗

11 Sept 2026

Links from this post

Link

Host

https://epoch.ai/gradient-updates/the-least-understood-driver-of-ai-progress

epoch.ai

https://github.com/allenai/olmes

github.com

http://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

cdn.openai.com

https://arxiv.org/abs/2501.00656

arxiv.org

http://huggingface.co/datasets/Skylion007/openwebtext

huggingface.co

http://huggingface.co/datasets/openbmb/Ultra-FineWeb

huggingface.co

Architecture Research as Addressing Constraints to Scaling Read at the source ↗

beren.io

https://arxiv.org/abs/2205.14135

arxiv.org

https://arxiv.org/pdf/2605.19407

arxiv.org

Ryan Greenblatt – What happens once AI can automate AI research? Read at the source ↗

dwarkesh.com

https://x.com/oneill_c?lang=en

x.com

https://arxiv.org/abs/2403.05812

arxiv.org

https://arxiv.org/pdf/2511.21622

arxiv.org

https://arxiv.org/abs/2305.13245

arxiv.org

https://epochai.substack.com/p/the-least-understood-driver-of-ai

epochai.substack.com