corpus.blog
56,966 blogs · [ { "id": "01a087de-9145-73ad-b82f-d63f42fea3f4", "title": "No, agentic AI will not massively boost productivity in scientific research; here’s how to actually boost scientific productivity", "url": "https://severelytheoretical.wordpress.com/2026/02/13/no-agentic-ai-will-not-massively-boost-productivity-in-scientific-research-heres-how-to-actually-boost-scientific-productivity/", "published_at": "2026-02-13T20:29:34+00:00" }, { "id": "01a087de-9145-73ad-b82f-d63f436028a9", "title": "Continual training of Llama-3.1-8B for 809B tokens", "url": "https://severelytheoretical.wordpress.com/2025/04/21/continual-training-of-llama-3-1-8b-for-809b-tokens/", "published_at": "2025-04-21T21:46:09+00:00" }, { "id": "01a087de-9145-73ad-b82f-d63f43bb445c", "title": "Language development in children is more like post-training than pre-training in LLMs", "url": "https://severelytheoretical.wordpress.com/2025/02/28/language-development-in-children-is-more-like-post-training-than-pre-training-in-llms/", "published_at": "2025-02-28T19:17:04+00:00" }, { "id": "01a087de-9145-73ad-b82f-d63f43e5c45e", "title": "On the entropic brain, trapped priors, and machine learning", "url": "https://severelytheoretical.wordpress.com/2024/10/28/on-the-entropic-brain-trapped-priors-and-machine-learning/", "published_at": "2024-10-29T03:51:21+00:00" }, { "id": "01a087de-9145-73ad-b82f-d63f449f47f6", "title": "IsoFLOP curves of large language models are extremely flat", "url": "https://severelytheoretical.wordpress.com/2024/07/31/isoflop-curves-of-large-language-models-are-extremely-flat/", "published_at": "2024-07-31T13:24:54+00:00" }, { "id": "01a087de-9145-73ad-b82f-d63f44bdaefb", "title": "Does Sora understand physics? A few simple observations", "url": "https://severelytheoretical.wordpress.com/2024/04/25/does-sora-understand-physics-a-few-simple-observations/", "published_at": "2024-04-25T23:39:26+00:00" }, { "id": "01a087de-9145-73ad-b82f-d63f44f6577b", "title": "The “it” of deep learning and convergent evolution", "url": "https://severelytheoretical.wordpress.com/2024/02/28/the-it-of-deep-learning-and-convergent-evolution/", "published_at": "2024-02-28T09:51:22+00:00" }, { "id": "01a087de-9145-73ad-b82f-d63f450d8e1e", "title": "Intelligence is a granularity problem (or the reality has a surprising amount of detail, so must intelligence)", "url": "https://severelytheoretical.wordpress.com/2024/01/02/intelligence-is-a-granularity-problem-or-the-reality-has-a-surprising-amount-of-detail-so-must-intelligence/", "published_at": "2024-01-02T20:51:13+00:00" }, { "id": "01a087de-9145-73ad-b82f-d63f45767838", "title": "Further thoughts on hallucinations in generative models", "url": "https://severelytheoretical.wordpress.com/2023/10/29/further-thoughts-on-hallucinations-in-generative-models/", "published_at": "2023-10-30T02:39:52+00:00" }, { "id": "01a087de-9145-73ad-b82f-d63f45d6ecc2", "title": "GPT-3.5 is surprisingly non-factual about literature", "url": "https://severelytheoretical.wordpress.com/2023/10/04/gpt-3-5-is-surprisingly-non-factual-about-literature/", "published_at": "2023-10-05T03:03:57+00:00" }, { "id": "01a0cb2b-3dd3-70a6-9c34-11b242f1cb13", "title": "FLOPs are all you need: a conjecture about what really makes deep learning work", "url": "https://severelytheoretical.wordpress.com/2023/08/14/flops-are-all-you-need-a-conjecture-about-what-really-makes-deep-learning-work/", "published_at": "2023-08-14T21:24:29+00:00" }, { "id": "01a0cb2b-3dd3-70a6-9c34-11b243cbc866", "title": "Can we get rid of all inductive biases?", "url": "https://severelytheoretical.wordpress.com/2023/07/08/can-we-get-rid-of-all-inductive-biases/", "published_at": "2023-07-08T22:16:28+00:00" }, { "id": "01a0cb2b-3dd3-70a6-9c34-11b244069e92", "title": "“Peer review” must die", "url": "https://severelytheoretical.wordpress.com/2023/06/30/peer-review-must-die/", "published_at": "2023-06-30T22:54:05+00:00" }, { "id": "01a0cb2b-3dd3-70a6-9c34-11b244c21519", "title": "Is RLHF really better than supervised finetuning?", "url": "https://severelytheoretical.wordpress.com/2023/04/30/is-rlhf-really-better-than-supervised-finetuning/", "published_at": "2023-04-30T19:35:14+00:00" }, { "id": "01a0cb2b-3dd3-70a6-9c34-11b24571da64", "title": "Self-supervised learning may solve all the major problems of cogsci", "url": "https://severelytheoretical.wordpress.com/2023/04/17/self-supervised-learning-may-solve-all-the-major-problems-of-cogsci/", "published_at": "2023-04-17T23:26:17+00:00" }, { "id": "01a0cb2b-3dd3-70a6-9c34-11b245b84498", "title": "Thoughts on GPT-4", "url": "https://severelytheoretical.wordpress.com/2023/04/09/thoughts-on-gpt-4/", "published_at": "2023-04-10T01:50:16+00:00" }, { "id": "01a0cb2b-3dd3-70a6-9c34-11b245c8b4fa", "title": "A rant on LLaMA: please stop training giant language models", "url": "https://severelytheoretical.wordpress.com/2023/03/05/a-rant-on-llama-please-stop-training-giant-language-models/", "published_at": "2023-03-05T19:44:41+00:00" }, { "id": "01a0cb2b-3dd3-70a6-9c34-11b246aabd00", "title": "Discussing ethics with ChatGPT", "url": "https://severelytheoretical.wordpress.com/2022/12/28/discussing-ethics-with-chatgpt/", "published_at": "2022-12-28T08:58:18+00:00" }, { "id": "01a0cb2b-3dd3-70a6-9c34-11b2479e508d", "title": "What will it take to achieve human-level sample efficiency in deep learning?", "url": "https://severelytheoretical.wordpress.com/2022/11/07/what-will-it-take-to-achieve-human-level-sample-efficiency-in-deep-learning/", "published_at": "2022-11-07T21:53:47+00:00" }, { "id": "01a0cb2b-3dd3-70a6-9c34-11b24833c674", "title": "The value of incremental, cumulative improvements is underestimated in AI/ML", "url": "https://severelytheoretical.wordpress.com/2022/08/18/the-value-of-incremental-cumulative-improvements-is-vastly-underestimated-in-ai-ml/", "published_at": "2022-08-18T22:42:24+00:00" }, { "id": "01a0cb2c-06fa-7319-b15b-f030e8d0de56", "title": "Thoughts on the new scaling laws for large language models", "url": "https://severelytheoretical.wordpress.com/2022/07/18/thoughts-on-the-new-scaling-laws-for-large-language-models/", "published_at": "2022-07-19T01:19:50+00:00" }, { "id": "01a0cb2c-06fa-7319-b15b-f030e921218e", "title": "Emergent problems with “emergent” capabilities", "url": "https://severelytheoretical.wordpress.com/2022/07/11/emergent-problems-with-emergent-capabilities/", "published_at": "2022-07-12T01:55:33+00:00" }, { "id": "01a0cb2c-06fa-7319-b15b-f030e9c1edc9", "title": "A simple plausibility argument for why scaling is probably not enough", "url": "https://severelytheoretical.wordpress.com/2022/07/05/a-simple-plausibility-argument-for-why-scaling-is-not-enough/", "published_at": "2022-07-05T18:31:08+00:00" }, { "id": "01a0cb2c-06fa-7319-b15b-f030eaa097c3", "title": "Neural networks are actually not very good at memorizing", "url": "https://severelytheoretical.wordpress.com/2022/05/11/neural-networks-are-actually-not-very-good-at-memorizing/", "published_at": "2022-05-11T16:04:26+00:00" }, { "id": "01a0cb2c-06fa-7319-b15b-f030eaa89e37", "title": "A critique of “Why Greatness Cannot Be Planned”", "url": "https://severelytheoretical.wordpress.com/2022/02/08/a-critique-of-why-greatness-cannot-be-planned/", "published_at": "2022-02-08T07:33:33+00:00" }, { "id": "01a0cb2c-06fa-7319-b15b-f030eba8200b", "title": "Catastrophic forgetting is yet another pet problem rendered obsolete by scale", "url": "https://severelytheoretical.wordpress.com/2021/10/11/catastrophic-forgetting-is-yet-another-pet-problem-rendered-obsolete-by-scale/", "published_at": "2021-10-11T08:23:19+00:00" }, { "id": "01a0cb2c-06fa-7319-b15b-f030ec576a67", "title": "How much “human-like” visual experience do current self-supervised learning algorithms need to achieve human-level object recognition?", "url": "https://severelytheoretical.wordpress.com/2021/09/24/how-much-human-like-visual-experience-do-current-self-supervised-learning-algorithms-need-to-achieve-human-level-object-recognition/", "published_at": "2021-09-24T04:16:48+00:00" }, { "id": "01a0cb2c-06fa-7319-b15b-f030ec898fa0", "title": "Ditching academic research in AI/ML", "url": "https://severelytheoretical.wordpress.com/2021/06/12/ditching-academic-research-in-ai-ml/", "published_at": "2021-06-13T00:24:39+00:00" }, { "id": "01a0cb2c-06fa-7319-b15b-f030ed6abf4c", "title": "Pinker on why humans are smarter than rats", "url": "https://severelytheoretical.wordpress.com/2021/06/07/pinker-on-why-humans-are-smarter-than-rats/", "published_at": "2021-06-07T19:03:23+00:00" }, { "id": "01a0cb2c-06fa-7319-b15b-f030ee61af91", "title": "AI research, wise passiveness, and negative capability", "url": "https://severelytheoretical.wordpress.com/2021/05/27/ai-research-wise-passiveness-and-negative-capability/", "published_at": "2021-05-27T20:21:37+00:00" }, { "id": "01a0cb2c-5de4-724e-b294-8123d108935e", "title": "Is compositionality/systematic generalization really a problem for neural networks?", "url": "https://severelytheoretical.wordpress.com/2021/03/19/is-compositionality-systematic-generalization-really-a-problem-for-neural-networks/", "published_at": "2021-03-19T21:47:25+00:00" }, { "id": "01a0cb2c-5de4-724e-b294-8123d2028266", "title": "On the futility of trying to be clever (the bitter lesson redux)", "url": "https://severelytheoretical.wordpress.com/2021/02/15/on-the-futility-of-trying-to-be-smart-the-bitter-lesson-redux/", "published_at": "2021-02-16T03:41:51+00:00" }, { "id": "01a0cb2c-5de4-724e-b294-8123d26e9f88", "title": "An optimistic perspective on the human-AI nexus", "url": "https://severelytheoretical.wordpress.com/2021/02/11/an-optimistic-perspective-on-the-human-ai-nexus/", "published_at": "2021-02-11T05:02:53+00:00" }, { "id": "01a0cb2c-5de4-724e-b294-8123d2b0f19e", "title": "Thoughts on Image-GPT", "url": "https://severelytheoretical.wordpress.com/2020/11/28/thoughts-on-image-gpt/", "published_at": "2020-11-29T01:20:03+00:00" }, { "id": "01a0cb2c-5de4-724e-b294-8123d2bc1948", "title": "Thoughts on GPT-3", "url": "https://severelytheoretical.wordpress.com/2020/09/01/thoughts-on-gpt-3/", "published_at": "2020-09-02T02:50:13+00:00" }, { "id": "01a0cb2c-5de4-724e-b294-8123d37c193f", "title": "Deep learning can make more use of available data", "url": "https://severelytheoretical.wordpress.com/2020/05/16/deep-learning-can-make-more-use-of-available-data/", "published_at": "2020-05-16T20:05:22+00:00" }, { "id": "01a0cb2c-5de4-724e-b294-8123d44786df", "title": "In defense of pure learning", "url": "https://severelytheoretical.wordpress.com/2020/02/10/in-defense-of-pure-learning/", "published_at": "2020-02-11T01:37:47+00:00" }, { "id": "01a0cb2c-5de4-724e-b294-8123d4de602a", "title": "Does the brain have to do deep credit assignment?", "url": "https://severelytheoretical.wordpress.com/2019/12/31/does-the-brain-have-to-do-deep-credit-assignment/", "published_at": "2019-12-31T23:57:47+00:00" }, { "id": "01a0cb2c-5de4-724e-b294-8123d5957e40", "title": "Google’s new paper on large-scale weakly supervised learning", "url": "https://severelytheoretical.wordpress.com/2019/11/17/googles-new-paper-on-large-scale-weakly-supervised-learning/", "published_at": "2019-11-18T00:14:05+00:00" }, { "id": "01a0cb2c-5de4-724e-b294-8123d5a759af", "title": "kin2vec: learning kinship relations", "url": "https://severelytheoretical.wordpress.com/2019/10/10/kin2vec-learning-kinship-relations/", "published_at": "2019-10-10T20:41:52+00:00" }, { "id": "01a0cb2c-a997-7010-88ea-e8cfbe44eedc", "title": "The relative value of learning over memorizing", "url": "https://severelytheoretical.wordpress.com/2019/06/15/the-relative-value-of-learning-over-memorizing/", "published_at": "2019-06-15T04:07:50+00:00" }, { "id": "01a0cb2c-a997-7010-88ea-e8cfbe55a932", "title": "A conjecture about “how the brain works”", "url": "https://severelytheoretical.wordpress.com/2019/02/06/a-conjecture-about-how-the-brain-works/", "published_at": "2019-02-07T02:29:05+00:00" }, { "id": "01a0cb2c-a997-7010-88ea-e8cfbe842655", "title": "Surprising things you can learn from lavish data with no supervision", "url": "https://severelytheoretical.wordpress.com/2018/12/31/surprising-things-you-can-learn-from-lavish-data-with-no-supervision/", "published_at": "2018-12-31T19:29:55+00:00" }, { "id": "01a0cb2c-a997-7010-88ea-e8cfbed388cd", "title": "Common initialization schemes for recurrent neural networks are likely suboptimal", "url": "https://severelytheoretical.wordpress.com/2018/09/13/common-initialization-schemes-for-recurrent-neural-networks-are-likely-suboptimal/", "published_at": "2018-09-13T22:19:25+00:00" }, { "id": "01a0cb2c-a997-7010-88ea-e8cfbf2e580a", "title": "Simple inductive biases to make neural networks train faster and generalize better: two case studies", "url": "https://severelytheoretical.wordpress.com/2018/08/22/simple-inductive-biases-to-make-neural-networks-train-faster-and-generalize-better-two-case-studies/", "published_at": "2018-08-23T01:22:27+00:00" }, { "id": "01a0cb2c-a997-7010-88ea-e8cfbf435c7d", "title": "Hyperreal numbers", "url": "https://severelytheoretical.wordpress.com/2018/08/03/hyperreal-numbers/", "published_at": "2018-08-04T02:31:04+00:00" }, { "id": "01a0cb2c-a997-7010-88ea-e8cfbf865bc6", "title": "Why do neural networks generalize well?", "url": "https://severelytheoretical.wordpress.com/2018/06/25/why-do-neural-networks-generalize-well/", "published_at": "2018-06-25T21:16:57+00:00" }, { "id": "01a0cb2c-a997-7010-88ea-e8cfbfae6840", "title": "The softmax bottleneck is a special case of a more general phenomenon", "url": "https://severelytheoretical.wordpress.com/2018/06/08/the-softmax-bottleneck-is-a-special-case-of-a-more-general-phenomenon/", "published_at": "2018-06-08T22:27:14+00:00" }, { "id": "01a0cb2c-a997-7010-88ea-e8cfc068c8c8", "title": "Comments on DeepMind’s grid cell paper", "url": "https://severelytheoretical.wordpress.com/2018/05/24/comments-on-deepminds-grid-cell-paper/", "published_at": "2018-05-24T23:53:33+00:00" }, { "id": "01a0cb2c-a997-7010-88ea-e8cfc0f63412", "title": "A simple cache model for image recognition", "url": "https://severelytheoretical.wordpress.com/2018/05/23/a-simple-cache-model-for-image-recognition/", "published_at": "2018-05-23T00:21:31+00:00" }, { "id": "01a0cb2c-f4f5-7194-99a8-d3d6a9fd114b", "title": "Short-horizon bias in meta-learning", "url": "https://severelytheoretical.wordpress.com/2018/04/09/short-horizon-bias-in-meta-learning/", "published_at": "2018-04-09T22:51:58+00:00" }, { "id": "01a0cb2c-f4f5-7194-99a8-d3d6aa70ac96", "title": "Human priors: you can’t have your cake and eat it too", "url": "https://severelytheoretical.wordpress.com/2018/03/14/human-priors-you-cant-have-your-cake-and-eat-it-too/", "published_at": "2018-03-14T22:06:19+00:00" }, { "id": "01a0cb2c-f4f5-7194-99a8-d3d6aa81cfeb", "title": "Compositional embeddings, again", "url": "https://severelytheoretical.wordpress.com/2018/02/26/compositional-embeddings-again/", "published_at": "2018-02-26T18:52:01+00:00" }, { "id": "01a0cb2c-f4f5-7194-99a8-d3d6ab8153a8", "title": "Holographic reduced representations and associative LSTMs", "url": "https://severelytheoretical.wordpress.com/2018/02/08/holographic-reduced-representations-and-associative-lstms/", "published_at": "2018-02-08T23:07:59+00:00" }, { "id": "01a0cb2c-f4f5-7194-99a8-d3d6aba67596", "title": "Optimal synaptic memory consolidation", "url": "https://severelytheoretical.wordpress.com/2018/01/14/optimal-synaptic-memory-consolidation/", "published_at": "2018-01-14T00:10:34+00:00" }, { "id": "01a0cb2c-f4f5-7194-99a8-d3d6abef60af", "title": "Why is it hard to train deep neural networks? Degeneracy, not vanishing gradients, is the key", "url": "https://severelytheoretical.wordpress.com/2018/01/01/why-is-it-hard-to-train-deep-neural-networks-degeneracy-not-vanishing-gradients-is-the-key/", "published_at": "2018-01-01T06:20:32+00:00" }, { "id": "01a0cb2c-f4f5-7194-99a8-d3d6ac1179d7", "title": "Introduction to hyperbolic geometry and hyperbolic embeddings", "url": "https://severelytheoretical.wordpress.com/2017/11/25/introduction-to-hyperbolic-geometry-and-hyperbolic-embeddings/", "published_at": "2017-11-25T22:58:26+00:00" }, { "id": "01a0cb2c-f4f5-7194-99a8-d3d6ac2e5a75", "title": "A note on fast weights: do they really work as advertised?", "url": "https://severelytheoretical.wordpress.com/2017/10/14/a-note-on-fast-weights-do-they-really-work-as-advertised/", "published_at": "2017-10-14T22:22:13+00:00" }, { "id": "01a0cb2c-f4f5-7194-99a8-d3d6ac3c53ea", "title": "No, information bottleneck (probably) doesn’t open the “black-box” of deep neural networks", "url": "https://severelytheoretical.wordpress.com/2017/09/28/no-information-bottleneck-probably-doesnt-open-the-black-box-of-deep-neural-networks/", "published_at": "2017-09-28T18:57:52+00:00" }, { "id": "01a0cb2c-f4f5-7194-99a8-d3d6ad010c5c", "title": "Elliptic tales 2", "url": "https://severelytheoretical.wordpress.com/2017/04/14/elliptic-tales-2/", "published_at": "2017-04-14T21:03:48+00:00" }, { "id": "01a0cb2d-2ddb-73db-ba08-4805917868eb", "title": "Elliptic tales 1", "url": "https://severelytheoretical.wordpress.com/2017/03/23/elliptic-tales-1/", "published_at": "2017-03-23T20:37:15+00:00" }, { "id": "01a0cb2d-2ddb-73db-ba08-48059214cd6a", "title": "Recurrence as an efficient way to achieve depth in neural networks", "url": "https://severelytheoretical.wordpress.com/2017/01/11/recurrence-as-an-efficient-way-to-achieve-depth-in-neural-networks/", "published_at": "2017-01-11T22:41:22+00:00" }, { "id": "01a0cb2d-2ddb-73db-ba08-480592a1d457", "title": "Backpropagation works fine with random feedback weights", "url": "https://severelytheoretical.wordpress.com/2016/11/21/backpropagation-works-fine-with-random-feedback-weights/", "published_at": "2016-11-21T02:25:34+00:00" }, { "id": "01a0cb2d-2ddb-73db-ba08-4805931a4a79", "title": "Logic in vector spaces", "url": "https://severelytheoretical.wordpress.com/2016/10/01/logic-in-vector-spaces/", "published_at": "2016-10-01T00:54:39+00:00" }, { "id": "01a0cb2d-2ddb-73db-ba08-4805931c3acf", "title": "Constraining neural networks", "url": "https://severelytheoretical.wordpress.com/2016/04/11/constraining-neural-networks/", "published_at": "2016-04-11T21:58:39+00:00" }, { "id": "01a0cb2d-2ddb-73db-ba08-480593f43399", "title": "Large deviations", "url": "https://severelytheoretical.wordpress.com/2015/06/17/large-deviations/", "published_at": "2015-06-17T15:51:58+00:00" }, { "id": "01a0cb2d-2ddb-73db-ba08-480593f8ff91", "title": "Impressive use of the determinant", "url": "https://severelytheoretical.wordpress.com/2015/05/22/impressive-use-of-the-determinant/", "published_at": "2015-05-22T02:15:59+00:00" }, { "id": "01a0cb2d-2ddb-73db-ba08-480594a5b8af", "title": "Differential geometry notes 7", "url": "https://severelytheoretical.wordpress.com/2014/05/02/differential-geometry-notes-7/", "published_at": "2014-05-02T22:25:25+00:00" }, { "id": "01a0cb2d-2ddb-73db-ba08-480594a9f44d", "title": "Differential geometry notes 6", "url": "https://severelytheoretical.wordpress.com/2014/04/20/differential-geometry-notes-6/", "published_at": "2014-04-20T01:28:58+00:00" }, { "id": "01a0cb2d-2ddb-73db-ba08-480594d8effa", "title": "Differential geometry notes 5", "url": "https://severelytheoretical.wordpress.com/2014/04/14/differential-geometry-notes-5/", "published_at": "2014-04-14T23:52:26+00:00" }, { "id": "01a0cb2d-75a5-7021-9ad3-22c1bb2cfb92", "title": "Differential geometry notes 4", "url": "https://severelytheoretical.wordpress.com/2014/04/07/differential-geometry-notes-4/", "published_at": "2014-04-07T20:35:55+00:00" }, { "id": "01a0cb2d-75a6-701c-b937-8dd4416ba175", "title": "Differential geometry notes 3", "url": "https://severelytheoretical.wordpress.com/2014/03/25/differential-geometry-notes-3/", "published_at": "2014-03-25T21:30:06+00:00" }, { "id": "01a0cb2d-75a6-701c-b937-8dd441b5e061", "title": "Differential geometry notes 2", "url": "https://severelytheoretical.wordpress.com/2014/03/22/differential-geometry-notes-2/", "published_at": "2014-03-22T18:38:09+00:00" }, { "id": "01a0cb2d-75a6-701c-b937-8dd4422ce9cf", "title": "Differential geometry notes 1", "url": "https://severelytheoretical.wordpress.com/2014/03/20/differential-geometry-notes-1/", "published_at": "2014-03-20T00:58:35+00:00" }, { "id": "01a0cb2d-75a6-701c-b937-8dd442b90b16", "title": "The beauty of circulant matrices (and of Toeplitz matrices, to a lesser extent)", "url": "https://severelytheoretical.wordpress.com/2014/02/17/the-beauty-of-circulant-matrices-and-of-toeplitz-matrices-to-a-lesser-extent/", "published_at": "2014-02-17T02:15:47+00:00" }, { "id": "01a0cb2d-75a6-701c-b937-8dd4431944c1", "title": "Note on the Hopfield model", "url": "https://severelytheoretical.wordpress.com/2014/02/04/note-on-the-hopfield-model/", "published_at": "2014-02-04T17:53:30+00:00" }, { "id": "01a0cb2d-75a6-701c-b937-8dd443483342", "title": "Hierarchical probabilistic inference for experimental psychologists", "url": "https://severelytheoretical.wordpress.com/2014/02/04/hierarchical-probabilistic-inference-for-experimental-psychologists/", "published_at": "2014-02-04T16:19:39+00:00" }, { "id": "01a0cb2d-75a6-701c-b937-8dd444262cd2", "title": "One-dimensional population dynamics in LIP", "url": "https://severelytheoretical.wordpress.com/2013/12/05/one-dimensional-population-dynamics-in-lip/", "published_at": "2013-12-05T22:21:51+00:00" }, { "id": "01a0cb2d-75a6-701c-b937-8dd444d30943", "title": "Difference of covariances method", "url": "https://severelytheoretical.wordpress.com/2013/11/29/difference-of-covariances-method/", "published_at": "2013-11-29T05:14:00+00:00" }, { "id": "01a0cb2d-75a6-701c-b937-8dd445ccc571", "title": "The cost of complexity in adaptation", "url": "https://severelytheoretical.wordpress.com/2013/10/04/the-cost-of-complexity-in-adaptation/", "published_at": "2013-10-04T18:01:58+00:00" }, { "id": "01a0cb2d-ad08-7005-98bc-4ecec8a53b0c", "title": "How long does it take to evolve an eye?", "url": "https://severelytheoretical.wordpress.com/2013/09/26/how-long-does-it-take-to-evolve-an-eye/", "published_at": "2013-09-26T17:45:58+00:00" } ] posts Claim your blog
Back to severelytheoretical.wordpress.com
Blog · corpus.blog/blogs/severelytheoretical.wordpress.com/posts

severelytheoretical.wordpress.com

severelytheoretical.wordpress.com

2026

No, agentic AI will not massively boost productivity in scientific research; here’s how to actually boost scientific productivityoriginal ↗
13 Feb 2026

2025

Continual training of Llama-3.1-8B for 809B tokensoriginal ↗
21 Apr 2025
Language development in children is more like post-training than pre-training in LLMsoriginal ↗
28 Feb 2025

2024

On the entropic brain, trapped priors, and machine learningoriginal ↗
29 Oct 2024
IsoFLOP curves of large language models are extremely flatoriginal ↗
31 Jul 2024
Does Sora understand physics? A few simple observationsoriginal ↗
25 Apr 2024
The “it” of deep learning and convergent evolutionoriginal ↗
28 Feb 2024
Intelligence is a granularity problem (or the reality has a surprising amount of detail, so must intelligence)original ↗
2 Jan 2024

2023

Further thoughts on hallucinations in generative modelsoriginal ↗
30 Oct 2023
GPT-3.5 is surprisingly non-factual about literatureoriginal ↗
5 Oct 2023
FLOPs are all you need: a conjecture about what really makes deep learning workoriginal ↗
14 Aug 2023
Can we get rid of all inductive biases?original ↗
8 Jul 2023
“Peer review” must dieoriginal ↗
30 Jun 2023
Is RLHF really better than supervised finetuning?original ↗
30 Apr 2023
Self-supervised learning may solve all the major problems of cogscioriginal ↗
17 Apr 2023
Thoughts on GPT-4original ↗
10 Apr 2023
A rant on LLaMA: please stop training giant language modelsoriginal ↗
5 Mar 2023

2022

Discussing ethics with ChatGPToriginal ↗
28 Dec 2022
What will it take to achieve human-level sample efficiency in deep learning?original ↗
7 Nov 2022
The value of incremental, cumulative improvements is underestimated in AI/MLoriginal ↗
18 Aug 2022
Thoughts on the new scaling laws for large language modelsoriginal ↗
19 Jul 2022
Emergent problems with “emergent” capabilitiesoriginal ↗
12 Jul 2022
A simple plausibility argument for why scaling is probably not enoughoriginal ↗
5 Jul 2022
Neural networks are actually not very good at memorizingoriginal ↗
11 May 2022
A critique of “Why Greatness Cannot Be Planned”original ↗
8 Feb 2022

2021

Catastrophic forgetting is yet another pet problem rendered obsolete by scaleoriginal ↗
11 Oct 2021
How much “human-like” visual experience do current self-supervised learning algorithms need to achieve human-level object recognition?original ↗
24 Sept 2021
Ditching academic research in AI/MLoriginal ↗
13 Jun 2021
Pinker on why humans are smarter than ratsoriginal ↗
7 Jun 2021
AI research, wise passiveness, and negative capabilityoriginal ↗
27 May 2021
Is compositionality/systematic generalization really a problem for neural networks?original ↗
19 Mar 2021
On the futility of trying to be clever (the bitter lesson redux)original ↗
16 Feb 2021
An optimistic perspective on the human-AI nexusoriginal ↗
11 Feb 2021

2020

Thoughts on Image-GPToriginal ↗
29 Nov 2020
Thoughts on GPT-3original ↗
2 Sept 2020
Deep learning can make more use of available dataoriginal ↗
16 May 2020
In defense of pure learningoriginal ↗
11 Feb 2020

2019

Does the brain have to do deep credit assignment?original ↗
31 Dec 2019
Google’s new paper on large-scale weakly supervised learningoriginal ↗
18 Nov 2019
kin2vec: learning kinship relationsoriginal ↗
10 Oct 2019
The relative value of learning over memorizingoriginal ↗
15 Jun 2019
A conjecture about “how the brain works”original ↗
7 Feb 2019

2018

Surprising things you can learn from lavish data with no supervisionoriginal ↗
31 Dec 2018
Common initialization schemes for recurrent neural networks are likely suboptimaloriginal ↗
13 Sept 2018
Simple inductive biases to make neural networks train faster and generalize better: two case studiesoriginal ↗
23 Aug 2018
Hyperreal numbersoriginal ↗
4 Aug 2018
Why do neural networks generalize well?original ↗
25 Jun 2018
The softmax bottleneck is a special case of a more general phenomenonoriginal ↗
8 Jun 2018
Comments on DeepMind’s grid cell paperoriginal ↗
24 May 2018
A simple cache model for image recognitionoriginal ↗
23 May 2018
Short-horizon bias in meta-learningoriginal ↗
9 Apr 2018
Human priors: you can’t have your cake and eat it toooriginal ↗
14 Mar 2018
Compositional embeddings, againoriginal ↗
26 Feb 2018
Holographic reduced representations and associative LSTMsoriginal ↗
8 Feb 2018
Optimal synaptic memory consolidationoriginal ↗
14 Jan 2018
Why is it hard to train deep neural networks? Degeneracy, not vanishing gradients, is the keyoriginal ↗
1 Jan 2018

2017

Introduction to hyperbolic geometry and hyperbolic embeddingsoriginal ↗
25 Nov 2017
A note on fast weights: do they really work as advertised?original ↗
14 Oct 2017
No, information bottleneck (probably) doesn’t open the “black-box” of deep neural networksoriginal ↗
28 Sept 2017
Elliptic tales 2original ↗
14 Apr 2017
Elliptic tales 1original ↗
23 Mar 2017
Recurrence as an efficient way to achieve depth in neural networksoriginal ↗
11 Jan 2017

2016

Backpropagation works fine with random feedback weightsoriginal ↗
21 Nov 2016
Logic in vector spacesoriginal ↗
1 Oct 2016
Constraining neural networksoriginal ↗
11 Apr 2016

2015

Large deviationsoriginal ↗
17 Jun 2015
Impressive use of the determinantoriginal ↗
22 May 2015

2014

Differential geometry notes 7original ↗
2 May 2014
Differential geometry notes 6original ↗
20 Apr 2014
Differential geometry notes 5original ↗
14 Apr 2014
Differential geometry notes 4original ↗
7 Apr 2014
Differential geometry notes 3original ↗
25 Mar 2014
Differential geometry notes 2original ↗
22 Mar 2014
Differential geometry notes 1original ↗
20 Mar 2014
The beauty of circulant matrices (and of Toeplitz matrices, to a lesser extent)original ↗
17 Feb 2014
Note on the Hopfield modeloriginal ↗
4 Feb 2014
Hierarchical probabilistic inference for experimental psychologistsoriginal ↗
4 Feb 2014

2013

One-dimensional population dynamics in LIPoriginal ↗
5 Dec 2013
Difference of covariances methodoriginal ↗
29 Nov 2013
The cost of complexity in adaptationoriginal ↗
4 Oct 2013
How long does it take to evolve an eye?original ↗
26 Sept 2013