Record · corpus.blog/posts/01a0b689-a373-732f-b76c-286e2fca452f

AgentX - InferenceXv3: Does CUDA Moat Hold up in Agentic Inferencing?

newsletter.semianalysis.com · published 24 August 2026

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at newsletter.semianalysis.com Wayback capture

2blogs citing

First cited: 24 Aug 2026
Most recent: 8 Sept 2026
Rank this month: not ranked
Links from this post: 285
Words captured: 16,443

Cited by

Blog

In the post

Date

vLLM Blog

vLLM x AgentX: Optimizing for Real-World Agentic Serving Read at the source ↗

8 Sept 2026

NVIDIA

Up to 30x More Work Per Watt: NVIDIA Vera Rubin NVL72 Sets a New Efficiency Standard for AI Agents Read at the source ↗

24 Aug 2026

Links from this post

Link

Host

https://newsletter.semianalysis.com/p/claude-code-is-the-inflection-point

newsletter.semianalysis.com

https://inferencex.semianalysis.com/

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!EW3z!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F864512bf-1feb-445d-a03a-bbf9f4f1b791_1928x1234.png

substackcdn.com

https://openai.com/signals/enterprise-data/

openai.com

https://github.com/SemiAnalysisAI/InferenceX

github.com

https://substackcdn.com/image/fetch/$s_!PGMD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8a065d23-de1d-4e2f-a3ae-7af724409c9d_1350x653.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!rfhK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F193e605e-579b-4ed2-9637-5bfdbee5cf71_1286x1515.png

substackcdn.com

https://inferencex.semianalysis.com/inference/agentic/440193?i_seq=agentic-traces&i_xmode=interactivity&view=logs

inferencex.semianalysis.com

http://recipes.vllm.ai

recipes.vllm.ai

https://docs.sglang.io/cookbook/intro

docs.sglang.io

https://inferencex.semianalysis.com/quotes

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!4ptj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe97525a9-216a-4624-91c6-373ee7a3850e_1123x437.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!t24X!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72b64a02-8a1f-4a19-929c-85c0ed4803cd_2048x909.png

substackcdn.com

https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126

huggingface.co

https://github.com/alibaba-edu/qwen-bailian-usagetraces-anon

github.com

https://github.com/ai-dynamo/aiperf

github.com

https://semianalysis.com/datacenter-industry-model/

semianalysis.com

https://substackcdn.com/image/fetch/$s_!JDdn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8337140-b620-476a-abd3-59d6cc447d30_2048x1256.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!FVP4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb5aee8b4-d11a-48d8-823c-1ca53412e689_2048x1253.png

substackcdn.com

https://inferencex.semianalysis.com/inference?g_rundate=2026-08-21&i_seq=agentic-traces&g_runid=32433563482&i_xmode=interactivity&i_label=0&i_e2e_xmetric=

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!nim3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd96697d9-bf24-4093-a7db-16d27a52aa08_2048x1200.png

substackcdn.com

https://github.com/sgl-project/sglang/blob/v0.5.17/python/sglang/srt/server_args.py

github.com

https://substackcdn.com/image/fetch/$s_!hqQ7!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81475383-510e-4823-9373-751b8a3722e3_2048x1202.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!EsdQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6254047b-173b-4b1b-9f43-2057eb720b05_2048x1200.png

substackcdn.com

https://inferencex.semianalysis.com/inference?g_rundate=2026-08-21&g_runid=32433563482&i_seq=agentic-traces&i_xmode=interactivity&i_best=0&i_active=mi355x_atom%2Cmi355x_mori-sglang&i_vendor=AMD&i_linelabel=0&i_metric=y_tpPerGpu

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!xGRQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe8d915c0-c962-4277-b62c-a1b08b8e7179_1924x1262.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!_s4G!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F360a051c-6117-4860-a704-b50198f01521_1978x1246.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!6j5N!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9ec4708c-148e-42ee-a7c5-3bf4d9cbdfaa_1938x1246.png

substackcdn.com

https://github.com/vllm-project/vllm/issues/52911

github.com

https://substackcdn.com/image/fetch/$s_!Y8fs!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9302da60-5bc1-46e1-9392-74e912b5c200_2024x1236.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!g7xN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96f9c9f9-eb86-4f87-8332-c85f23040836_2048x916.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!9DNn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff79e53f5-c8ff-4ed0-a8a3-c05b9e0b4bab_2048x1191.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!EH7B!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F512c6225-5816-493e-839d-e4f539e5b1c4_2048x1215.png

substackcdn.com

https://inferencex.semianalysis.com/inference?g_rundate=2026-08-21&g_runid=32433563482&i_seq=agentic-traces&i_metric=y_tpPerGpu&i_xmode=interactivity&i_linelabel=0&i_best=0&i_active=gb200_dynamo-vllm%2Cgb300_dynamo-trt

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!Tnx1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939b9197-321d-4869-ba13-f0b800343b7b_2048x1300.png

substackcdn.com

https://inferencex.semianalysis.com/inference?g_rundate=2026-08-21&g_runid=32433563482&i_seq=agentic-traces&i_xmode=interactivity&i_best=0&i_active=b200_vllm%2Cb300_vllm

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!Z3WJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7ff2d61-bdab-4c2f-96a0-ec5a52469489_2048x884.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!VIDn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff86888c4-c1a7-47b0-a580-d1a74876cb9c_2048x898.png

substackcdn.com

https://inferencex.semianalysis.com/inference/agentic/439522

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!pmXE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61956823-2928-44ec-b00c-8bb5581b8a13_2048x892.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!gbwY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70a7a7c7-7918-459c-a4e0-f7fc1edca816_2048x887.png

substackcdn.com

https://inferencex.semianalysis.com/inference/agentic/439967

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!CM8u!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F770b205f-12a1-4d34-81a7-fb235742d005_2048x1305.png

substackcdn.com

https://inferencex.semianalysis.com/inference?g_rundate=2026-08-21&g_runid=32433563482&i_seq=agentic-traces&i_xmode=interactivity&i_best=0&i_active=b200_sglang%2Ch200_dynamo-sglang%2Cmi355x_sglang

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!iOYr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91dfd9ca-d13d-43e9-a57e-3be0fcc4a5cf_2048x1315.png

substackcdn.com

https://inferencex.semianalysis.com/inference?g_rundate=2026-08-21&g_runid=32433563482&i_seq=agentic-traces&i_xmode=ttft&i_best=0&i_active=h200_dynamo-sglang&i_optimal=0

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!Mcaw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba6532b3-aa46-4477-be84-8979b1cce35d_2048x1297.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!H-lN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F29cfd340-d61b-46c4-8d87-e081d937b80c_2048x1303.png

substackcdn.com

https://inferencex.semianalysis.com/inference?g_rundate=2026-08-21&g_runid=32433563482&i_seq=agentic-traces&i_xmode=ttft&i_best=0&i_active=b200_vllm%2Cb300_vllm%2Cmi355x_atom%2Cmi355x_vllm&i_label=0

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!W5x3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1440922-e1ea-4a74-ab62-f1263ae31ca0_2048x1473.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!xHtW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce2b58fc-b7b7-46c2-ae05-baa14419d233_2048x1459.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!lDUZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80eb810f-a9ea-4f01-a41b-e19e3409ce74_2048x1459.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!rIV5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73ded698-c86e-411b-8e7c-f9bb125487b7_2048x1459.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!kn-R!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27a3f80e-7959-4651-90eb-f498989908a3_1934x1284.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!eFzt!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5396c3d-1823-4818-83de-d89cd8242451_2048x1286.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!SVVs!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5179bd8e-c147-46c8-aeb4-b347049b28a7_2048x1303.png

substackcdn.com

https://inferencex.semianalysis.com/inference?g_model=MiniMax-M3&i_seq=agentic-traces&i_xmode=ttft

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!TM89!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faec34e63-34fd-4c42-87a1-97e5c14661ed_2048x1302.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!TCJq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b40624a-223c-44db-a403-0be563b2af82_2048x1298.png

substackcdn.com

https://inferencex.semianalysis.com/inference?g_model=MiniMax-M3&i_seq=agentic-traces&i_xmode=interactivity&i_best=0&i_hc=1&i_active=mi355x_vllm

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!9ESj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68f7ad68-3886-4980-9b01-390582728e47_2048x1292.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!mD8-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F729199d9-66c0-42b5-b1b8-f228503c1585_2048x1289.png

substackcdn.com

https://inferencex.semianalysis.com/inference?g_model=MiniMax-M3&i_seq=agentic-traces&i_xmode=ttft&i_best=0&i_active=b200_trt%2Cb200_vllm%2Cb300_trt%2Cb300_vllm&i_hc=1&i_label=0

inferencex.semianalysis.com

https://inferencex.semianalysis.com/agentx/cc-traces-weka-062126-256k

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!Tgnw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc92da639-dcff-4e39-b9b5-49742677e93c_2048x1289.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!4h-m!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F519a7cd0-ac51-4a5c-9c45-ba7c41f6d058_2048x1300.png

substackcdn.com

https://inferencex.semianalysis.com/inference?g_model=Qwen-3.5-397B-A17B&i_seq=agentic-traces&i_xmode=ttft&i_best=0&i_hc=1

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!9bPR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc2cf6c7-aa58-4a76-a0a2-ee0457d9243c_2048x1298.png

substackcdn.com

https://inferencex.semianalysis.com/inference?g_model=Qwen-3.5-397B-A17B&i_seq=agentic-traces&i_xmode=interactivity&i_hc=1&i_best=0&i_prec=fp4%2Cfp8&i_active=b300_sglang%2Ch100_sglang%2Ch200_sglang

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!5PHJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff85bdb45-792c-4330-9745-674b38b03425_2048x1289.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!10XC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a21d140-b588-4b33-941d-7c8090ac42b5_2048x1286.png

substackcdn.com

https://inferencex.semianalysis.com/inference?g_model=GLM-5.2&i_seq=agentic-traces&i_xmode=ttft&i_best=0&i_prec=fp4&i_active=b200_sglang%2Cb300_sglang%2Cgb200_dynamo-sglang%2Cgb300_dynamo-trt%2Cmi355x_atom%2Cmi355x_sglang

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!rzqP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9337ed5b-a320-49f7-b31b-cc4b5751e0ff_2048x1290.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!vo0y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e472a49-d228-4d8d-876a-6bc0d59f9d97_746x594.png

substackcdn.com

https://github.com/vllm-project/router/blob/main/src/policies/consistent_hash.rs

github.com

https://substackcdn.com/image/fetch/$s_!T1Z9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58ac4a3a-cc05-4ff6-9d1e-b28dc0d0f760_1814x2048.png

substackcdn.com

https://github.com/vllm-project/router

github.com

https://github.com/llm-d/llm-d-router

github.com

https://github.com/sgl-project/sglang/tree/main/sgl-model-gateway

github.com

https://github.com/ROCm/ATOM/tree/main/atom/mesh

github.com

https://pypi.org/project/mooncake-transfer-engine/

pypi.org

https://github.com/kvcache-ai/Mooncake/tree/main/mooncake-transfer-engine

github.com

https://substackcdn.com/image/fetch/$s_!ZyoZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf5c4776-221b-43b4-b2fd-d46e46f1ab38_1772x960.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!8e51!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e2801db-53ff-4029-8b55-01b3c1a7037d_1632x1140.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!o8e4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa9544ced-f418-44e0-8910-0cbc63fa26ec_2048x768.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!jsOJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F512c3a96-2ab4-452f-be6a-4fa60900575c_2048x651.png

substackcdn.com

https://docs.vllm.ai/en/latest/design/attention_backends/

docs.vllm.ai

https://github.com/vllm-project/vllm/pull/43447

github.com

https://github.com/vllm-project/vllm/pull/44774

github.com

https://github.com/vllm-project/vllm/pull/45444

github.com

https://github.com/vllm-project/vllm/pull/42258

github.com

https://github.com/vllm-project/vllm/pull/44082

github.com

https://substackcdn.com/image/fetch/$s_!CVRv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf1a505d-b8c3-45f9-b03c-a8798fba54c0_2048x632.png

substackcdn.com

https://github.com/vllm-project/vllm/pull/37160

github.com

https://github.com/vllm-project/vllm/pull/40549

github.com

https://github.com/vllm-project/vllm/pull/42296

github.com

https://github.com/vllm-project/vllm/pull/42828

github.com

https://github.com/vllm-project/vllm/pull/51052

github.com

https://github.com/vllm-project/vllm/pull/41289

github.com

https://github.com/vllm-project/vllm/pull/46412

github.com

https://github.com/vllm-project/vllm/pull/46906

github.com

https://github.com/vllm-project/vllm/pull/45659

github.com

https://github.com/vllm-project/vllm/pull/45969

github.com

https://github.com/vllm-project/vllm/pull/45971

github.com

https://github.com/vllm-project/vllm/pull/46188

github.com

https://substackcdn.com/image/fetch/$s_!E7LD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c2cfea4-a217-47ad-934b-dbd226bb248c_2048x837.png

substackcdn.com

https://github.com/vllm-project/vllm/pull/44103

github.com

https://github.com/vllm-project/vllm/pull/45371

github.com

https://github.com/vllm-project/vllm/pull/46855

github.com

https://github.com/vllm-project/vllm/pull/45340

github.com

https://github.com/vllm-project/vllm/pull/49069

github.com

https://github.com/vllm-project/vllm/pull/49071

github.com

https://github.com/vllm-project/vllm/pull/51183

github.com

https://github.com/vllm-project/vllm/pull/51714

github.com

https://github.com/vllm-project/vllm/pull/51713

github.com

https://github.com/vllm-project/vllm/pull/52882

github.com

https://github.com/sgl-project/sglang/pull/26907

github.com

https://github.com/sgl-project/sglang/pull/27210

github.com

https://github.com/sgl-project/sglang/pull/29369

github.com

https://github.com/sgl-project/sglang/pull/34565

github.com

https://github.com/sgl-project/sglang/pull/30339

github.com

https://github.com/sgl-project/sglang/pull/29417

github.com

https://github.com/sgl-project/sglang/pull/28534

github.com

https://github.com/sgl-project/sglang/pull/29735

github.com

https://substackcdn.com/image/fetch/$s_!VyzT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F078ad449-7657-4468-9ff9-3480a7999d89_2048x720.png

substackcdn.com

https://github.com/sgl-project/sglang/pull/30255

github.com

https://substackcdn.com/image/fetch/$s_!aKLG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a807d67-383b-4aa9-93af-e8f3a68313f4_855x634.png

substackcdn.com

https://github.com/sgl-project/sglang/pull/30365

github.com

https://substackcdn.com/image/fetch/$s_!50q6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb29e003e-db43-45d4-93be-bac0be04d869_1209x646.png

substackcdn.com

https://github.com/sgl-project/sglang/pull/34888

github.com

https://github.com/sgl-project/sglang/pull/35017

github.com

https://substackcdn.com/image/fetch/$s_!mne_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47ce130f-13f2-4732-8431-17e26184b8c3_1846x806.png

substackcdn.com

https://github.com/sgl-project/sglang/pull/26091

github.com

https://github.com/sgl-project/sglang/pull/26245

github.com

https://github.com/sgl-project/sglang/pull/26293

github.com

https://github.com/sgl-project/sglang/pull/26387

github.com

https://github.com/sgl-project/sglang/pull/26579

github.com

https://github.com/sgl-project/sglang/pull/30461

github.com

https://github.com/sgl-project/sglang/pull/30497

github.com

https://github.com/sgl-project/sglang/pull/31294

github.com

https://github.com/sgl-project/sglang/pull/33662

github.com

https://github.com/sgl-project/sglang/pull/32042

github.com

https://github.com/sgl-project/sglang/pull/32196

github.com

https://github.com/sgl-project/sglang/pull/30545

github.com

https://substackcdn.com/image/fetch/$s_!CEcE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88c2d49d-2612-4334-90c2-8a76ab55fd60_1418x1004.png

substackcdn.com

https://github.com/sgl-project/sglang/pull/35070

github.com

https://github.com/sgl-project/sglang/pull/35071

github.com

https://github.com/sgl-project/sglang/pull/30762

github.com

https://github.com/sgl-project/sglang/pull/32368

github.com

https://github.com/sgl-project/sglang/pull/34216

github.com

https://substackcdn.com/image/fetch/$s_!-k_t!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed24f1b9-b591-4674-bc94-06394962d03f_2048x544.png

substackcdn.com

https://github.com/NVIDIA/TensorRT-LLM/pull/17462

github.com

https://substackcdn.com/image/fetch/$s_!BiHQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30a51cf2-e628-4c20-a4a5-5978234ca713_952x541.png

substackcdn.com

https://github.com/NVIDIA/TensorRT-LLM/pull/16231

github.com

https://substackcdn.com/image/fetch/$s_!sqbT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00b459b8-9c4c-4c8c-8659-07e14bf5699f_2048x784.png

substackcdn.com

https://github.com/NVIDIA/TensorRT-LLM/pull/17518

github.com

https://substackcdn.com/image/fetch/$s_!BA9Z!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52bd05ab-914f-4dc5-9f16-a0ea1b495383_959x501.png

substackcdn.com

https://github.com/NVIDIA/TensorRT-LLM/pull/17428

github.com

https://github.com/NVIDIA/TensorRT-LLM/pull/16734

github.com

https://github.com/NVIDIA/TensorRT-LLM/pull/17473

github.com

https://github.com/NVIDIA/TensorRT-LLM/pull/16876

github.com

https://github.com/NVIDIA/TensorRT-LLM/pull/17316

github.com

https://github.com/NVIDIA/TensorRT-LLM/pull/17105

github.com

https://github.com/NVIDIA/TensorRT-LLM/pull/17285

github.com

https://substackcdn.com/image/fetch/$s_!29HP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff48290ac-3361-4b0c-a085-ff84dd3dce51_2048x1347.png

substackcdn.com

https://github.com/NVIDIA/TensorRT-LLM/pull/16279

github.com

https://github.com/NVIDIA/TensorRT-LLM/pull/17278

github.com

https://github.com/NVIDIA/TensorRT-LLM/pull/15727

github.com

https://github.com/NVIDIA/TensorRT-LLM/pull/17526

github.com

https://substackcdn.com/image/fetch/$s_!09sS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe8eeb92e-1aad-4a97-b3c1-28409005809e_2048x769.png

substackcdn.com

https://github.com/ROCm/ATOM/pull/1640

github.com

https://substackcdn.com/image/fetch/$s_!QjVe!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c880335-e1e3-44d9-a123-8e316c9c3808_822x555.png

substackcdn.com

https://github.com/ROCm/ATOM/pull/902

github.com

https://github.com/ROCm/ATOM/pull/939

github.com

https://github.com/ROCm/ATOM/pull/1345

github.com

https://github.com/ROCm/ATOM/pull/1771

github.com

https://github.com/ROCm/ATOM/pull/1318

github.com

https://github.com/ROCm/ATOM/pull/1406

github.com

https://github.com/ROCm/ATOM/pull/1725

github.com

https://github.com/ROCm/ATOM/pull/1807

github.com

https://substackcdn.com/image/fetch/$s_!RVON!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7bb7f74-e5a3-4da3-b954-89905e54c4da_2048x420.png

substackcdn.com

https://github.com/ROCm/ATOM/pull/1737

github.com

https://github.com/ROCm/ATOM/pull/1331

github.com

https://github.com/ROCm/ATOM/pull/1647

github.com

https://github.com/ROCm/ATOM/pull/1220

github.com

https://github.com/ROCm/ATOM/pull/1701

github.com

https://github.com/ROCm/ATOM/pull/1746

github.com

https://github.com/ROCm/ATOM/pull/1911

github.com

https://github.com/ROCm/ATOM/pull/1552

github.com

https://substackcdn.com/image/fetch/$s_!ZHbg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8184b9e6-883b-446c-884b-226b5e9fcca1_2048x549.png

substackcdn.com

https://github.com/ROCm/aiter/pull/3728

github.com

https://github.com/ROCm/aiter/pull/3267

github.com

https://github.com/ROCm/aiter/pull/2893

github.com

https://github.com/ROCm/aiter/pull/4474

github.com

https://github.com/ROCm/aiter/pull/4680

github.com

https://substackcdn.com/image/fetch/$s_!jW3c!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce356c05-883b-4c63-a359-8cfd72dddb63_1408x1046.png

substackcdn.com

https://github.com/ROCm/aiter/pull/3459

github.com

https://github.com/ai-dynamo/dynamo/pull/10540

github.com

https://github.com/ai-dynamo/dynamo/pull/10836

github.com

https://github.com/ai-dynamo/dynamo/pull/11095

github.com

https://substackcdn.com/image/fetch/$s_!J5Un!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7af8faf5-300e-4ee5-bc0c-040b7630e923_1016x671.png

substackcdn.com

https://github.com/ai-dynamo/dynamo/pull/11503

github.com

https://github.com/ai-dynamo/dynamo/pull/11508

github.com

https://github.com/ai-dynamo/dynamo/pull/12329

github.com

https://substackcdn.com/image/fetch/$s_!I5WI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3039950-98e0-4cd0-b7fe-929a120bd7d2_2048x576.png

substackcdn.com

https://github.com/ai-dynamo/dynamo/pull/10521

github.com

https://github.com/ai-dynamo/dynamo/pull/10676

github.com

https://github.com/ai-dynamo/dynamo/pull/11644

github.com

https://github.com/ai-dynamo/dynamo/pull/10645

github.com

https://substackcdn.com/image/fetch/$s_!LUiA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F878dd41d-fc0f-407e-8421-ad05ad9e740c_1087x624.png

substackcdn.com

https://github.com/ai-dynamo/dynamo/pull/12158

github.com

https://github.com/ai-dynamo/dynamo/pull/13447

github.com

https://github.com/ai-dynamo/dynamo/pull/10437

github.com

https://github.com/ai-dynamo/dynamo/pull/11104

github.com

https://github.com/ai-dynamo/dynamo/pull/11539

github.com

https://github.com/ai-dynamo/dynamo/pull/11574

github.com

https://github.com/ai-dynamo/dynamo/pull/11569

github.com

https://github.com/ai-dynamo/dynamo/pull/10433

github.com

https://github.com/ai-dynamo/dynamo/pull/11820

github.com

https://github.com/ai-dynamo/dynamo/pull/12161

github.com

https://github.com/ai-dynamo/dynamo/pull/12999

github.com

https://substackcdn.com/image/fetch/$s_!wxlg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F945da9bc-59ec-4084-9f3a-da58da04b3c5_2048x638.png

substackcdn.com

https://github.com/LMCache/LMCache/pull/3382

github.com

https://github.com/LMCache/LMCache/pull/3635

github.com

https://github.com/LMCache/LMCache/pull/3869

github.com

https://github.com/LMCache/LMCache/pull/3908

github.com

https://github.com/LMCache/LMCache/pull/4524

github.com

https://github.com/LMCache/LMCache/pull/3092

github.com

https://github.com/LMCache/LMCache/pull/3101

github.com

https://github.com/LMCache/LMCache/pull/3843

github.com

https://github.com/LMCache/LMCache/pull/4273

github.com

https://github.com/LMCache/LMCache/pull/4363

github.com

https://substackcdn.com/image/fetch/$s_!qIlB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89c2f79c-a704-4144-a900-ddd69ce306dd_1112x667.png

substackcdn.com

https://github.com/LMCache/LMCache/pull/3561

github.com

https://substackcdn.com/image/fetch/$s_!CRox!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32c734aa-1f52-47af-a98e-334978b4e6fa_1778x1386.png

substackcdn.com

https://github.com/kvcache-ai/Mooncake/pull/2225

github.com

https://github.com/kvcache-ai/Mooncake/pull/3184

github.com

https://substackcdn.com/image/fetch/$s_!IdCM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F12cd94ca-2aa7-4e06-a4bd-3153982234c5_1220x680.png

substackcdn.com

https://github.com/kvcache-ai/Mooncake/pull/3338

github.com

https://newsletter.semianalysis.com/p/can-amd-break-the-cuda-moat-amd-advancing

newsletter.semianalysis.com

https://github.com/vllm-project/vllm/pull/45879

github.com

https://github.com/vllm-project/vllm/pull/45720

github.com

https://github.com/vllm-project/vllm/pull/45546

github.com

https://arxiv.org/abs/2506.02634

arxiv.org

https://arxiv.org/html/2608.09867v1

arxiv.org

https://substackcdn.com/image/fetch/$s_!VqTM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F767f2f58-fa9d-40d3-8711-751d243cae9a_500x487.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!qMxe!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7123b5de-9942-4298-8ccd-1b69b8a71df0_2048x1799.png

substackcdn.com

https://github.com/callanjfox/kv-cache-tester

github.com

https://docs.nvidia.com/aiperf/benchmark-modes/trace-replay-with-mooncake-traces

docs.nvidia.com

https://substackcdn.com/image/fetch/$s_!YCZ3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48045991-20d7-4685-863d-c9aa1d6d3142_1812x2048.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!-pfO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff22c9c7e-44ba-4ffd-bdc3-58fbbef8f30d_2048x2004.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!X2Hg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4792809-087e-431b-acce-aecdd3855b3b_2048x1390.png

substackcdn.com

https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126-256k

huggingface.co

https://github.com/SemiAnalysisAI/aiperf

github.com

https://substackcdn.com/image/fetch/$s_!N8B_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20662be0-c311-49e9-8e39-0973d8a1865d_2048x1169.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!ASAb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F18adec21-15d9-40cd-af7d-0772da4a9af0_2048x1729.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!PI1j!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d5f01ad-1736-496b-91b4-31672002bf47_1920x2048.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!g27W!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2f465f9-372c-4ee1-b42f-e79fa9fe352b_1677x2048.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!2ADX!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F058757ed-531f-4cfb-a1ad-a06b1499ecbd_1360x612.png

substackcdn.com

https://inferencex.semianalysis.com/inference/agentic/439239

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!yoYG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43184e50-b71a-4a48-a569-98d6906cb259_2048x1167.png

substackcdn.com

https://inferencex.semianalysis.com/inference?g_rundate=2026-08-13&g_runid=31723588108&i_seq=agentic-traces&i_xmode=interactivity&g_model=MiniMax-M3&i_best=0&i_active=b200_vllm

inferencex.semianalysis.com

https://en.wikipedia.org/wiki/Thundering_herd_problem

en.wikipedia.org

https://substackcdn.com/image/fetch/$s_!VgNS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F963dce62-c49b-4deb-afee-e8dd448526c5_2048x1322.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!PFJR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85116f5f-be0d-405b-a5d9-6a9237b44cc3_2048x895.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!b5Nw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffef10e0d-261e-4978-8650-b10304ec04d0_2048x897.png

substackcdn.com

https://platform.claude.com/docs/en/build-with-claude/prompt-caching

platform.claude.com

Open vs Closed Loop: A Benchmarking Crime Read at the source ↗

notpeerreviewed.com

https://newsletter.semianalysis.com/i/188090866/multi-token-prediction-mtp

newsletter.semianalysis.com

https://github.com/SemiAnalysisAI/InferenceX/tree/main/golden_al_distribution

github.com

https://huggingface.co/datasets/nvidia/SPEED-Bench

huggingface.co

https://substackcdn.com/image/fetch/$s_!9hPf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc15f47a8-fcb3-4220-80ee-4b95bfa7b0b0_2048x1376.png

substackcdn.com

https://inferencex.semianalysis.com/inference/agentic/439903

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!hiWr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46159be7-8b5b-49d7-990f-2ce1cf1217d0_1110x687.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!Jfa1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadd2d50c-a0be-4771-a00b-e6b4f1fe8502_2048x1256.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!9zfy!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91abef64-d639-4da8-8d7e-3499be750011_1001x1541.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!VEvb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc8fc6236-52bd-45ff-ac2e-539899c00d87_2048x1097.png

substackcdn.com

https://inferencex.semianalysis.com/datasets

inferencex.semianalysis.com

https://substackcdn.com/image/fetch/$s_!qm5t!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0e991d9-6ab6-43e7-9cc9-93ac9ca5b7bd_2048x879.png

substackcdn.com

https://github.com/SemiAnalysisAI/aiperf/releases

github.com

https://github.com/sgl-project/sglang/issues/27574

github.com

https://substackcdn.com/image/fetch/$s_!AXto!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36899201-da3b-42ec-888f-2443c63980aa_2048x1260.png

substackcdn.com

https://github.com/SemiAnalysisAI/InferenceX-app/issues/new/choose

github.com

https://github.com/anthropics/claude-code/issues/49207

github.com

https://github.com/anthropics/claude-code/issues/66761

github.com