corpus.blog
56,966 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a087c2-4502-73eb-9450-3d74c2c2a595

AI Agent Benchmark Compendium

philschmid.de · published 15 October 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at philschmid.de Wayback capture
2blogs citing
First cited
15 Oct 2025
Most recent
3 Jan 2026
Rank this month
not ranked
Links from this post
131
Words captured
2,370

Cited by

Blog
In the post
Date
simmering.dev
The Reliability Gap: Agent Benchmarks for Enterprise Read at the source ↗
3 Jan 2026
seroter.com
Daily Reading List – October 15, 2025 (#649) Read at the source ↗
15 Oct 2025

Links from this post

Link
Host
https://github.com/philschmid/ai-agent-benchmark-compendium
github.com
https://openreview.net/pdf?id=2GmDdhBdDk
openreview.net
https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard
github.com
https://gorilla.cs.berkeley.edu/leaderboard.html
gorilla.cs.berkeley.edu
https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard
huggingface.co
https://arxiv.org/abs/2307.16789
arxiv.org
https://github.com/OpenBMB/ToolBench
github.com
https://huggingface.co/spaces/qiantong-xu/toolbench-leaderboard
huggingface.co
https://huggingface.co/datasets/Maurus/ToolBench
huggingface.co
https://arxiv.org/abs/2501.10132
arxiv.org
https://github.com/THUDM/ComplexFuncBench
github.com
https://huggingface.co/datasets/zai-org/ComplexFuncBench
huggingface.co
https://arxiv.org/abs/2406.12045
arxiv.org
https://github.com/sierra-research/tau-bench
github.com
https://github.com/ComposioHQ/Composio-Function-Calling-Benchmark
github.com
https://arxiv.org/pdf/2304.08244.pdf
arxiv.org
https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/api-bank
github.com
https://arxiv.org/abs/2412.16516
arxiv.org
https://github.com/MadeAgents/HammerBench
github.com
https://huggingface.co/datasets/MadeAgents/HammerBench
huggingface.co
https://huggingface.co/blog/andthattoo/dpab-a
huggingface.co
https://github.com/nexusflowai/NexusRaven-V2
github.com
https://huggingface.co/spaces/Nexusflow/Nexus_Function_Calling_Leaderboard
huggingface.co
https://arxiv.org/abs/2409.03215
arxiv.org
https://github.com/SalesforceAIResearch/xLAM
github.com
https://arxiv.org/abs/2409.00920
arxiv.org
https://arxiv.org/abs/2508.01780
arxiv.org
https://icip-cas.github.io/LiveMCPBench
icip-cas.github.io
https://docs.google.com/spreadsheets/d/1EXpgXq1VKw5A7l7-N2E9xt3w0eLJ2YPVPT-VrRxKZBw/edit?gid=0
docs.google.com
https://huggingface.co/datasets/ICIP/LiveMCPBench
huggingface.co
https://arxiv.org/pdf/2508.14704.pdf
arxiv.org
https://github.com/SalesforceAIResearch/MCP-Universe
github.com
https://mcp-universe.github.io/
mcp-universe.github.io
https://arxiv.org/abs/2311.12983
arxiv.org
https://huggingface.co/spaces/gaia-benchmark/leaderboard
huggingface.co
https://huggingface.co/datasets/gaia-benchmark/GAIA
huggingface.co
https://arxiv.org/abs/2308.03688
arxiv.org
https://github.com/THUDM/AgentBench
github.com
https://github.com/THUDM/AgentBench?tab=readme-ov-file
github.com
https://arxiv.org/abs/2407.15711
arxiv.org
https://github.com/oriyor/assistantbench
github.com
https://huggingface.co/spaces/AssistantBench/leaderboard
huggingface.co
https://huggingface.co/datasets/AssistantBench/AssistantBench
huggingface.co
https://arxiv.org/abs/2406.19314
arxiv.org
https://github.com/livebench/livebench
github.com
https://livebench.ai/
livebench.ai
https://huggingface.co/datasets/livebench
huggingface.co
https://arxiv.org/abs/2501.14249
arxiv.org
https://github.com/centerforaisafety/hle
github.com
https://lastexam.ai/
lastexam.ai
https://huggingface.co/datasets/cais/hle
huggingface.co
https://arxiv.org/abs/2506.14922
arxiv.org
https://scale.com/leaderboard/fortress
scale.com
https://arxiv.org/abs/2503.03750
arxiv.org
https://github.com/centerforaisafety/mask
github.com
https://scale.com/leaderboard/mask
scale.com
https://cdn.openai.com/papers/simpleqa.pdf
cdn.openai.com
https://github.com/openai/simple-evals
github.com
https://huggingface.co/datasets/basicv8vc/SimpleQA
huggingface.co
https://arxiv.org/abs/2509.07968
arxiv.org
https://www.kaggle.com/benchmarks/deepmind/simpleqa-verified
kaggle.com
https://deepmind.google/discover/blog/facts-grounding-a-new-benchmark-for-evaluating-the-factuality-of-large-language-models/
deepmind.google
https://github.com/google-deepmind/long-form-factuality
github.com
https://www.kaggle.com/facts-leaderboard
kaggle.com
https://galileo.ai/blog/agent-leaderboard
galileo.ai
https://github.com/rungalileo/agent-leaderboard
github.com
https://arxiv.org/abs/2310.06770
arxiv.org
https://github.com/SWE-bench/SWE-bench
github.com
https://swebench.com/
swebench.com
https://www.swebench.com/SWE-bench/guides/datasets/
swebench.com
Introducing SWE-bench Verified Read at the source ↗
openai.com
https://www.swebench.com/
swebench.com
https://t.co/5ArWF1bKmS
t.co
https://github.com/scaleapi/SWE-bench_Pro-os/tree/main?tab=readme-ov-file
github.com
https://scale.com/leaderboard/swe_bench_pro_public
scale.com
https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro
huggingface.co
https://arxiv.org/abs/2403.07974
arxiv.org
https://github.com/LiveCodeBench/LiveCodeBench
github.com
https://www.kaggle.com/benchmarks/open-benchmarks/livecodebench
kaggle.com
https://arxiv.org/pdf/2504.08703
arxiv.org
https://github.com/amazon-science/SWE-PolyBench
github.com
https://amazon-science.github.io/SWE-PolyBench/
amazon-science.github.io
https://github.com/Aider-AI/aider
github.com
https://aider.chat/docs/leaderboards/
aider.chat
https://github.com/Aider-AI/polyglot-benchmark
github.com
https://arxiv.org/pdf/2307.13854.pdf
arxiv.org
https://github.com/web-arena-x/webarena
github.com
https://aclanthology.org/2024.acl-long.50/
aclanthology.org
https://github.com/web-arena-x/visualwebarena
github.com
https://halluminate.ai/blog/benchmark
halluminate.ai
https://github.com/Halluminate/WebBench
github.com
https://huggingface.co/datasets/Halluminate/WebBench
huggingface.co
https://arxiv.org/abs/2401.13919
arxiv.org
https://github.com/MinorJerry/WebVoyager
github.com
https://leaderboard.steel.dev/
leaderboard.steel.dev
https://arxiv.org/abs/2504.12516
arxiv.org
https://github.com/openai/simple-evals/tree/main
github.com
https://arxiv.org/abs/2306.06070
arxiv.org
https://github.com/osu-nlp-group/Mind2Web
github.com
https://huggingface.co/spaces/osunlp/Online_Mind2Web_Leaderboard
huggingface.co
https://huggingface.co/datasets/osunlp/Mind2Web
huggingface.co
https://arxiv.org/abs/2502.18356
arxiv.org
https://github.com/convergence-ai/webgames
github.com
https://webgames.convergence.ai/
webgames.convergence.ai
https://huggingface.co/datasets/convergence-ai/webgames
huggingface.co
https://arxiv.org/abs/2410.06703
arxiv.org
https://github.com/segev-shlomov/ST-WebAgentBench
github.com
https://arxiv.org/abs/2404.07972
arxiv.org
https://os-world.github.io/
os-world.github.io
https://arxiv.org/abs/2505.03570
arxiv.org
https://github.com/agentsea/osuniverse
github.com
https://github.com/huggingface/screensuite
github.com
https://arxiv.org/abs/2407.05291
arxiv.org
https://arxiv.org/abs/2405.14573
arxiv.org
https://github.com/google-research/android_world
github.com
https://arxiv.org/abs/2502.08047
arxiv.org
https://github.com/showlab/WorldGUI
github.com
https://arxiv.org/abs/2506.04135
arxiv.org
https://github.com/showlab/macosworld
github.com
https://arxiv.org/abs/2407.19056
arxiv.org
https://github.com/zlwang-cs/OfficeBench
github.com
https://github.com/EmergenceAI/emergence-benchmarks/blob/main/papers/e-web/e-web-v0.pdf
github.com
https://github.com/EmergenceAI/emergence-benchmarks
github.com
https://github.com/EmergenceAI/emergence-benchmarks/blob/main/benchmarks/e-web/e-web_benchmark_lite.csv
github.com
https://arxiv.org/abs/1912.01734
arxiv.org
https://github.com/askforalfred/alfred
github.com
https://askforalfred.com/leaderboard/leaderboard.html
askforalfred.com
https://arxiv.org/abs/2502.09560
arxiv.org
https://github.com/EmbodiedBench/EmbodiedBench
github.com
https://twitter.com/_philschmid
twitter.com
https://www.linkedin.com/in/philipp-schmid-a6a2bb196/
linkedin.com