Record · corpus.blog/posts/01a0fab0-76ee-71a6-bcb7-2592e368ccf7

LLM Evaluation Harness Settings Behind a Benchmark Score - Prompt Format, Few-Shot Examples, Answer Extraction, Agent Sandboxes, and What to Disclose Alongside the Number | hidekazu-konishi.com

hidekazu-konishi.com · published 1 October 2026

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at hidekazu-konishi.com Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
External links: 46
Words captured: 14,773

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

https://github.com/laude-institute/harbor

github.com

https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.13

github.com

https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.13/README.md

github.com

https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.13/docs/interface.md

github.com

https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.13/docs/task_guide.md

github.com

https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.13/docs/new_task_guide.md

github.com

https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.13/docs/model_guide.md

github.com

https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.13/docs/chat-template-readme.md

github.com

https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.13/docs/footguns.md

github.com

https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.13/docs/API_guide.md

github.com

https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.13/lm_eval/tasks/arc/arc_easy.yaml

github.com

https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.13/lm_eval/tasks/gsm8k/gsm8k.yaml

github.com

https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.13/lm_eval/tasks/gpqa/generative/_gpqa_generative_n_shot_yaml

github.com

https://inspect.aisi.org.uk/

inspect.aisi.org.uk

https://inspect.aisi.org.uk/tasks.html

inspect.aisi.org.uk

https://inspect.aisi.org.uk/solvers.html

inspect.aisi.org.uk

https://inspect.aisi.org.uk/scorers.html

inspect.aisi.org.uk

https://inspect.aisi.org.uk/scoring-policy.html

inspect.aisi.org.uk

https://inspect.aisi.org.uk/metrics.html

inspect.aisi.org.uk

https://inspect.aisi.org.uk/sandboxing.html

inspect.aisi.org.uk

https://inspect.aisi.org.uk/setting-limits.html

inspect.aisi.org.uk

https://inspect.aisi.org.uk/options.html

inspect.aisi.org.uk

https://inspect.aisi.org.uk/eval-logs.html

inspect.aisi.org.uk

https://inspect.aisi.org.uk/reference/inspect_ai.log.html

inspect.aisi.org.uk

https://inspect.aisi.org.uk/scoring-workflow.html

inspect.aisi.org.uk

https://inspect.aisi.org.uk/CHANGELOG.html

inspect.aisi.org.uk

https://pypi.org/project/swebench/5.0.2/

pypi.org

https://github.com/SWE-bench/SWE-bench/blob/main/README.md

github.com

https://github.com/SWE-bench/SWE-bench/blob/main/docs/guides/evaluation.md

github.com

https://github.com/SWE-bench/SWE-bench/blob/main/docs/faq.md

github.com

https://docs.harborframework.com/

docs.harborframework.com

https://docs.harborframework.com/core-concepts

docs.harborframework.com

https://docs.harborframework.com/getting-started/quick-start

docs.harborframework.com

https://docs.harborframework.com/core-concepts/jobs/run-a-job

docs.harborframework.com

https://docs.harborframework.com/core-concepts/tasks/configuration

docs.harborframework.com

https://docs.harborframework.com/core-concepts/tasks/network-policies

docs.harborframework.com

https://docs.harborframework.com/core-concepts/tasks/verifier

docs.harborframework.com

https://docs.harborframework.com/core-concepts/jobs/configs

docs.harborframework.com

https://docs.harborframework.com/core-concepts/datasets/metrics

docs.harborframework.com

https://docs.harborframework.com/core-concepts/datasets/registries

docs.harborframework.com

https://docs.harborframework.com/core-concepts/results/view-job-results

docs.harborframework.com

https://docs.harborframework.com/changelog

docs.harborframework.com

https://pypi.org/project/harbor/0.23.0/

pypi.org

https://crfm-helm.readthedocs.io/en/latest/maintenance_mode/

crfm-helm.readthedocs.io

https://github.com/openai/simple-evals

github.com

https://github.com/harbor-framework/terminal-bench-1

github.com

Elsewhere on hidekazu-konishi.com (11)

LLM Benchmark History and Timeline - Who Built Each Yardstick, Who Declared It Saturated, and What Replaced It | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Reproducible LLM Inference - Batch Invariance, the Cost of Determinism, and Where Bitwise Identity Is Actually Required | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

LLMOps Observability and Evaluation Architecture on AWS - Tracing, Metrics, and Automated Evaluation Gates with CloudWatch and OpenTelemetry | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Amazon Bedrock Model Evaluation Practical Guide - Automatic Metrics, LLM-as-a-Judge, RAG Evaluation, and CI/CD Quality Gates | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Amazon Bedrock AgentCore Evaluations Practical Guide - Built-In Evaluators and CI/CD Regression Testing for AI Agents | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Anthropic Claude Model Migration Guide - Upgrading Prompts and Workloads Across Model Generations | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

LLM API Parameter Compatibility Reference - Anthropic, OpenAI, Google Gemini, and Amazon Bedrock | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Agent Sandboxing and Blast-Radius Isolation on AWS - Choosing the Unit of Each Containment Boundary and What Each One Does Not Stop | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Publishing npm and PyPI Packages Without Long-Lived Tokens - Trusted Publishing, Staged Publishing, and What Trusted Publishing Does Not Assert | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

EC2 Image Builder - Recipes, Workflows, Distribution, and Lifecycle Policies, and What the Image Resource Records About How an AMI Was Built | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Hidekazu Konishi - Profile / Biography | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com