corpus.blog
41554 blogs · 4,756,248 posts Claim your blog
Record · corpus.blog/posts/01a087de-bae4-7045-8bed-ae7f295416e3

Evaluation harnesses list and notes

serhii.net · published 4 June 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at serhii.net Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
Links from this post
45
Words captured
4,204

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://serhii.net/dtb/days/250317-1847-current-llm-evaluation-landscape/
serhii.net
https://serhii.net/dtb/days/250505-1019-evaluating-rag/
serhii.net
https://serhii.net/dtb/days/250415-1640-evaluating-nlg/
serhii.net
https://github.com/EleutherAI/lm-evaluation-harness
github.com
https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/leaderboard/README.md
github.com
https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md
github.com
https://github.com/EvolvingLMMs-Lab/lmms-eval
github.com
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/tasks/gqa/gqa.yaml
github.com
https://evalverse.gitbook.io/evalverse-docs
evalverse.gitbook.io
https://evalverse.gitbook.io/evalverse-docs/documents/evaluation
evalverse.gitbook.io
https://github.com/openai/evals
github.com
https://github.com/openai/evals/blob/main/docs/eval-templates.md
github.com
https://github.com/openai/evals/blob/main/docs/completion-fns.md
github.com
https://huggingface.co/docs/lighteval/index
huggingface.co
https://huggingface.co/docs/lighteval/adding-a-custom-task
huggingface.co
https://github.com/huggingface/lighteval/blob/main/community_tasks/_template.py
github.com
https://github.com/huggingface/lighteval/blob/main/community_tasks/german_rag_evals.py
github.com
https://github.com/huggingface/lighteval/tree/main/src/lighteval/tasks
github.com
https://github.com/huggingface/lighteval/blob/main/examples/custom_tasks_templates/custom_yourbench_task_mcq.py
github.com
https://github.com/huggingface/lighteval/tree/main/examples/model_configs
github.com
https://github.com/huggingface/lighteval/blob/main/examples/model_configs/litellm_model.yaml
github.com
https://github.com/huggingface/lighteval/blob/53df85985d3239b59c02922e83c0c099fd0b0ddf/src/lighteval/tasks/default_prompts.py
github.com
https://github.com/stanford-crfm/helm?tab=readme-ov-file
github.com
https://crfm.stanford.edu/helm/
crfm.stanford.edu
https://crfm-helm.readthedocs.io/en/latest/
crfm-helm.readthedocs.io
https://crfm-helm.readthedocs.io/en/latest/adding_new_models/
crfm-helm.readthedocs.io
https://crfm-helm.readthedocs.io/en/latest/huggingface_models/
crfm-helm.readthedocs.io
https://github.com/stanford-crfm/helm/blob/main/src/helm/config/model_deployments.yaml
github.com
https://opencompass.readthedocs.io/en/latest/
opencompass.readthedocs.io
https://github.com/open-compass/opencompass
github.com
https://opencompass.readthedocs.io/en/latest/advanced_guides/subjective_evaluation.html
opencompass.readthedocs.io
https://github.com/open-compass/VLMEvalKit
github.com
https://github.com/confident-ai/deepeval
github.com
https://deepeval.com/integrations/frameworks/llamaindex
deepeval.com
https://deepeval.com/integrations/models/vllm
deepeval.com
https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge
github.com
https://github.com/braintrustdata/autoevals
github.com
https://ai.pydantic.dev/evals
ai.pydantic.dev
https://github.com/explodinggradients/ragas
github.com
https://docs.ragas.io/en/latest/references/metrics/
docs.ragas.io
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/examples/models/openai_compatible.sh
github.com
https://github.com/openai/evals/blob/main/docs/build-eval.md
github.com
https://github.com/openai/evals/blob/main/evals/registry/modelgraded/humor.yaml
github.com
https://github.com/openai/evals/blob/main/evals/registry/modelgraded/humor.yaml);
github.com
https://github.com/openai/evals/blob/main/evals/registry/modelgraded/closedqa.yaml
github.com