corpus.blog
42,593 blogs · 5,361,753 posts Claim your blog
Record · corpus.blog/posts/01a07df4-f730-7318-aaea-d7837f6d96fb

Challenges and Opportunities in NLP Benchmarking

Sebastian Ruder · published 23 August 2021

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at ruder.io Wayback capture
2blogs citing
First cited
6 Nov 2021
Most recent
13 May 2024
Rank this month
not ranked
External links
87
Words captured
4,029

Cited by

Blog
In the post
Date
NLP News
The Evolving Landscape of LLM Evaluation Read at the source ↗
13 May 2024
NLP News
Command R+ Read at the source ↗
15 Apr 2024
NLP News
NLP Research in the Era of LLMs Read at the source ↗
19 Dec 2023
serhii.net
LM paper notes Read at the source ↗
18 Nov 2022
NLP News
Moving to Substack ➡️, Scaling Up 📈, Image Generation 🖼 Read at the source ↗
6 Nov 2022
NLP News
Pre-training + Massive Multi-tasking, Benchmarking in NLP, EMNLP primer, 🤗 NLP Course, ACL 2021 recap, Read at the source ↗
6 Nov 2021

Links from this post

Link
Host
https://docs.google.com/presentation/d/1fIhGikFPnb7G5kr58OvYC3GN4io7MznnM0aAgadvJfc/edit?usp=sharing&ref=ruder.io
docs.google.com
https://aiindex.stanford.edu/wp-content/uploads/2021/03/2021-AI-Index-Report_Master.pdf?ref=ruder.io
aiindex.stanford.edu
https://super.gluebenchmark.com/?ref=ruder.io
super.gluebenchmark.com
https://rajpurkar.github.io/SQuAD-explorer/?ref=ruder.io
rajpurkar.github.io
https://github.com/kwchurch/Benchmarking_past_present_future?ref=ruder.io
github.com
https://paperswithcode.com/sota/question-answering-on-squad20?ref=ruder.io
paperswithcode.com
https://youtu.be/t_A36DDcG_0?t=964&ref=ruder.io
youtu.be
https://en.wikipedia.org/wiki/Surveying?ref=ruder.io
en.wikipedia.org
https://en.wikipedia.org/wiki/Level_staff?ref=ruder.io
en.wikipedia.org
https://gluebenchmark.com/?ref=ruder.io
gluebenchmark.com
https://github.com/google-research/xtreme?ref=ruder.io
github.com
https://gem-benchmark.com/?ref=ruder.io
gem-benchmark.com
https://github.com/google/BIG-bench?ref=ruder.io
github.com
https://www.youtube.com/watch?t=964&v=t_A36DDcG_0&feature=youtu.be&ref=ruder.io
youtube.com
https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology?ref=ruder.io
deepmind.com
https://www.springer.com/gp/book/9783030417048?ref=ruder.io
springer.com
https://en.wikipedia.org/wiki/Benchmark_(computing
en.wikipedia.org
https://en.wikipedia.org/wiki/Standard_Performance_Evaluation_Corporation?ref=ruder.io
en.wikipedia.org
https://en.wikipedia.org/wiki/Geometric_mean?ref=ruder.io
en.wikipedia.org
https://en.wikipedia.org/wiki/Instructions_per_second?ref=ruder.io
en.wikipedia.org
https://mlcommons.org/en/?ref=ruder.io
mlcommons.org
https://mlcommons.org/en/training-normal-10/?ref=ruder.io
mlcommons.org
https://github.com/baidu-research/DeepBench?ref=ruder.io
github.com
https://dawn.cs.stanford.edu/benchmark/?ref=ruder.io
dawn.cs.stanford.edu
https://en.wikipedia.org/wiki/DARPA?ref=ruder.io
en.wikipedia.org
https://en.wikipedia.org/wiki/National_Institute_of_Standards_and_Technology?ref=ruder.io
en.wikipedia.org
https://en.wikipedia.org/wiki/TIMIT?ref=ruder.io
en.wikipedia.org
https://catalog.ldc.upenn.edu/LDC97S62?ref=ruder.io
catalog.ldc.upenn.edu
https://en.wikipedia.org/wiki/MNIST_database?ref=ruder.io
en.wikipedia.org
https://en.wikipedia.org/wiki/Text_Retrieval_Conference?ref=ruder.io
en.wikipedia.org
https://en.wikipedia.org/wiki/Cranfield_experiments?ref=ruder.io
en.wikipedia.org
https://googleblog.blogspot.com/2008/03/why-data-matters.html?ref=ruder.io
googleblog.blogspot.com
https://www.nist.gov/video/coopetition-ir-research-presented-ellen-voorhees?ref=ruder.io
nist.gov
https://en.wikipedia.org/wiki/ImageNet?ref=ruder.io
en.wikipedia.org
https://aclanthology.org/D15-1075/?ref=ruder.io
aclanthology.org
https://twitter.com/OriolVinyalsML/status/1333436710303264772?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1333436710303264772%7Ctwgr%5E%7Ctwcon%5Es1_c10&ref_url=https%3A%2F%2Fsyncedreview.com%2F2020%2F11%2F30%2Fbiologys-imagenet-moment-deepmind-says-its-alphafold-has-cracked-a-50-year-old-biology-challenge%2F&ref=ruder.io
twitter.com
https://thegradient.pub/nlp-imagenet/?ref=ruder.io
thegradient.pub
http://nlpprogress.com/english/part-of-speech_tagging.html?ref=ruder.io
nlpprogress.com
http://nlpprogress.com/english/dependency_parsing.html?ref=ruder.io
nlpprogress.com
https://research.google/pubs/pub47761/?ref=ruder.io
research.google
https://wilds.stanford.edu/?ref=ruder.io
wilds.stanford.edu
https://aclanthology.org/2021.naacl-main.324.pdf?ref=ruder.io
aclanthology.org
https://aclanthology.org/N18-2017.pdf?ref=ruder.io
aclanthology.org
https://aclanthology.org/D17-1215/?ref=ruder.io
aclanthology.org
https://aclanthology.org/2020.acl-main.441/?ref=ruder.io
aclanthology.org
https://aclanthology.org/2020.tacl-1.43.pdf?ref=ruder.io
aclanthology.org
https://dynabench.org/?ref=ruder.io
dynabench.org
https://en.wikipedia.org/wiki/F-score?ref=ruder.io
en.wikipedia.org
https://youtu.be/t_A36DDcG_0?t=1824&ref=ruder.io
youtu.be
https://en.wikipedia.org/wiki/Word_error_rate?ref=ruder.io
en.wikipedia.org
https://youtu.be/a-ukPup8gKw?t=770&ref=ruder.io
youtu.be
https://aclanthology.org/2021.acl-long.566.pdf?ref=ruder.io
aclanthology.org
https://openreview.net/forum?id=SkeHuCVFDr&ref=ruder.io
openreview.net
https://aclanthology.org/2021.acl-short.5.pdf?ref=ruder.io
aclanthology.org
https://github.com/GEM-benchmark/GEM-metrics?ref=ruder.io
github.com
https://aclanthology.org/2021.gem-1.10.pdf?ref=ruder.io
aclanthology.org
https://youtu.be/t_A36DDcG_0?t=278&ref=ruder.io
youtu.be
https://youtu.be/Jb6G25ZJ-VU?t=307&ref=ruder.io
youtu.be
https://aclanthology.org/D18-1514/?ref=ruder.io
aclanthology.org
https://youtu.be/Jb6G25ZJ-VU?t=657&ref=ruder.io
youtu.be
https://arxiv.org/abs/2104.08481?ref=ruder.io
arxiv.org
https://ai.stanford.edu/~ang/papers/acl11-WordVectorsSentimentAnalysis.pdf?ref=ruder.io
ai.stanford.edu
https://youtu.be/t_A36DDcG_0?t=661&ref=ruder.io
youtu.be
https://aclanthology.org/2021.naacl-main.385.pdf?ref=ruder.io
aclanthology.org
https://aclanthology.org/2020.acl-main.560/?ref=ruder.io
aclanthology.org
https://nlpprogress.com/?ref=ruder.io
nlpprogress.com
https://aclanthology.org/2021.findings-acl.413/?ref=ruder.io
aclanthology.org
https://youtu.be/FXCSWvIsdEE?t=1226&ref=ruder.io
youtu.be
https://aclanthology.org/2020.emnlp-main.393/?ref=ruder.io
aclanthology.org
https://arxiv.org/abs/2106.15590?ref=ruder.io
arxiv.org
https://en.wikipedia.org/wiki/FLOPS?ref=ruder.io
en.wikipedia.org
https://arxiv.org/abs/2101.00133?ref=ruder.io
arxiv.org
https://arxiv.org/abs/2102.07033?ref=ruder.io
arxiv.org
http://explainaboard.nlpedia.ai/?ref=ruder.io
explainaboard.nlpedia.ai
https://aclanthology.org/2021.acl-demo.34/?ref=ruder.io
aclanthology.org
https://aclanthology.org/2020.acl-main.442/?ref=ruder.io
aclanthology.org
http://explainaboard.nlpedia.ai/leaderboard/task-ner/index.php?ref=ruder.io
explainaboard.nlpedia.ai
https://aclanthology.org/2021.acl-demo.34.pdf?ref=ruder.io
aclanthology.org
https://dynabench.org/tasks/nli?ref=ruder.io
dynabench.org
https://aclanthology.org/2021.findings-acl.334.pdf?ref=ruder.io
aclanthology.org
https://aclanthology.org/2021.acl-long.242/?ref=ruder.io
aclanthology.org
https://aclanthology.org/P14-2083.pdf?ref=ruder.io
aclanthology.org
https://aclanthology.org/P19-1449.pdf?ref=ruder.io
aclanthology.org
https://sites.research.google/xtreme/?ref=ruder.io
sites.research.google
https://arxiv.org/abs/2102.01672?ref=ruder.io
arxiv.org
https://www.luge.ai/?ref=ruder.io
luge.ai
https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/keywords_to_tasks.md?ref=ruder.io
github.com
Elsewhere on ruder.io (2)