GPT, Claude, Llama? How to tell which AI model is best
Beware model-makers marking their own homework
原文：
When Meta, the parent company of Facebook, announced its latest open-
source large language model (LLM) on July 23rd, it claimed that the most
powerful version of Llama 3.1 had “state-of-the-art capabilities that rival the
best closed-source models” such as GPT-4o and Claude 3.5 Sonnet. Meta’s
announcement included a table, showing the scores achieved by these and
other models on a series of popular benchmarks with names such as
MMLU, GSM8K and GPQA.
当脸书的母公司Meta在7月23日公布其最新的开源大型语言模子(LLM)时，它声称最强盛的版本Llama 3.1具有“开始进的本领，可以与GPT-4o和Claude 3.5 Sonnet等最好的闭源模子相媲美”。Meta的公告包括一个表格，显示了这些模子和其他模子在一系列盛行的基准测试中取得的结果，如MMLU，GSM8K和GPQA。
原文：
On MMLU, for example, the most powerful version of Llama 3.1 scored 88.6%,
against 88.7% for GPT-4o and 88.3% for Claude 3.5 Sonnet, rival models
made by OpenAI and Anthropic, two AI startups, respectively. Claude 3.5
Sonnet had itself been unveiled on June 20th, again with a table of
impressive benchmark scores. And on July 24th, the day after Llama 3.1’s
debut, Mistral, a French AI startup, announced Mistral Large 2, its latest LLM,
with—you’ve guessed it—yet another table of benchmarks. Where do such
numbers come from, and can they be trusted?
比方，在MMLU上，最强盛的版本Llama 3.1的得分为88.6%，而GPT-4o的得分为88.7%，Claude 3.5 Sonnet的得分为88.3%，这两个版天职别由两家人工智能初创公司OpenAI和Anthropic制作。6月20日，Claude 3.5 Sonnet也发布了，同样是令人印象深刻的基准分数表。7月24日，在Llama 3.1首次亮相后的第二天，法国人工智能初创公司Mistral公布了其最新的Mistral Large 2，你可能已经猜到了，这是另一个基准表。如许的数字从何而来，是否可信？
原文：
Having accurate, reliable benchmarks for AI models matters, and not just for
the bragging rights of the firms making them. Benchmarks “define and drive
progress”, telling model-makers where they stand and incentivising them to
improve, says Percy Liang of the Institute for Human-Centred Artificial
Intelligence at Stanford University. Benchmarks chart the field’s overall
progress and show how AI systems compare with humans at specific tasks.
They can also help users decide which model to use for a particular job and
identify promising new entrants in the space, says Clémentine Fourrier, a
specialist in evaluating LLMs at Hugging Face, a startup that provides tools for
AI developers.
拥有准确、可靠的人工智能模子基准很重要，不但仅是为了制造这些模子的公司的吹嘘资本。斯坦福大学以人为中心的人工智能研究所的Percy Liang说，基准“定义并推动进步”，告诉模子制作者他们的立场并激励他们改进。基准测试记录了该领域的整体进展，并显示了人工智能系统在特定任务上与人类相比的体现。为人工智能开发人员提供工具的初创公司Hugging Face评估LLM的专家Clémentine Fourrier表示，它们还可以资助用户决定特定工作使用哪种模子，并确定该领域有前程的新进入者。
学习：
bragging rights：暂时的上风；炫耀的权利；吹嘘的资本
chart：记录；追踪（发展、进展）
这里的 “chart” 意思是“记录”或“描绘”。在这种情况下，“chart the field’s overall progress” 意味着记录并描绘AI领域的整体进展情况。
例子：
- Scientists chart the course of a storm to predict its path and potential impact.（科学家记录风暴的路线，以预测其路径和潜伏影响。）
- The book charts the rise and fall of ancient civilizations, detailing their achievements and eventual decline.（这本书记录了古代文明的兴衰，详细形貌了它们的成绩和终极的衰落。）
在上述例子中，“chart” 都表示详细记录和描绘事物的发展过程。
原文：
But, says Dr Fourrier, benchmark scores “should be taken with a pinch of
salt”. Model-makers are, in effect, marking their own homework—and then
using the results to hype their products and talk up their company valuations.
Yet all too often, she says, their grandiose claims fail to match real-world
performance, because existing benchmarks, and the ways they are applied,
are flawed in various ways.
但是，Fourrier博士说，基准分数“不能全信”。现实上，模子制作者正在给给自己的作业打分——然后用结果来宣传他们的产物，举高他们的公司估值。然而，她表示，他们夸大的说法每每与现实天下的体现不符，因为现有的基准以及应用基准的方式存在各种各样的缺陷。
学习：
a pinch of： 一点；一撮；少许
hype：大肆宣传；炒作；过甚其辞
talk up：吹捧；议论起来了, 畅谈,
grandiose：美 [ˈɡrændioʊs] 夸大的；脆而不坚的；不切现实的
are flawed：存在缺陷
这里的 “marking their own homework” 意思是“自己给自己的作业打分”。在这种情况下，它比喻的是模子制作者自己评估他们的AI模子，并用这些评估结果来宣传他们的产物和提高公司估值。
“Talk up” 意思是“吹捧”或“夸大”。在这种情况下，它指的是公司通过夸大其产物或服务的优点来提高其价值或声誉。
例子：
- When students are allowed to mark their own homework, the grades they give themselves might not be accurate.（当学生被答应自己给自己的作业打分时，他们给自己的分数可能不准确。）
- The company talked up its new software, claiming it would revolutionize the industry, but users found it to be full of bugs.（这家公司吹捧其新软件，声称它将彻底改变行业，但用户发现它充满了毛病。）
在这些例子中，“marking their own homework” 表示自我评估可能带有偏见，而 “talk up” 表示夸大某事的优点以提升其声誉或价值。
原文：
One problem with benchmarks such as MMLU (massive multi-task language
understanding) is that they are simply too easy for today’s models. MMLU was
created in 2020 and consists of 15,908 multiple-choice questions, each with
four possible answers, across 57 topics including maths, American history,
science and law. At the time, most language models scored little better than
25% on MMLU, which is what you would get by picking answers at random;
OpenAI’s GPT-3 did best, with a score of 43.9%. But since then, models have
improved, with the best now scoring between 88% and 90%.
MMLU(大规模多任务语言明白)等基准测试的一个问题是，对于本日的模子来说，它们太简单了。MMLU创建于2020年，由15908道选择题组成，每道题有四个可能的答案，涵盖57个主题，包括数学、美国历史、科学和法律。当时，大多数语言模子在MMLU上的得分比25%好不了多少，这是你随机挑选答案得到的结果；OpenAI的GPT-3体现最好，得分为43.9%。但自那以后，模子有所改进，如今最好的得分在88%到90%之间。
学习：
multiple-choice questions：多选题
原文：
This means it is difficult to draw meaningful distinctions from their scores, a
problem known as “saturation” (see chart). “It’s like grading high-school
students on middle-school tests,” says Dr Fourrier. More difficult
benchmarks have been devised—MMLU-Pro has tougher questions and ten
possible answers rather than four. GPQA is like MMLU at PhD level, on selected
science topics; today’s best models tend to score between 50% and 60% on
it. Another benchmark, MuSR (multi-step soft reasoning), tests reasoning
ability using, for example, murder-mystery scenarios. When a person reads
such a story and works out who the killer is, they are combining an
understanding of motivation with language comprehension and logical
deduction. AI models are not so good at this kind of “soft reasoning” over
multiple steps. So far, few models score better than random on MuSR
这意味着很难从他们的分数中得出故意义的区别，这个问题被称为“分数饱和”(见图表)。“这就像在中学考试中给高中生打分一样，”Fourrier博士说。更难的基准已经被计划出来——MMLU-Pro有更难的问题和十个可能的答案，而不是四个。GPQA类似于博士级别的MMLU，针对选定的科学主题；当今最好的模子在这方面的得分每每在50%到60%之间。另一个基准是MuSR(多步软推理),它测试推理本领，好比使用秘密谋杀场景。当一个人读到如许一个故事，并想出谁是凶手时，他们是在把对动机的明白与语言明白和逻辑推理结合起来。人工智能模子不太擅长这种多步调的“软推理”。到目前为止，很少有模子在MuSR上比瞎猜得分更高。
原文：
MMLU also highlights two other problems. One is that the answers in such tests
are sometimes wrong. A study carried out by Aryo Gema of the University
of Edinburgh and colleagues, published in June, found that, of the questions
they sampled, 57% of MMLU’s virology questions and 26% of its logical-fallacy
ones contained errors. Some had no correct answer; others had more than
one. (The researchers cleaned up the MMLU questions to create a new
benchmark, MMLU-Redux.)
MMLU还强调了别的两个问题。一是这类测试的答案偶然是错误的。爱丁堡大学的Aryo Gema和他的同事进行的一项研究在6月发表，研究发现，在他们抽样的问题中，MMLU 57%的病毒学问题和26%的逻辑谬误问题包罗错误。有些问题没有正确答案；其他问题有不止一个答案。(研究人员清理了MMLU问题，创建了一个新的基准，MMLU-Redux。)
学习：
virology：美 [vaɪˈrɑlədʒi] 病毒学
fallacy：谬误；谬论；
原文：
Then there is a deeper issue, known as “contamination”. LLMs are trained
using data from the internet, which may include the exact questions and
answers for MMLU and other benchmarks. Intentionally or not, the models may
be cheating, in short, because they have seen the tests in advance. Indeed,
some model-makers may deliberately train a model with benchmark data to
boost its score. But the score then fails to reflect the model’s true ability.
One way to get around this problem is to create “private” benchmarks for
which the questions are kept secret, or released only in a tightly controlled
manner, to ensure that they are not used for training (GPQA does this). But then
only those with access can independently verify a model’s scores.
另有一个更深层次的问题，叫做“污染”。LLM使用来自互联网的数据进行训练，这些数据可能包括MMLU和其他基准简直切问题和答案。不管是故意照旧偶然，模子们可能都在作弊，简而言之，因为它们已经提前看过测试了。事实上，一些模子制作者可能会故意用基准数据来训练模子，以提高其得分。但是这个分数不能反映模子的真实本领。解决这个问题的一个方法是创建“私有”基准，对这些问题保密，或者以严格控制的方式发布，以确保它们不被用于培训(GPQA就是如许做的)。但是只有那些有权限的人才气独立验证一个模子的分数。
学习：
contamination：美 [kənˌtæmɪˈneɪʃn] 污染；污染物；弄脏
get around：绕过，解决
原文：
To complicate matters further, it turns out that small changes in the way
questions are posed to models can significantly affect their scores. In a
multiple-choice test, asking an AI model to state the answer directly, or to
reply with the letter or number corresponding to the correct answer, can
produce different results. That affects reproducibility and comparability.
让事情变得更复杂的是，事实证实，向模子提问的方式上的微小变化会显著影响它们的分数。在多项选择测试中，让一个AI模子直接陈述答案，或者用正确答案对应的字母或数字来回答，可以产生不同的结果。这会影响可复现性和可比性。
学习：
reproducibility：可复现
原文：
Automated testing systems are now used to test models against benchmarks
in a standardised manner. Dr Liang’s team at Stanford has built one such
system, called HELM (holistic evaluation of language models), which generates
leaderboards showing how a range of models perform on various
benchmarks. Dr Fourrier’s team at Hugging Face uses another such system,
EleutherAI Harness, to generate leaderboards for open-source models. These
leaderboards are more trustworthy than the tables of results provided by
model-makers, because the benchmark scores have been generated in a
consistent way
自动化测试系统如今被用来以标准化的方式对照基准测试模子。斯坦福大学梁博士的团队已经创建了一个如许的系统，称为HELM(语言模子的整体评估)，它可以生成排行榜，显示一系列模子在各种基准上的体现。Fourrier博士在Hugging Face的团队使用另一个如许的系统，EleutherAI Harness，为开源模子生成排行榜。这些排行榜比模子制作者提供的结果表更值得信赖，因为基准分数是以同等的方式生成的
学习：
holistic： 美 [hoʊˈlɪstɪk] 整体的；全面的；
The greatest trick AI ever pulled
原文：
As models gain new skills, new benchmarks are being developed to assess
them. GAIA, for example, tests AI models on real-world problem-solving. (Some
of the answers are kept secret to avoid contamination.) NoCha (novel
challenge), announced in June, is a “long context” benchmark consisting of
1,001 questions about 67 recently published English-language novels. The
answers depend on having read and understood the whole book, which is
supplied to the model as part of the test. Recent novels were chosen because
they are unlikely to have been used as training data. Other benchmarks
assess models’ ability to solve biology problems or their tendency to
hallucinate.
随着模子得到新的技能，新的基准被开发来评估它们。比方，GAIA在现实天下解决问题的过程中测试人工智能模子。(部分答案保密，以免沾染。6月份公布的NoCha(小说挑战)是一个“长语境”基准，由67部最近出版的英语小说的1001个问题组成。答案取决于是否阅读并明白了整本书，这是作为测试的一部分提供给模子的。选择最近的小说是因为它们不太可能被用作训练数据。其他基准评估模子解决生物学问题的本领或产生幻觉的倾向。
原文：
But new benchmarks can be expensive to develop, because they often
require human experts to create a detailed set of questions and answers. One
answer is to use LLMs themselves to develop new benchmarks. Dr Liang is
doing this with a project called AutoBencher, which extracts questions and
answers from source documents and identifies the hardest ones.
但是开发新的基准可能会很昂贵，因为它们通常必要人类专家来创建一组详细的问题和答案。一个答案是使用LLM本身来开发新的基准。梁博士正在通过一个名为AutoBencher的项目进行这项工作，该项目从源文档中提取问题和答案，并识别出最难的问题和答案。
原文：
Anthropic, the startup behind the Claude LLM, has started funding the creation
of benchmarks directly, with a particular emphasis on AI safety. “We are
super-undersupplied on benchmarks for safety,” says Logan Graham, a
researcher at Anthropic. “We are in a dark forest of not knowing what the
models are capable of.” On July 1st the company began inviting proposals
for new benchmarks, and tools for generating them, which it will co-fund,
with a view to making them available to all. This might involve developing
ways to assess a model’s ability to develop cyber-attack tools, say, or its
willingness to provide advice on making chemical or biological weapons.
These benchmarks can then be used to assess the safety of a model before
public release.
Claude LLM背后的初创公司Anthropic已经开始直接资助基准的创建，特别强调人工智能的安全。Anthropic的研究员洛根·格雷厄姆说:“我们在安全基准方面供应严重不足”。"我们处在一片黑暗的森林中，不知道这些模子能做什么."7月1日，该公司开始为新的基准和生成基准的工具征求建议，并共同出资，以期让所有人都能使用。这些基准可以用来在公开发布之前评估模子的安全性。
原文：
Historically, says Dr Graham, AI benchmarks have been devised by
academics. But as AI is commercialised and deployed in a range of fields,
there is a growing need for reliable and specific benchmarks. Startups that
specialise in providing AI benchmarks are starting to appear, he notes. “Our
goal is to pump-prime the market,” he says, to give researchers, regulators
and academics the tools they need to assess the capabilities of AI models,
good and bad. The days of AI labs marking their own homework could soon
be over. ■
格雷厄姆博士说，从历史上看，人工智能基准是由学者计划的。但随着人工智能商业化并部署在一系列领域，对可靠和具体基准的需求越来越多。他指出，专门提供人工智能基准的初创公司开始出现。“我们的目标是刺激市场，”他说，为研究人员、监管者和学者提供他们必要的工具，以评估人工智能模子的本领，无论好坏。人工智能实验室自己批改作业的日子可能很快就要结束了。■
学习：
pump-prime：通过初始投资或资源注入来启动或刺激某个市场
在这里，“pump-prime”指的是通过初始投资或资源注入来启动或刺激某个市场或行业的发展。这个词源自水泵的操作，意味着先用少量的水或力来启动水泵的运作，进而得到更大的水流。在商业或经济领域，这个术语通常用于形貌当局或机构通过初步投资或支持来激活某个市场或行业。
举个例子：
假设当局盼望促进绿色能源产业的发展，他们可能会提供初期的补贴或税收优惠给太阳能或风能公司。这种做法就是一种“pump-priming”，目标是通过初期的支持和激励，来推动整个绿色能源市场的发展和成熟。
在这段话中，Dr. Graham的意思是，通过提供AI基准测试的工具和资源，初期支持研究人员、监管机构和学术界，使他们能够更好地评估AI模子的本领，从而推动整个AI基准测试市场的发展。
后记
2024年8月3日18点25分于上海。
免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。 |
