Benchmark

A benchmark is a standardised test used to compare model capability on a defined task. Each one fixes a dataset, a prompting format, and a scoring rule so that different models produce numbers on the same scale. Knowledge benchmarks such as MMLU and GPQA use multiple-choice questions, coding benchmarks like HumanEval and SWE-bench run generated code against tests, and math sets such as GSM8K check final answers. Human-preference rankings work differently: Chatbot Arena collects blind pairwise votes from users and converts them into Elo-style ratings, capturing qualities that fixed tests miss. Benchmarks let a lab show progress and let buyers shortlist models, but they degrade over time. Contamination is the core problem, because once a test set is on the public web it can appear in training data, turning reasoning into recall. Scores are also sensitive to prompt format, sampling settings, and how partial answers are graded, so the same model can post different numbers in two reports. Teams increasingly build private evaluation sets from their own traffic, since leaderboard position rarely predicts performance on a specific product task.