LiveBench is an open LLM benchmark that uses contamination-free test data and objective scoring

LiveBench, a new benchmark developed by a team from Nvidia, Abacus.ai, New York University, the University of Maryland, and the University of Southern California, addresses "serious limitations" with existing benchmarks. It offers contamination-free test data, which occurs when more models train on the same dataset3. It utilizes frequently-updated questions from recent sources, scoring answers automatically according to objective ground-truth values. The benchmark contains a wide variety of challenging tasks, spanning math, coding, reasoning, language, instruction following, and data analysis.

When compared to other benchmarks like LMSYS's Chatbot Arena and Arena-Hard, LiveBench shows "generally similar" trends, though some models performed "noticeably stronger on one benchmark versus the other." This potentially indicates some downsides of LLM judging. While these benchmarks show which models perform best, the individual LLM scoring differs, and the metric is not an exact apples-to-apples comparison. As LiveBench points out, it could be attributed to unknown factors such as "known bias."

LiveBench is an open-source benchmark that anyone can use and contribute to. The team plans to maintain it by releasing more questions every month and adding more categories and tasks to broaden their ability to evaluate LLMs as their abilities change and adapt.

LiveBench addresses the issue of contamination in test data for LLM evaluations by releasing new questions every month. These questions are sourced from recently released datasets, math competitions, arXiv papers, news articles, and IMDb movie synopses. By using frequently updated questions from recent sources, LiveBench minimizes the potential for test data contamination. Additionally, each question has a verifiable and objective ground-truth answer, allowing for accurate and automatic scoring without needing LLM judges. This approach helps ensure that the benchmark measures the LLM's broad abilities rather than its memorization capacity, providing a more reliable evaluation of LLM performance.