When Benchmarks Age: Temporal Misalignment through Large Language Model Factuality Evaluation

Summary

This paper investigates whether the tests we use to check if large language models (LLMs) are telling the truth are still relevant and accurate, given how quickly both the models and the real world change.

What's the problem?

Currently, we rely on existing benchmarks – sets of questions and facts – to evaluate how truthfully LLMs respond. However, these benchmarks were created at a specific point in time and don't necessarily reflect current facts. The paper points out that because the world is constantly changing, and LLMs are rapidly improving, these older benchmarks might be giving us a misleading picture of how well LLMs *actually* perform. No one has systematically studied how much these benchmarks have 'aged' and how that affects our ability to trust their results.

What's the solution?

The researchers looked at five popular factuality benchmarks and tested eight different LLMs, including some older and some newer versions. They created a system to automatically check if the facts within the benchmarks were still up-to-date. They then used this system, along with specific measurements, to determine how 'old' the benchmarks were and how that affected the scores the LLMs received. Essentially, they wanted to see if LLMs were being penalized for knowing *current* information that contradicted outdated benchmark facts.

Why it matters?

This work is important because it shows that many of the benchmarks we use to assess LLM factuality are becoming unreliable. If we continue to use outdated tests, we might underestimate the capabilities of newer models or overestimate the accuracy of older ones. The research provides a way to test benchmarks for 'age' and encourages the development of more current and reliable evaluation methods, ultimately leading to a better understanding of how truthfully LLMs are performing.