BenchMIRT: What Are LLM Benchmarks Actually Measuring?

BenchMIRT: What Are LLM Benchmarks Actually Measuring?

As foundation models approach ceiling scores on traditional standardized evaluations, AI researchers face a fundamental measurement crisis: are benchmarks evaluating true generalized reasoning, or merely memorized test distributions?

BenchMIRT introduces Multi-dimensional Item Response Theory (MIRT) to foundation model evaluation, decomposing benchmark questions into granular psychometric latent traits to uncover what benchmarks actually measure.

Key Breakthroughs

1. Psychometric Latent Trait Decomposition

• Item Difficulty & Discrimination Parameters: Evaluates each test item for its statistical power to differentiate between frontier capabilities.
• Contamination & Leakage Detection: Detects synthetic or pre-training data contamination through abnormal response probability curves.

2. Adaptive Testing for Frontier Models

• Computerized Adaptive Testing (CAT): Reduces required test questions by 75% while achieving higher measurement confidence intervals than static thousand-question suites.
• Cross-Family Calibration: Calibrates scores across OpenAI, Anthropic, Google, and open-weights models onto an invariant difficulty scale.

Technical Specifications & Benchmark Overview

Verified Integration & API Usage

pythonCopy
import benchmirt

evaluator = benchmirt.Evaluator(
    models=["claude-3-7-sonnet", "gpt-4o", "gemini-2-0-flash"],
    benchmark="math-500",
    adaptive=True
)

report = evaluator.run()
report.plot_latent_traits()