The aggregate view
Leaderboard
Environments, benchmarks, and model rankings. One place to explore the catalog.
Snapshot 2026-11-05 · Model scores retain their source rankings. Environments and benchmarks are catalogs, not scored rankings.
RL environment companies and startups
Dedicated view ↗| Tier | Company | Funding | Domains |
|---|---|---|---|
| Confirmed 81Each placed by a line from its own site or paper. These are on the map. | |||
| S | Scale AI Sells expert-sourced training data and evaluations to frontier model builders, and builds and operates AI… | $15BY Combinator S16 | Coding Computer Use+3 |
| S | Snorkel AI Sells specialized expert training data, RL environments and benchmarks/evals to frontier AI labs and AI teams. | $570M | Coding Computer Use+3 |
| S | Prime Intellect Sells an open-source stack for agentic RL: an Environments Hub, hosted evaluations, hosted RL post-training,… | $150M | Coding Enterprise Workflows+2 |
| S | AfterQuery Sells expert-generated AI training data (SFT, RLHF, rubric-based RL), tool-calling and computer-use RL… | $30MY Combinator W25 | Multi-Domain Coding+2 |
| S | Modal Sells serverless cloud infrastructure for AI: on-demand GPU compute, multi-node clusters, inference serving… | $111M | Coding |
| S | Mercor Runs a network of professional experts that supplies human data, evals, benchmarks and RL environments to… | $450M | Multi-Domain Coding+2 |
| S | Turing Supplies frontier AI labs with expert human data, RL environments, datasets and benchmarks, and provides… | $225M | Coding Enterprise Workflows+1 |
| S | micro1 Sells expert human data, RL environments (Realm), contextual agent evaluations (Cortex) and real-world… | $35M | Multi-Domain Enterprise Workflows |
| S | Vals AI Is an independent evaluator that benchmarks AI models and applications on professional tasks and frontier… | $40M | |
| S | Handshake Supplies human expert data and agentic training environments, graders and benchmarks to AI labs, drawing on… | Multi-Domain | |
| S | Surge AI Sells expert human data (RLHF, SFT, rubrics, human evaluation) plus custom and off-the-shelf RL environments… | Enterprise Workflows Multi-domain+12 | |
| S | Toloka Sells expert-sourced human data, custom datasets, RL-gym environments and evaluations for training and… | $72M | Multi-Domain |
| S | Uber AI Solutions sells human data services — data collection, annotation, testing and evaluation from a global… | Audio | |
| A | Gray Swan AI Sells AI security products: Shade automated red teaming of models and agents, Cygnal runtime guardrails that… | $50M | |
| A | Daytona Sells secure, elastic sandbox infrastructure where AI agents run AI-generated code in isolated, stateful… | $31M | Coding Computer Use |
| A | Fleet AI Builds high-fidelity simulated training environments ('gyms') and evals that model real-world work for… | Computer Use Enterprise Workflows+1 | |
| A | Deeptune Acquired by MercorBuilds RL training gyms for frontier labs: simulated environments mimicking software like Slack and Salesforce… | $48.3M | Coding Computer Use+2 |
| A | E2B Sells isolated Firecracker microVM sandboxes for AI agents, offered as a hosted cloud API or deployed in the… | $32.5MPioneer | Coding Computer Use |
| A | Bespoke Labs Builds company-scale RL environments and infrastructure, plus agent evaluation and GEPA-based optimization,… | $40M | Coding Long-Horizon |
| A | Halluminate Builds RL environments and benchmarks for AI agents doing knowledge work, starting with financial services due… | $38.5MY Combinator S25 | Finance Enterprise Workflows+1 |
| A | Patronus AI Builds RL tasks with expert reference solutions and graders, plus ready-built and custom simulated… | $70M | Multi-Domain Coding |
| A | Revelo Sells expert-authored coding data, RL environments, evals and engineer staffing to AI labs, drawing on its… | $48M | Coding |
| A | Parsewave Sells custom, fully human-authored datasets, solution traces and expert evaluations, written mainly by senior… | Long-Horizon | |
| A | Wirestock Sells licensed and custom multimodal image and video datasets for AI training, sourced from a global network… | $23M | Creative |
| A | Latch Builds verifiable benchmarks for evaluating AI agents on biology and drug-discovery tasks, alongside a data… | $33M | Science |
| B | Proximal Is a research lab that builds complex post-training data and long-horizon coding environments for frontier… | $15M | Coding Long-Horizon+1 |
| B | Mechanize Builds high-fidelity, long-horizon RL environments and evals for frontier coding agents, including the open… | $9.1M | Coding Private Codebases |
| B | Andon Labs Builds long-horizon agent evals such as Vending-Bench and runs real AI-operated businesses as safety testbeds,… | Y Combinator W24 | Computer Use Long-Horizon |
| B | Veris AI Sells a simulation platform that builds stateful twins of an enterprise's systems, data, APIs and users so AI… | $8.5M | Enterprise Workflows |
| B | Collinear Sells Simulation Lab RL environments with simulated users, tools and verifiers, plus off-the-shelf tasks,… | Security Enterprise Workflows+3 | |
| B | Good Start Labs Sells game-based RL environments, human gameplay data, synthetic data and benchmarks to AI labs for training… | $3.6M | Games Long-Horizon |
| B | General Reasoning Is an AI research lab that runs OpenReward, a platform for serving and running RL environments, and also sells… | Finance Coding+1 | |
| B | Chakra Labs Sells deterministic clones of production software (Notion, Salesforce, Gmail, GitHub and more) as GUI and MCP… | $10.1M | Computer Use Tool Use |
| B | Huzzle Labs Builds long-horizon RL environments for coding, computer use and enterprise workflows for frontier AI labs,… | Long-Horizon Coding+3 | |
| B | Idler Builds reinforcement learning environments and evals from real production work for frontier AI labs, spanning… | Y Combinator S25 | Coding Enterprise Workflows+1 |
| B | Datacurve Sells expert-sourced training data for frontier AI labs, including RL environments, long-horizon tasks, agent… | $17.7MY Combinator W24 | Coding Private Codebases |
| B | Runloop Sells secure, VM-isolated cloud sandboxes (devboxes) and managed infrastructure for building, evaluating and… | $7M | Coding |
| B | Pareto Supplies expert human data for training frontier AI models through a global network of specialists, and… | $5.1M | Multi-Domain Tool Use |
| B | Emulated Builds post-training datasets and environments for software engineering and AI R&D, and publishes benchmarks… | ML Science | |
| B | HUD Sells an SDK and hosted platform for building, running, training and evaluating models on RL environments,… | $16MY Combinator W25 | Long-Horizon Coding+2 |
| B | Taste Labs Builds design-focused post-training data and evaluation infrastructure for frontier labs, and sells a Brand… | $18.5MLatitud Fellowship 2024 | Creative |
| B | Contra Labs Sells creative-AI evaluations, benchmarks, expert human preference data, datasets and computer-use… | Creative Coding | |
| B | Aptura Supplies expert human data, curated datasets and RL environments for training and evaluating AI in finance,… | Finance Enterprise Workflows | |
| B | Quesma Sells an analytics product that collects Claude Code, Codex and Cursor session logs to show what coding agents… | $2.5M | Security |
| B | ReasonCore AI sells expert-verified reasoning datasets, sandboxed RL environments and adaptive evals-as-a-service across… | Science Coding | |
| B | Verita AI Runs a community of PhD-level experts and consultants who provide personalized human preference data, tasks… | Creative | |
| B | ARIMLABS Builds RLVR training environments and benchmarks for long-horizon AI agents in cybersecurity, SRE, compilation… | Security Long-Horizon | |
| B | Incalmo Builds high-fidelity, verifiable RL environments for training AI models on realistic cybersecurity tasks,… | Security | |
| B | Preference Model Builds hardened RL environments with robust graders for frontier labs to train capable, better-aligned models,… | $16M | ML Coding |
| B | Symbal Sells evals, benchmarks and post-training datasets built with an expert labor pool to frontier labs, and… | Coding Enterprise Workflows | |
| B | dmodel D_model is an AI research lab that builds RL environments for open-ended interpretability and alignment… | $500KY Combinator S24 | ML Alignment |
| C | Refresh Builds RL environments and evals with verifiable rewards for coding, computer use (cloned software suites like… | Y Combinator X25 | Coding Computer Use+1 |
| C | Normal Is an applied research lab selling data and evals, such as CAD benchmarks and datasets, to advance AI… | Y Combinator S25 | Chip Design |
| C | Hillclimb Builds and sells RL environments and aggregated human research data to frontier labs to improve models' AI… | Y Combinator F25 | Math |
| C | BenchFlow Is an open-source frontier environment lab that builds RL environments and agent benchmarks (SkillsBench,… | Enterprise Workflows Browser+2 | |
| C | Vmax Is an RL research lab building Campaign, self-play task generation infrastructure that turns a customer's… | ML Coding+1 | |
| C | Cua Sells computer-use agent infrastructure — background desktop drivers, local and cloud sandboxes and elastic… | Y Combinator X25 | Coding Computer Use |
| C | Markov Sells computer-use datasets and CAD environments built from recorded human workflows (screen recordings,… | Y Combinator S26 | Computer Use Tool Use |
| C | Plato Builds simulated replicas of real software (CRMs, GitHub, Jira, Stripe, Linux desktops) as scored environments… | Browser Enterprise Workflows+1 | |
| C | Abundant Builds RL environments and datasets for AI labs and enterprises, with a focus on coding and physical sciences. | $500KY Combinator F24 | Coding |
| C | Aviro Builds reinforcement learning environments and post-training datasets for long-horizon coding, computer use,… | Y Combinator X25 | Coding ML |
| C | Ulam Sells research-grade mathematical reasoning datasets, private benchmarks and evaluations, and verifier-backed… | Math Science | |
| C | Ooak Data Turns real, anonymized company data into digital-twin RL environments and evaluation infrastructure for… | Y Combinator S26 | Enterprise Workflows |
| C | Theta Builds custom RL environments from client and expert data to train specialized agents for frontier labs and… | Y Combinator X25 | Enterprise Workflows |
| C | Third Moment Research sells fully managed, OpenAI API-compatible inference endpoints that it continually improves by… | Coding Finance | |
| C | Vetto AI Is a post-training data research partner for frontier labs, providing RL environments, rubrics and verifiers,… | Coding Computer Use | |
| C | Andromede Is an RL data lab that programmatically generates RL environments, tasks and verifiers from real-world data… | Long-Horizon | |
| C | EdotEnv Builds multi-step RL environments and benchmarks from financial market data for frontier labs to evaluate and… | Y Combinator S26 | Long-Horizon ML |
| C | Rise Data Labs Sells human data services, supplying US-based expert trainers and annotation tools for labeling, RLHF and… | Enterprise Workflows Finance+5 | |
| D | Morph Sells Morph Cloud, snapshot-and-branch VM infrastructure (Infinibranch devboxes and instances) for running AI… | Coding | |
| D | AIChamp Sells non-invasive factory loss checks and digital twins of manufacturing stations in which vendor-neutral… | Enterprise Workflows Long-Horizon | |
| D | Habitat Inc Builds verifiable coding RL tasks from real open-source commits using a paid network of engineers, and now… | Coding Computer Use+1 | |
| D | Metaphi AI builds training environments to make AI agents reliable for enterprise workloads, and offers a terminal… | Coding Enterprise Workflows | |
| D | pre.dev Sells a coding agent for building and shipping software, and also sells long-horizon, verified RL coding tasks… | Coding Long-Horizon | |
| D | Tacit Labs Builds data, long-horizon evaluations and RL research environments for training frontier models on biology,… | Science Long-Horizon | |
| D | Align AI Sells a runtime verification layer for AI agents, along with domain-specific RL environments, a ground-truth… | Enterprise Workflows Coding | |
| D | Anthromind Provides expert-curated, fully managed data for post-training, RLHF and custom evaluations of AI models. | Healthcare Long-Horizon | |
| D | Diffuse Labs Builds reinforcement learning environments, generated from merged PRs, agent traces and CI failures, that… | ML Long-Horizon | |
| D | Dissei Sells finance-focused AI training data, reference trajectories, sealed evaluations and graders, and packaged… | Finance | |
| D | Exabite Sells off-the-shelf and custom-scraped pre-training, post-training and RL datasets to frontier AI labs. | Coding | |
| D | Trajectory Labs Provides expert red teaming, safety evaluations and RL environments to frontier AI labs, connecting top red… | Alignment | |
| Not yet confirmed 5Listed in the catalog; no line from its own site or paper confirms it yet. | |||
| A | Originator | Computer Use Coding | |
| B | Phinity | $5.2MPearX S25 | Chip Design |
| B | Matrices | AI Grant Batch 3 | Computer Use |
| C | Sepal AI | Y Combinator S24 | Enterprise Workflows Long-Horizon Math |
| D | Ionic | Finance Law | |
No entries match. Clear the filter above to see all 86.
AI benchmarks for LLMs and agents
Dedicated view ↗| Tier | Benchmark | Domains |
|---|---|---|
| S | Humanity's Last ExamHumanity's Last Exam measures frontier LLMs' expert-level academic knowledge and reasoning with 2,500… | Coding Computer Use+2 |
| S | Terminal-Bench 2.1Terminal-Bench 2.1 measures a model's ability to navigate and complete difficult tasks in a sandboxed terminal… | Agentic |
| S | SWE-bench VerifiedSWE-bench Verified measures whether AI models can resolve real-world GitHub issues by generating patches that… | Coding |
| S | AIME 2025AIME 2025 measures olympiad-level mathematical reasoning on all 30 problems from the 2025 American… | Math |
| S | Terminal-Bench HardTerminal-Bench Hard measures AI agents' ability to complete challenging terminal tasks such as software… | Coding |
| S | GPQA DiamondGPQA Diamond measures model accuracy on the 198 hardest graduate-level, Google-proof multiple-choice questions… | Reasoning |
| S | GPQA-D | Knowledge |
| S | MMLU-ProMMLU-Pro measures language models' knowledge and reasoning through over 12,000 ten-option multiple-choice… | Knowledge |
| S | DeepSWE 1.1DeepSWE 1.1 measures models' ability to resolve real-world software engineering tasks, scored by pass rate… | Coding |
| S | ProgramBenchProgramBench measures whether AI agents can rebuild complete programs from scratch given only a compiled… | Coding |
| S | Tau2-Bench TelecomTau2-Bench Telecom measures how well conversational agents guide users through telecom technical support in a… | Agentic Business |
| A | Terminal-Bench 3.0Terminal-Bench 3.0 measures whether AI agents can complete difficult, expert-level tasks in command-line… | Coding Enterprise Workflows+1 |
| A | Terminal-BenchTerminal-Bench measures how well AI agents complete hard, realistic tasks in computer terminal / command-line… | Coding Computer Use+2 |
| A | OSWorld 2.0OSWorld 2.0 measures computer-use agents on 108 long-horizon, real-world desktop workflows across everyday and… | Coding Computer Use+2 |
| A | SWE-bench ProSWE-Bench Pro measures AI agents' ability to resolve long-horizon software engineering tasks (bug fixes and… | Multi-Domain Coding |
| A | AIMEAIME scores models on the American Invitational Mathematics Examination's integer-answer competition math… | Math |
| A | Terminal-Bench 4.0Terminal-Bench 4.0 measures the percentage of practical terminal-based tasks AI agents resolve successfully… | Agentic |
| A | OSWorldOSWorld measures how well multimodal agents complete 369 open-ended real-world computer tasks across desktop… | Agentic |
| A | SWE-benchSWE-bench measures whether language models can resolve real-world GitHub issues by generating patches to… | Coding |
| A | Agents' Last ExamAgents' Last Exam evaluates AI agents on long-horizon, economically valuable real-world professional tasks… | Coding Computer Use+2 |
| A | APEX-AgentsAPEX-Agents measures whether AI agents can complete multi-hour, cross-application professional tasks in… | Coding Enterprise Workflows+1 |
| A | APEXAPEX (AI Productivity Index) is Mercor's family of benchmarks measuring economically valuable professional… | Multi-Domain |
| A | DeepSWEDeepSWE measures frontier coding agents on 113 original, long-horizon software engineering tasks across 91… | Coding |
| A | ARC-AGI-1ARC-AGI-1 measures how efficiently AI systems learn and generalize abstract rules from a few examples on 800… | Reasoning |
| A | ARC-AGI-2ARC-AGI-2 measures fluid intelligence and abstract reasoning in AI systems through novel input-output grid… | Reasoning |
| A | Terminal-Bench-Science 0.1Terminal-Bench-Science 0.1 measures AI agents' resolution rate on 70 expert-curated, verifiable scientific… | Agentic |
| A | LiveCodeBenchLiveCodeBench measures LLM coding ability on continuously collected competitive programming problems from… | Coding |
| A | GPQAGPQA measures model accuracy on 448 extremely difficult, Google-proof graduate-level multiple-choice questions… | Knowledge |
| A | HLE-VerifiedHLE-Verified is a verified and revised version of Humanity's Last Exam that evaluates frontier language models… | Knowledge |
| A | SimpleQASimpleQA measures whether language models can accurately answer short, fact-seeking questions with single… | Knowledge |
| A | HELM GSM8KHELM GSM8K scores language models on grade school math word problems within Stanford CRFM's HELM evaluation… | Math |
| A | HELM MMLUHELM's MMLU leaderboard scores language models by exact-match accuracy on the Massive Multitask Language… | Knowledge |
| A | MCP AtlasMCP Atlas measures language models' tool-use competency on tasks run against real Model Context Protocol (MCP)… | Coding Computer Use+2 |
| A | AutomationBenchAutomationBench measures how well AI agents complete cross-application business workflows via REST APIs across… | Agentic |
| A | Finance Agent v2Finance Agent v2 measures how well AI agents perform the work of entry-level financial analysts, answering… | Agentic |
| A | CritPtCritPt measures whether LLMs can solve unpublished, research-level frontier physics challenges and checkpoint… | Reasoning |
| A | ChartographyChartography measures whether frontier models can accurately read and reason over 100 specialized professional… | Multimodal |
| A | AgentBench FCAgentBench FC evaluates LLM agents with function-calling prompts on ALFWorld, database, knowledge graph,… | Agentic |
| A | JobBenchJobBench evaluates AI agents on 130 professional workflow tasks across 35 occupations that experts identify as… | Agentic |
| A | GDPvalGDPval measures how well AI models produce real-world, economically valuable knowledge-work deliverables… | Agentic |
| A | SAGESAGE (Student Assessment with Generative Evaluation) measures how accurately AI models grade handwritten… | Multimodal |
| A | MGSMMGSM measures language models' ability to solve grade-school math word problems translated into 10… | Multilingual |
| A | INCLUDEINCLUDE measures multilingual LLM knowledge and reasoning using 197,243 multiple-choice questions from local… | Multilingual |
| A | MATHMATH measures mathematical problem solving of models on 12,500 challenging competition mathematics problems… | Math |
| A | BFCL-V4BFCL V4 measures how accurately LLMs call functions (tools), covering single-turn, multi-turn, and agentic… | Tool Use |
| A | LOCA-BenchLOCA-bench measures how reliably language agents complete tool-using tasks as their context grows to extreme,… | Long Context |
| A | E-BenchE-Bench measures LLM agents' multi-step tool use on 323 synthetic state-changing tasks across Tencent product… | Agentic |
| A | SWE-LancerSWE-Lancer measures whether frontier LLMs can complete over 1,400 real freelance software engineering tasks… | Coding |
| A | RewardBenchRewardBench evaluates reward models on prompt-chosen-rejected trios spanning chat, reasoning, and safety to… | Alignment |
| A | DRACODRACO measures deep research agents on complex, open-ended research tasks across 10 domains, grading outputs… | Agentic |
| A | AA BriefcaseAA-Briefcase measures frontier agentic capability on long-horizon knowledge work, with models completing… | Agentic |
| A | BankerToolBenchBankerToolBench measures how well AI agents complete end-to-end junior investment banking workflows, producing… | Agentic |
| A | BFCL v3BFCL v3 evaluates how accurately large language models call functions (tools), adding multi-turn and… | Agentic |
| A | Finance Agent v1.1Finance Agent v1.1 measures how well AI agents using tools like SEC search and web search perform entry-level… | Agentic |
| A | ALE-BenchALE-Bench evaluates AI systems on score-based algorithmic programming contests drawn from AtCoder Heuristic… | Agentic |
| A | BigCodeBenchBigCodeBench evaluates LLM code generation on practical, real-world programming tasks, with results tracked on… | Coding |
| A | CursorBenchCursorBench scores models on coding-agent tasks such as edits, refactors, bugfixes, codebase understanding and… | Coding |
| A | AA-LCRAA-LCR measures language models' ability to extract, reason about, and synthesize information from long… | Reasoning |
| A | CharXivCharXiv measures how well multimodal LLMs understand and reason about natural, diverse scientific charts drawn… | Multimodal |
| A | BabyVisionBabyVision measures core visual perception and reasoning abilities of multimodal LLMs independent of… | Multimodal |
| A | CorpFin v2CorpFin v2 measures how accurately models comprehend long credit agreements, including questions over very… | Knowledge |
| A | CaseLaw v2CaseLaw v2 measures how accurately language models answer questions over private, recent Canadian court case… | Knowledge |
| A | GDPval-AAGDPval-AA measures AI models' agentic performance on real-world, economically valuable knowledge-work tasks… | Agentic |
| A | Gert LabsGBENCH by Gert Labs ranks AI models by their performance in complex games, with separate scores for agentic… | Agentic |
| A | AndroidWorldAndroidWorld measures how well autonomous agents complete 116 parameterized tasks across 20 real-world Android… | Agentic |
| A | PinchBenchPinchBench measures how well LLMs perform as the model inside an OpenClaw agent, scoring successful completion… | Agentic |
| A | QwenClawBenchQwenClawBench measures how well OpenClaw agents complete 100 realistic real-user-distribution tasks across… | Agentic |
| A | Harvey's Legal Agent BenchmarkHarvey's Legal Agent Benchmark measures how well AI agents complete agentic legal work using files, scored by… | Agentic |
| A | Vending-Bench 2Vending-Bench 2 measures how well AI models can run a simulated vending machine business over a year, scored… | Agentic |
| A | Tau3-BankingTau3-Banking measures whether conversational agents can retrieve policies from a ~700-document banking… | Agentic |
| A | SWE MultilingualSWE-bench Multilingual measures how well LLM agents resolve real-world GitHub issues across 300 tasks in 42… | Coding |
| A | MLS-Bench LiteMLS-Bench Lite is the official 30-task subset of MLS-Bench, scoring whether AI systems can invent… | Coding |
| A | PaperBenchPaperBench measures AI agents' ability to replicate state-of-the-art AI research papers from scratch,… | Coding |
| A | IOIIOI measures models' ability to solve International Olympiad in Informatics competitive programming problems… | Coding |
| A | MuSRMuSR evaluates language models on multistep soft reasoning tasks posed as long natural-language narratives… | Reasoning |
| A | LisanBenchLisanBench measures LLMs' rule-following, vocabulary knowledge, planning, recall and persistence by scoring… | Reasoning |
| A | GeneBench-ProGeneBench-Pro measures whether AI agents can carry out realistic multi-stage statistical analyses of messy… | Reasoning |
| A | WildBenchWildBench evaluates LLMs on about 1,024 challenging real-user tasks, scoring responses with checklist-guided… | Reasoning |
| A | BenchCAD Vision2CodeBenchCAD Vision2Code measures whether models can generate CadQuery programs from rendered views of mechanical… | Multimodal |
| A | MMMU-ProMMMU-Pro measures multimodal models' expert-level understanding and reasoning over college-level questions… | Multimodal |
| A | OmniDocBench 1.5OmniDocBench 1.5 evaluates how well models parse diverse real-world PDF documents, scoring end-to-end parsing,… | Multimodal |
| A | Video-MMEVideo-MME measures how well multimodal large language models understand and analyze videos. | Multimodal |
| A | MathVisionMathVision (MATH-V) measures large multimodal models' mathematical reasoning on 3,040 competition math… | Multimodal |
| A | ScreenSpot ProScreenSpot-Pro measures how accurately multimodal models locate target GUI elements in high-resolution… | Multimodal |
| A | ZeroBenchZeroBench measures large multimodal models' visual reasoning on a small set of adversarially filtered image… | Multimodal |
| A | Visual-Language UnderstandingVISTA (Visual-Language Understanding) measures multimodal models' complex visual-language understanding on… | Multimodal |
| A | MortgageTaxMortgageTax measures how accurately multimodal models extract semantic fields and calculate annualized tax… | Multimodal |
| A | MathVistaMathVista measures mathematical reasoning of foundation models in visual contexts, such as figures, geometry… | Multimodal |
| A | HealthBenchHealthBench measures how well AI models respond in 5,000 realistic multi-turn, multilingual health… | Knowledge |
| A | HealthBench ProfessionalHealthBench Professional scores language models on real clinician tasks (care consult, writing and… | Knowledge |
| A | SpatialBench VerifiedAnalysis of spatial transcriptomics data across externally validated biological problems. | Knowledge |
| A | SingleCellBenchLatchBio's bioinformatics evaluations measure a model's ability to analyze spatial transcriptomics… | Knowledge |
| A | ProteinGym HardProteinGym Hard is a hard subset of the ProteinGym protein engineering benchmark, scoring how well models… | Knowledge |
| A | LABBench2LABBench2 measures AI systems' real-world capabilities in performing biology research tasks, with nearly 1,900… | Knowledge |
| A | HealthBench HardHealthBench Hard is a 1,000-example subset of HealthBench measuring model performance on difficult health… | Knowledge |
| A | MMMLUMMMLU measures multilingual general knowledge by testing models on the MMLU test set professionally translated… | Knowledge |
| A | C-EvalC-Eval measures the knowledge and reasoning of foundation models in Chinese through 13,948 multiple-choice… | Knowledge |
| A | MedCodeMedCode measures whether AI models can assign correct primary and secondary ICD-10-CM diagnosis codes from… | Knowledge |
| A | MedScribeMedScribe measures whether AI models can write accurate, compliant SOAP clinical notes from patient… | Knowledge |
| A | LegalBenchLegalBench measures large language models' legal reasoning across 162 expert-crafted tasks covering six types… | Knowledge |
| A | TaxEval v2TaxEval v2 measures models' accuracy and stepwise reasoning when answering hard tax-related questions. | Knowledge |
| A | MedQAMedQA measures how well models answer multiple-choice medical questions drawn from professional medical board… | Knowledge |
| A | MILUMILU measures LLM knowledge and language understanding across 11 Indic languages with about 80,000… | Multilingual |
| A | IFBenchIFBench measures how well language models generalize precise instruction following to 58 new, diverse,… | Instruction Following |
| A | WritingBenchWritingBench measures large language models' generative writing ability across 6 core writing domains and 100… | Instruction Following |
| A | IMO 2026IMO 2026 measures a model's ability to write rigorous proofs for the six problems of the 2026 International… | Math |
| A | RiemannBenchRiemannBench measures models on research-level mathematics problems with unique, programmatically verified… | Math |
| A | MATH-500MATH-500 scores models' accuracy on 500 competition math problems covering algebra, probability, trigonometry… | Math |
| A | Apex ShortlistApex-Shortlist is a mathematical reasoning benchmark that scores models with and without tools, reported in… | Math |
| A | USAMO 2026USAMO 2026 evaluates models on proof-based problems from the 2026 USA Mathematical Olympiad. | Math |
| A | ProofBenchProofBench measures whether models can construct Lean 4 formal proofs, accepted by the proof checker, for… | Math |
| A | AA-OmniscienceAA-Omniscience measures factual recall and knowledge calibration across 6,000 questions in 42 economically… | Factuality |
| A | AI Energy ScoreAI Energy Score measures and compares the energy efficiency of AI models across common tasks like text… | Efficiency |
| A | AlpacaEvalAlpacaEval measures how well language models follow general user instructions, using GPT-4-based… | Generalization |
| A | Berkeley Function-Calling LeaderboardBFCL measures how accurately LLMs call functions and tools, including single-turn, multi-turn, relevance… | Agentic |
| A | BFCL_v3_MultiTurnBFCL V3 MultiTurn measures how accurately large language models handle multi-turn, multi-step function (tool)… | Tool Use |
| A | BoolQBoolQ measures models' accuracy at answering yes/no questions about passages, scored by exact match on the… | Knowledge |
| A | CL-benchCL-bench measures language models' ability to learn new knowledge, rule systems, procedures and empirical laws… | Long Context |
| A | ClawEval-MMClawEval-MM is a multimodal agent benchmark, listed under Visual Agent & Coding, that scores models on… | Agentic |
| A | CodeEditorBenchCodeEditorBench measures LLMs' code editing ability across debugging, code translation, code polishing and… | Coding |
| A | NVIDIA ComputeEvalComputeEval measures how well LLMs generate correct and performant CUDA and CUDA Python code across kernels,… | Coding |
| A | Context ArenaContext Arena is a long-context leaderboard that scores models on retrieval and reasoning across increasing… | Reasoning |
| A | Creative Writing v3An LLM-judged benchmark that scores models on creative writing quality using rubric scores and Elo ratings. | Writing |
| A | CursorBench 3.2CursorBench 3.2 is a version of Cursor's agentic coding benchmark that scores models on coding tasks such as… | Coding |
| A | DAXBenchDAXBench measures how well language models write correct, efficient DAX code for Power BI… | Data Business |
| A | Design ArenaDesign Arena is a crowdsourced benchmark where users vote on model-generated designs (websites, web apps,… | Multimodal |
| A | Ego3D-BenchEgo3D-Bench evaluates models' understanding of driving scenes, including dynamic scenes, traffic participants… | Multimodal |
| A | EQ-BenchEQ-Bench measures language models' emotional and social intelligence, currently via multi-turn roleplay chats… | Generalization |
| A | FutureEvalFutureEval measures how accurately AI models and agents forecast future real-world outcomes on Metaculus… | Reasoning Knowledge |
| A | GQAGQA measures compositional visual reasoning and question answering over real-world images, scoring accuracy,… | Knowledge |
| A | Gray Swan IPI — Combined Attack SetMeasures a model's robustness to indirect prompt-injection attacks using Gray Swan's combined attack set,… | Safety Cybersecurity |
| A | HELMHELM is a holistic leaderboard that scores language models across many scenarios such as question answering,… | Knowledge |
| A | HELM AIR-BenchAIR-Bench 2024 measures language model safety refusal across 314 risk categories derived from government… | Generalization |
| A | HELM LiteHELM Lite is a leaderboard that scores language models on ten core scenarios spanning question answering,… | Knowledge |
| A | HELM SafetyHELM Safety is a Stanford CRFM leaderboard that evaluates recent language models on safety-related benchmarks… | Generalization |
| A | HieroglyphBenchHieroglyphBench measures how well vision-language models transcribe photographed ancient Egyptian hieroglyph… | Vision |
| A | JSONSchemaBenchJSONSchemaBench measures how efficiently and completely structured-output engines and LLMs generate JSON that… | Structured Output |
| A | LawBenchLawBench measures how well large language models memorize, understand and apply Chinese legal knowledge across… | Legal |
| A | Matbench DiscoveryMatbench Discovery evaluates machine-learning energy models (e.g. universal interatomic potentials) on… | Materials |
| A | MathArena ApexMathArena Apex measures whether AI models can solve a small set of extremely hard final-answer math… | Math |
| A | MedmarksMedmarks is an open medical AI benchmark suite and leaderboard that scores language models on verifiable and… | Healthcare |
| A | MixEval ChatMixEval Chat reports chat-model scores on MixEval and MixEval-Hard, ground-truth-based dynamic benchmark… | General Knowledge |
| A | MRCR-v2 128kMRCR-v2 128k measures long-context reasoning by asking models to reproduce a specific instance of repeated… | Long Context |
| A | MRCR v2 256KMRCR v2 256K measures long-context reasoning by asking models to reproduce a specific instance of assistant… | Long Context |
| A | $OneMillion-Bench$OneMillion-Bench evaluates language agents on 400 bilingual expert-curated, economically consequential… | Professional |
| A | Open Catalyst OC20OC20 scores machine learning models on predicting catalyst-adsorbate energies and relaxed structures across… | Materials |
| A | Open Catalyst OC22OC22 is a public leaderboard scoring machine learning models on predicting relaxed energies and forces for… | Materials |
| A | ProteinGymProteinGym measures how well models predict protein fitness effects of substitutions on deep mutational… | Science |
| A | ReXrankReXrank is a public leaderboard scoring models on chest X-ray interpretation, covering radiology report… | Healthcare |
| A | Rule2DRCRule2DRC measures whether LLM agents can synthesize correct, executable KLayout DRC scripts from… | Coding |
| A | RuneBenchRuneBench measures how well AI coding agents play RuneScape by writing TypeScript against an emulated game… | Agentic |
| A | ScienceQAScienceQA measures multimodal multiple-choice science question answering with explanations across natural,… | Multimodal |
| A | SEED-BenchSEED-Bench measures generative comprehension of multimodal LLMs using 19K human-annotated multiple-choice… | Knowledge |
| A | SQuAD 2.0SQuAD 2.0 measures reading comprehension: answering questions from Wikipedia passages with text spans and… | Knowledge |
| A | SWE-fficiencySWE-fficiency measures whether language model agents can optimize the runtime of real Python repositories on… | Coding |
| A | E-Bench-CodeE-Bench-Code is the code-execution extension of E-Bench, measuring how reliably LLM agents complete… | Coding |
| A | Hy-Backend 2.0Hy-Backend 2.0 is a Tencent-designed internal agentic coding benchmark that scores models on backend software… | Coding |
| A | Hy-FinmodelBench v2Hy-FinmodelBench v2 is a Tencent internal benchmark scoring models on financial-modeling tasks as part of the… | Finance |
| A | Hy-SWE Max VerifiedHy-SWE Max Verified is Tencent Hunyuan's internal 300-task agentic software-engineering benchmark, scored… | Coding |
| A | TextClass BenchmarkTextClass Benchmark continuously evaluates LLMs and transformers on social-science text classification tasks… | Classification |
| A | VCRVCR measures cognition-level visual commonsense reasoning: models answer multiple-choice questions about… | Knowledge |
| A | AA-Omniscience Net ScoreAA-Omniscience Net Score measures factual knowledge reliability as correct answers minus incorrect answers,… | Behavioral |
| A | AgentDriveAgentDrive measures LLM reasoning for autonomous driving through 300,000 simulated driving scenarios and a… | Agentic |
| A | ArxivMathArxivMath measures models' final-answer accuracy on research-level mathematics problems extracted monthly from… | Math |
| A | BBQBBQ measures demographic social bias in question answering, scoring accuracy on ambiguous and disambiguated… | Language-understanding |
| A | BC-VLBC-VL measures multimodal search and knowledge question answering, where models combine visual inputs with… | Multimodal |
| A | Beam128KBeam128K is a 128K-context long-context evaluation reported under general capabilities in Meta's Muse Glimmer… | Long Context |
| A | BenchCAD Vision2Code-1000BenchCAD Vision2Code-1000 measures how well models convert images of 3D CAD parts into CAD code, scored by… | Multimodal |
| A | BioMysteryBench Human-DifficultBioMysteryBench Human-Difficult measures model accuracy on 23 real-world bioinformatics analysis questions… | Science |
| A | BioMysteryBench Human-SolvableMeasures model accuracy on 76 real-world bioinformatics analysis tasks from BioMysteryBench that at least one… | Science |
| A | BioPipelineBench VerifiedBioPipelineBench Verified measures how well models complete biological data-analysis (bioinformatics pipeline)… | Biology |
| A | BixBenchBixBench measures the ability of LLM-based agents to perform real-world bioinformatics data analysis,… | Science |
| A | BRIDGE Medical LeaderboardBRIDGE measures how well large language models understand real-world clinical practice text across 87 tasks in… | Healthcare |
| A | ChartMuseumChartMuseum measures large vision-language models' visual reasoning on chart question answering over 1,162… | Multimodal |
| A | ChartQAProChartQAPro measures how well large vision-language models answer diverse, challenging questions about… | Multimodal |
| A | CharXiv-DCharXiv-D is the descriptive-questions subset of CharXiv, measuring how well multimodal LLMs extract basic… | Multimodal |
| A | CountQACountQA measures how accurately multimodal models answer counting questions about objects in images, as part… | Multimodal |
| A | EvalVerseEvalVerse is an expert-calibrated evaluation framework that scores generative video models on professional… | Video |
| A | FormationEvalFormationEval measures language models' knowledge of petroleum geoscience and subsurface disciplines through… | Science |
| A | Gaia2Gaia2 is an agentic benchmark scoring models on general autonomous agent task completion, reported in Meta's… | Agentic |
| A | GDPval-MMGDPval-MM is MiniMax's internal agentic evaluation based on the open-source GDPval test set, scoring… | Multimodal |
| A | Graphwalks BFS 1M F1Graphwalks BFS 1M F1 measures long-context reasoning by having models perform breadth-first-search over graphs… | Long Context |
| A | Graphwalks BFS 256k F1Graphwalks BFS 256k F1 measures long-context reasoning by having models perform breadth-first search over… | Long Context |
| A | Graphwalks Parents 1M F1Graphwalks Parents 1M F1 measures long-context reasoning by having models find parent nodes in graphs… | Long Context |
| A | Graphwalks Parents 256k F1Graphwalks Parents 256k F1 measures long-context reasoning by having models find parent nodes in graphs… | Long Context |
| A | HAKARI-BenchHAKARI-Bench measures the retrieval quality and efficiency of embedding, sparse, late-interaction, BM25 and… | Retrieval |
| A | HiPhOHiPhO is a multimodal reasoning benchmark scoring models on challenging visual reasoning involving physical… | Multimodal |
| A | Histopath-CHistopath-C measures the robustness of histopathology vision-language models and test-time adaptation methods… | Biomedicine Healthcare |
| A | HMMT 2025HMMT 2025 scores models on final-answer problems from the Harvard-MIT Mathematics Tournament, a prestigious… | Math |
| A | HotpotQAHotpotQA measures multi-hop question answering over Wikipedia paragraphs, scoring both answer accuracy and… | Knowledge |
| A | IMO 2025IMO 2025 scores models on writing proofs for the six 2025 International Mathematical Olympiad problems, graded… | Math |
| A | JP-TL-BenchJP-TL-Bench measures bidirectional Japanese-English translation quality using anchored pairwise LLM-judged… | Translation |
| A | Kernel Bench L3Kernel Bench L3 measures GPU kernel optimization ability, reporting median speedup over a PyTorch eager… | Coding |
| A | K-MetBenchK-MetBench measures expert meteorological knowledge, reasoning, Korean geo-cultural understanding and… | Weather |
| A | LABBench2 Clinical TrialsLABBench2 Clinical Trials is a subset of LABBench2, a benchmark measuring AI systems' real-world capabilities… | Biology Healthcare |
| A | LABBench2 Database QuestionsLABBench2 Database Questions (DBQA2) is a multiple-choice subset of LABBench2 that measures models' ability to… | Biology |
| A | LABBench2 FigQALABBench2 FigQA measures models' ability to answer multiple-choice questions about scientific figures in… | Biology |
| A | LABBench2 Literature QuestionsLABBench2 Literature Questions (LitQA3 subset) scores models on multiple-choice questions requiring retrieval… | Biology |
| A | LABBench2 Patent QuestionsA subset of LABBench2 that tests AI systems on patent-related questions as part of measuring real-world… | Biology |
| A | LABBench2 Reading TablesLABBench2 Reading Tables is a subset of LABBench2, a benchmark measuring AI systems' ability to perform… | Biology |
| A | LABBench2 Supplementary MaterialsLABBench2 measures AI systems' real-world capabilities in performing biology research tasks, with nearly 1,900… | Biology |
| A | LatchBio BioinformaticsLatchBio evaluations of real-world bioinformatics problem solving. SpatialBench Verified tests analysis of… | Biology |
| A | Lech Mazur WritingMeasures LLMs' creative short-story writing under constrained briefs with ten required elements, scored via… | Writing |
| A | MATH Level 5MATH Level 5 measures models' ability to solve the hardest tier of challenging competition mathematics… | Math |
| A | MCPMark-VerifiedMCPMark-Verified is a human-verified edition of MCPMark that evaluates how well models use MCP tools across… | Agentic |
| A | MEDIC BenchmarkMEDIC evaluates LLMs across five dimensions of clinical competence, including medical QA, clinical… | Healthcare |
| A | MMBCMMBC is a multimodal search-augmented benchmark testing whether models can combine visual inputs with external… | Multimodal |
| A | MMSearchPlusMMSearchPlus measures multimodal models' ability to answer visual questions by combining image inputs with… | Multimodal |
| A | MOODMOOD measures how well LLM safety monitors detect out-of-distribution alignment failures such as jailbreaking,… | Alignment |
| A | MTEB-BRMTEB-BR measures text embedding model quality on 22 native Brazilian-Portuguese tasks across classification,… | Language |
| A | MultiLexNorm++MultiLexNorm++ measures how well models perform lexical normalization of informal social media text, extending… | Multilingual |
| A | NarrativeQANarrativeQA measures reading comprehension and question answering over long narrative texts such as stories,… | Generalization |
| A | Natural QuestionsNatural Questions measures how well language models answer real open-domain questions, scored by F1 in… | Knowledge |
| A | NeoEvalPlusNNeoEvalPlusN scores LLMs on proprietary tests of command following under distraction, expectation-breaking… | Creative |
| A | NYT Connections ExtendedMeasures LLMs' verbal associative reasoning by scoring how many groups they correctly identify in 940 NYT… | Knowledge |
| A | ObviousBenchObviousBench measures whether language models avoid simple, visible mistakes (arithmetic, counting, spelling… | Knowledge |
| A | Open Medical-LLM LeaderboardA leaderboard that scores LLMs by accuracy on medical question-answering datasets including MedQA, MedMCQA,… | Healthcare |
| A | MLE-benchMLE-bench measures how well AI agents perform machine learning engineering on 75 Kaggle competitions, such as… | Self Improvement |
| A | OpenAI MRCR v2 8-needle 128K-256KOpenAI MRCR v2 8-needle 128K-256K measures a model's ability to retrieve and distinguish multiple similar… | Long Context |
| A | OpenAI MRCR v2 8-needle 16K-32KMeasures long-context multi-round coreference retrieval, requiring models to locate eight needles within… | Long Context |
| A | OpenAI MRCR v2 8-needle 256K-512KOpenAI MRCR v2 8-needle 256K-512K measures a model's ability to retrieve and distinguish eight similar needles… | Long Context |
| A | OpenAI MRCR v2 8-needle 32K-64KOpenAI MRCR v2 8-needle 32K-64K measures a model's ability to retrieve multiple needles from long contexts of… | Long Context |
| A | OpenAI MRCR v2 8-needle 4K-8KOpenAI MRCR v2 8-needle 4K-8K measures a model's ability to retrieve and distinguish multiple (8) similar… | Long Context |
| A | OpenAI MRCR v2 8-needle 512K-1MOpenAI MRCR v2 8-needle 512K-1M measures a model's ability to retrieve eight needles from contexts of 512K to… | Long Context |
| A | OpenAI MRCR v2 8-needle 64K-128KOpenAI MRCR v2 8-needle 64K-128K measures a model's ability to retrieve and distinguish multiple similar items… | Long Context |
| A | OpenAI MRCR v2 8-needle 8K-16KOpenAI MRCR v2 8-needle 8K-16K measures a model's ability to retrieve multiple (8) needles across… | Long Context |
| A | Organic chemistryAn Anthropic internal evaluation of fundamental organic chemistry skills such as structure elucidation from… | Chemistry |
| A | OSWorld2.0OSWorld 2.0 measures computer-use agents on 108 long-horizon, real-world desktop workflows across everyday and… | Computer Use |
| A | ProfBenchProfBench scores models on professional work tasks, with a search subset reported for agentic knowledge-work… | Agentic |
| A | Protocol troubleshootingMeasures a model's ability to detect and fix errors in molecular biology protocols, including using web search… | Biology |
| A | Real-World Finance v2Real-World Finance v2 measures a model's ability to complete complex, long-horizon financial-analysis tasks… | Finance |
| A | SearchAgent LeaderboardA standardized leaderboard scoring search-augmented question-answering agents by average exact match across… | Agentic |
| A | SURDSSURDS is an autonomous-driving evaluation measuring multimodal models' understanding of driving scenes,… | Multimodal |
| A | SVFSearchSVFSearch measures how well multimodal models and agents answer knowledge-intensive four-choice questions… | Multimodal |
| A | ThemisThemis-CodeRewardBench evaluates code reward models across five preference criteria and eight programming… | Coding |
| A | TVBenchTVBench is a video understanding benchmark that scores models on reasoning over events, actions and temporal… | Multimodal |
| A | UniGenBenchUniGenBench++ measures how accurately text-to-image models' generated images reflect the semantics of their… | Multimodal |
| A | UniGenBench English LongUniGenBench English Long measures how accurately text-to-image models reflect the semantics of long-form… | Multimodal |
| A | Vals.ai Financial Agent 1.1Vals.ai Financial Agent 1.1 scores models as financial agents, reported with and without web search. | Agentic |
| A | Vibe Code Bench v1.1Vibe Code Bench measures whether AI models can build complete working web applications from scratch, evaluated… | Coding |
| A | VBench Video Generation LeaderboardVBench scores video generative models (text-to-video and image-to-video) across 16 fine-grained dimensions of… | Creative Media Video Generation |
| A | Vectara HHEM Hallucination LeaderboardMeasures how often LLMs hallucinate information not stated in a source document when summarizing it, scored by… | Behavioral |
| A | VideoMME w/o sub.Video-MME measures how well multi-modal LLMs understand and answer questions about videos of varied types and… | Multimodal |
| A | VideoMME w sub.Video-MME evaluates multi-modal LLMs on video understanding across short to hour-long videos, using frames,… | Multimodal |
| A | ViroBenchViroBench evaluates nucleotide foundation models on viral genomics tasks, covering biological understanding… | Biology |
| A | VisFactorVisFactor is a visual reasoning benchmark that scores multimodal models on visual perception, spatial… | Multimodal |
| A | VLADBenchVLADBench evaluates multimodal models on autonomous-driving scene understanding, including dynamic scenes,… | Multimodal |
| A | VNTL LeaderboardVNTL Leaderboard ranks LLMs and translation tools on translating Japanese visual novel lines into English,… | Translation |
| A | WorldVQAWorldVQA measures multimodal models' ability to answer knowledge-seeking visual questions, combining visual… | Multimodal |
| A | YapBenchYapBench measures how much assistant LLMs over-generate (excess response length beyond a minimal baseline) on… | Efficiency |
| A | ZebraLogicZebraLogic measures LLM logical reasoning on programmatically generated logic grid (zebra) puzzles, scoring… | Knowledge |
| B | Senior SWE-benchSenior SWE-bench evaluates coding agents on senior-level software engineering work: under-specified feature… | Coding Computer Use+2 |
| B | ARC-AGI-3ARC-AGI-3 measures whether AI agents can explore novel interactive game environments, acquire goals, build… | Reasoning |
| B | FinanceQAFinanceQA evaluates LLMs on complex numerical financial analysis tasks mirroring real-world investment work,… | Coding Computer Use+1 |
| B | IDE-BenchIDE-Bench evaluates LLMs acting as IDE agents on 80 real-world software engineering tasks across eight… | Coding Computer Use+1 |
| B | Continual Learning BenchContinual Learning Bench measures whether LLM-based agent systems genuinely improve through sequential… | Coding Computer Use+2 |
| B | APEX-Agents 1.1APEX-Agents 1.1 measures how well AI agents perform long-horizon, multi-step professional knowledge-work tasks… | Coding Enterprise Workflows+1 |
| B | APEX-SWEAPEX-SWE measures whether frontier AI models can perform real-world software engineering work, including… | Coding Enterprise Workflows+1 |
| B | APEX-AccountingAPEX-Accounting measures AI agents' ability to complete long-horizon professional accounting tasks across… | Coding Enterprise Workflows+1 |
| B | CodeforcesCodeforces evaluates models on competition-level programming problems from the Codeforces platform, reporting… | Coding |
| B | FinanceBenchFinanceBench measures how well LLMs answer open-book financial questions about publicly traded companies using… | Multi-Domain Coding |
| B | TRAILTRAIL measures how well LLMs can debug and localize errors in long agentic workflow traces, using 148… | Multi-Domain Coding |
| B | SWEMeasures whether AI agents can deploy, debug, migrate, and maintain live production software systems across… | ML Science |
| B | CoreCraftCoreCraft measures whether AI agents can perform multi-step, domain-specific customer support work in a… | Multi-Domain |
| B | MMLU-ProXMMLU-ProX measures LLMs' advanced knowledge and cross-linguistic reasoning using 11,829 identical… | Multilingual |
| B | GSM8KGSM8K measures language models' multi-step mathematical reasoning on 8.5K grade school math word problems. | Math |
| B | SWE-bench LiteSWE-bench Lite measures whether language models can resolve real GitHub issues by editing Python codebases,… | Coding |
| B | Terminal-Bench-ScienceTerminal-Bench-Science measures how well AI agents complete research workflows across scientific domains… | Agentic Science+1 |
| B | Code MigrationCode Migration measures how well models can reimplement existing programs in another language (CLI tools… | Coding |
| B | AIME 2026AIME 2026 scores models on the 30 final-answer math problems from the 2026 American Invitational Mathematics… | Math |
| B | BrowseComp-zhBrowseComp-ZH measures how well LLM agents browse the Chinese web to answer 289 hard multi-hop questions with… | Search |
| B | FrontierMath 2025-02-28 PrivateFrontierMath 2025-02-28 Private measures model performance on research-level mathematics problems at the… | Math |
| B | FrontierMath Tier 4 2025-07-01 PrivateMeasures model accuracy on the hardest, research-level Tier 4 FrontierMath mathematics problems from a private… | Math |
| B | Long-Horizon Terminal-BenchLong-Horizon Terminal-Bench measures how well LLM agents sustain progress over hundreds of steps on 46… | Agentic |
| B | OTIS Mock AIME 2024-2025Measures models on competition-level math problems from the OTIS Mock AIME, evaluating olympiad-level… | Math |
| B | Video SimpleQAVideo SimpleQA measures factual grounding of large video language models using short-form, multi-hop,… | Multimodal |
| B | Remote Labor IndexRemote Labor Index measures how well AI agents complete real-world, economically valuable freelance remote… | Coding Computer Use+2 |
| B | HiL-BenchHiL-Bench measures agents' help-seeking judgment (selective escalation): detecting missing, ambiguous or… | Coding Computer Use+2 |
| B | CEO BenchCEO Bench measures how well models perform executive and departmental decision work inside a simulated… | Coding Enterprise Workflows+1 |
| B | VLMBenchVLMBench measures vision-language models on open-ended, image-based real-world business and STEM tasks… | Coding |
| B | CWE-bench | Security Enterprise Workflows |
| B | WebBenchWebBench measures how effectively browser agents complete realistic read and write web workflows across 2,454… | Finance Enterprise Workflows |
| B | SpreadsheetBench 2SpreadsheetBench 2 measures how well LLM agents complete end-to-end business spreadsheet workflows… | Coding Computer Use+1 |
| B | SkillsBenchSkillsBench measures whether curated Agent Skills (procedural knowledge packages) improve LLM agent pass rates… | Coding Computer Use+1 |
| B | Realm: Pathology-report reasoning benchmark | Enterprise Workflows |
| B | GDP.pdfGDP.pdf measures whether models can parse, understand, and synthesize complex real-world professional PDFs… | Agentic |
| B | ToolathlonToolathlon measures how well language agents complete realistic, long-horizon, multi-app tasks using 604 tools… | Agentic |
| B | OfficeQA ProOfficeQA Pro measures how well models and agents answer hard grounded-reasoning questions by finding and… | Multimodal |
| B | FrontierSWEFrontierSWE measures how well coding agents handle ultra-long-horizon, open-ended software engineering tasks… | Coding |
| B | UI-BenchUI-Bench measures the visual design quality of websites and apps generated by AI text-to-app tools through… | Coding Computer Use+1 |
| B | Market-BenchMarket-Bench evaluates LLMs on introductory quantitative trading by having them write executable backtesters… | Coding Computer Use+1 |
| B | YC-BenchYC-Bench measures whether LLM agents can maintain long-term planning and consistent execution by running a… | Coding Computer Use+2 |
| B | VERITASVeriTaS is a dynamic, quarterly-updated benchmark that scores automated fact-checking systems on real-world… | Coding Computer Use+2 |
| B | Vending-BenchVending-Bench measures an LLM agent's long-term coherence by having it run a simulated vending machine… | Computer Use Long-Horizon |
| B | Crosby-micro1 RedlineBenchCrosby-micro1 RedlineBench measures how well models act as negotiation participants in multi-turn SaaS MSA… | Enterprise Workflows |
| B | RealmRealm is micro1's benchmark series evaluating multi-step domain reasoning, notably long-horizon legal… | Multi-Domain |
| B | AppBenchApp-Bench measures how much required functionality AI coding assistants and web-app builders deliver when… | Multi-Domain |
| B | Cua-BenchCua-Bench measures AI agents' computer-use mastery on desktop and mobile tasks, including expert KiCad… | Computer Use |
| B | Code IndexCode Index is a live coding leaderboard scoring models and agents on contamination-free private task sets… | Coding |
| B | BinaryAuditBinaryAudit measures AI agents' ability to find backdoors and timebombs hidden in stripped compiled binaries… | Security |
| B | unix-ctfunix-ctf procedurally generates capture-the-flag tasks in Linux containers to train and evaluate Unix… | ML |
| B | AutomationBench-AAAutomationBench-AA measures how well AI agents complete cross-application business workflow tasks via REST… | Agentic Business |
| B | τ-benchτ-bench measures how reliably language agents use domain-specific API tools and follow policy guidelines in… | Agentic |
| B | BOOMBOOM measures how well time-series forecasting models predict real-world observability and production… | Data |
| B | CHOICECHOICE measures the remote sensing perception and reasoning capabilities of vision-language models on 10,507… | Geospatial |
| B | CLEARCLEAR is an evaluation framework that measures how answer-option count, abstention options and semantic… | Biomedicine Healthcare |
| B | DoneStandard embodied evaluations do not independently score whether an agent correctly commits to task completion… | Embodied |
| B | FACTS GroundingFACTS Grounding measures how accurately LLMs generate long-form responses fully grounded in a provided… | Knowledge |
| B | ForgeForge (OPT-BENCH) evaluates LLMs on NP-hard optimization problems, scoring both solution feasibility (Success… | Reasoning |
| B | HEARHEAR evaluates general-purpose audio embedding models on nineteen downstream tasks spanning speech,… | Audio |
| B | PostTrainBenchPostTrainBench measures how well LLM agents can autonomously post-train small base language models under a… | Agentic |
| B | GBA EvalGBA Eval measures whether frontier AI coding agents can write a complete Game Boy Advance emulator in… | Coding |
| B | LeapLEAPBench measures the in-context learning efficiency of LLMs on iterative scientific design optimization… | Multi-Domain Tool Use |
| B | AttuneAttuneBench measures LLM emotional intelligence by having models predict real participants' self-labeled… | Multi-Domain Tool Use |
| B | AutoCAD-BenchAutoCAD Bench measures whether multimodal agents can produce correct, editable 2D drawings and 3D models in… | Computer Use Tool Use |
| B | Toloka ArenaToloka Arena measures agentic intelligence via multi-turn tool-use tasks in simulated customer service and… | Multi-Domain |
| B | Long-Horizon PlanningLong-Horizon Planning measures an agent's long-horizon planning by its trading performance (PnL, Sharpe,… | Long-Horizon ML |
| B | CAD ArenaCAD Arena measures how well AI agents reconstruct real manufactured parts from engineering drawings as… | Chip Design |
| B | TB3 resultsTerminal-Bench 3.0 (TB3) measures how well AI agents complete hard coding tasks in terminal environments,… | Coding Long-Horizon |
| B | Math RL TasksMath RL Tasks scores models on 20 constructive mathematics problems where an executable verifier checks a… | Math Science |
| B | DABstepDABstep measures how well LLM agents perform multi-step data analysis, answering ~450 questions that require… | Agentic Business |
| B | t2-benchτ2-bench evaluates conversational AI agents in a dual-control environment (e.g., telecom technical support)… | Agentic |
| B | TRACETRACE evaluates conversational tourism recommender systems on multi-turn dialogues for recommending the right… | Tool Use |
| B | TRACKTRACK measures how well LLMs propagate updated knowledge that conflicts with their parametric knowledge… | Reasoning |
| B | TRAPTRAP measures the trade-off between agents' task accuracy when using private information in tool calls and… | Agentic |
| B | VeRAVeRA converts reasoning benchmark problems (e.g., AIME, BeyondAIME, AMO-Bench math problems) into executable… | Reasoning |
| B | TRACESTRACES measures whether LLMs recognize and reject flawed premises from retracted, fraudulent, and… | |
| B | CyberGymCyberGym measures AI agents' real-world cybersecurity capabilities by having them generate proof-of-concept… | Agentic |
| B | DeepSearchQADeepSearchQA measures how well deep research agents execute multi-step web search plans to produce exhaustive,… | Agentic |
| B | Claw-EvalClaw-Eval measures how well autonomous LLM agents complete real-world multi-step tasks across service… | Agentic |
| B | DeepPlanningDeepPlanning measures whether agentic LLMs can do long-horizon planning, such as multi-day travel and… | Agentic |
| B | CyberBenchCyberBench measures whether AI agents can craft proof-of-concept inputs that trigger real OSS-Fuzz… | Agentic |
| B | EnigmaEvalEnigmaEval measures LLMs' creative, multi-step reasoning and multimodal processing on complex puzzle-hunt… | Reasoning |
| B | DynaMathDynaMath measures the robustness of vision-language models' mathematical reasoning by evaluating them on… | Multimodal |
| B | CC-OCRCC-OCR measures the OCR and literacy capabilities of large multimodal models across multi-scene text reading,… | Multimodal |
| B | Blueprint-Bench 2Blueprint-Bench 2 measures AI agents' spatial reasoning by having them convert ~20 apartment photographs per… | Multimodal |
| B | AudioMC – Text OutputMeasures conversational intelligence of spoken dialogue and audio-language models in multi-turn human-speech… | Multimodal |
| B | DocVQADocVQA measures how well models answer questions asked about document images, testing document understanding. | Multimodal |
| B | CMMLUCMMLU measures the knowledge and reasoning of large language models in Chinese across many academic subjects,… | Knowledge |
| B | CRUXEvalCRUXEval measures code reasoning, understanding and execution by asking models to predict inputs and outputs… | Decision Models |
| B | APEX-Agents-AAAPEX-Agents-AA is Artificial Analysis' implementation of APEX-Agents, scoring AI agents by pass@1 on… | Agentic |
| B | SWE-Atlas RefactoringSWE Atlas - Refactoring measures a coding agent's ability to restructure production code across real… | Agentic |
| B | ResearchClawBenchResearchClawBench measures whether AI agents can autonomously conduct end-to-end scientific research from… | Agentic |
| B | SWE-RebenchSWE-rebench measures how well LLM-based software engineering agents resolve fresh, decontaminated real-world… | Coding |
| B | Vibe Code BenchVibe Code Bench measures whether AI models can build complete, deployable web applications from natural… | Coding |
| B | SWE-Atlas Codebase QnASWE-Atlas Codebase QnA measures AI coding agents' deep code comprehension by having them explore, run and… | Coding |
| B | NL2RepoNL2Repo-Bench measures whether coding agents can generate a complete, installable Python library from a single… | Coding |
| B | React Native EvalsReact Native Evals measures the success rate, token usage and cost of AI coding agents on React Native code… | Coding |
| B | ReactBenchReactBench measures how well coding agents implement and fix realistic React tasks, checking correctness plus… | Coding |
| B | KernelBench HardKernelBench Hard measures how well frontier coding agents write and optimize GPU kernels (CUDA) for… | Coding |
| B | GameDevBenchGameDevBench measures how well coding agents complete 333 multimodal game-development tasks in the Godot… | Coding |
| B | SWE Atlas – Test WritingSWE Atlas – Test Writing measures how well AI coding agents can autonomously explore real production… | Coding |
| B | WeirdMLWeirdML scores language models on unusual machine learning tasks that test their adaptability. | Coding |
| B | BBHBBH (BIG-Bench Hard) measures language model performance on 23 challenging BIG-Bench tasks, many requiring… | Reasoning |
| B | HellaSwagHellaSwag measures commonsense natural language inference by asking models to select the most plausible… | Reasoning |
| B | WinoGrandeWinoGrande measures commonsense reasoning through 44k adversarially filtered Winograd-style pronoun resolution… | Reasoning |
| B | LongBench v2LongBench v2 measures LLMs' deep understanding and reasoning over long contexts (8k to 2M words) via 503… | Reasoning |
| B | OCRBench V2OCRBench v2 measures large multimodal models' OCR abilities, including text recognition, localization,… | Multimodal |
| B | OmniDocBenchOmniDocBench measures how well models parse diverse real-world PDF documents, scoring end-to-end parsing,… | Multimodal |
| B | PerceptionBenchPerceptionBench measures atomic visual perception in multimodal LLMs across ten capabilities such as counting,… | Multimodal |
| B | RealWorldQARealWorldQA measures basic real-world spatial understanding of multimodal models using over 700 real-world… | Multimodal |
| B | MMLongBench-DocMMLongBench-Doc measures how well large vision-language models understand and answer questions about long,… | Multimodal |
| B | ERQAERQA measures multimodal models' embodied reasoning via 400 multiple-choice visual questions on spatial… | Multimodal |
| B | MLVUMLVU evaluates multimodal large language models on long video understanding across diverse video genres,… | Multimodal |
| B | LVBenchLVBench measures how well multimodal models understand and extract information from extremely long videos… | Multimodal |
| B | MMVUMMVU measures expert-level, knowledge-intensive video understanding of multimodal foundation models across 27… | Multimodal |
| B | TIR-BenchTIR-Bench evaluates multimodal models' agentic thinking-with-images reasoning across 13 tasks requiring tool… | Multimodal |
| B | MedXpertQAMedXpertQA measures expert-level medical knowledge and reasoning with 4,460 text and multimodal exam questions… | Multimodal |
| B | SuperGPQASuperGPQA measures LLMs' graduate-level knowledge and reasoning with 26,529 multiple-choice questions across… | Knowledge |
| B | OpenBookQAOpenBookQA measures whether models can answer elementary-level science multiple-choice questions by combining… | Knowledge |
| B | MultiNRCMultiNRC measures LLM reasoning on native, culturally grounded questions in French, Spanish and Chinese across… | Knowledge |
| B | PRBench FinancePRBench Finance measures LLMs' high-stakes professional reasoning on expert-written finance tasks, scored… | Knowledge |
| B | PRBench LegalPRBench Legal measures LLMs on high-stakes legal professional reasoning tasks written by lawyers, scored… | Knowledge |
| B | SciPredictSciPredict measures whether LLMs can forecast the outcomes of real, recently published experiments in physics,… | Knowledge |
| B | TutorBenchTutorBench measures how well LLMs perform core tutoring tasks—adaptive explanations, feedback on student work,… | Knowledge |
| B | Gray Swan IPIMeasures how often indirect prompt-injection attacks, hidden in content like web pages, emails or documents,… | Instruction Following |
| B | IFEvalIFEval measures how well large language models follow verifiable natural language instructions, such as… | Instruction Following |
| B | IMOAnswerBenchIMO-AnswerBench tests models on 400 diverse International Mathematical Olympiad-level problems with verifiable… | Math |
| B | LiveBenchLiveBench is a frequently updated, contamination-limited LLM benchmark with objectively scored tasks across… | Cross-domain |
| B | AdvBenchAdvBench measures whether models follow harmful instructions, scoring their tendency to produce harmful… | Generalization |
| B | Age of LLMAge of LLM measures LLMs' strategic reasoning, diplomacy and action reliability by pitting two models against… | Game |
| B | AgentDojoAgentDojo evaluates the robustness of tool-using LLM agents against prompt injection attacks and the… | Generalization |
| B | AgentFloorAgentFloor is a deterministic 30-task benchmark organized as a six-tier capability ladder measuring how well… | Agentic |
| B | AgentHarmAgentHarm measures whether LLM agents refuse explicitly malicious multi-step agentic tasks and how well… | Generalization |
| B | Aider PolyglotAider Polyglot measures how well LLMs edit code to solve 225 hard Exercism exercises across C++, Go, Java,… | Coding |
| B | AMA-BenchAMA-Bench evaluates long-horizon memory of LLM agents over agent-environment interaction trajectories using… | Agentic |
| B | Android BenchAndroid Bench measures how well LLMs and coding agents resolve real Android development tasks, including… | Coding |
| B | LifeSci DiamondLifeSci Diamond measures how well AI systems produce free-text regulatory affairs deliverables for drugs,… | Biomedicine Business+1 |
| B | Arena AI Image-to-VideoA crowdsourced pairwise human-preference leaderboard ranking image-to-video generation models by votes on… | Multimodal |
| B | Arena-HardArena-Hard-Auto scores instruction-tuned LLMs on hard real-world user queries and creative writing prompts… | Generalization |
| B | Arena-Hard v2Arena-Hard v2 measures instruction-tuned LLMs on 500 challenging real-world user queries plus 250 creative… | General Knowledge |
| B | ASD-BenchASD-Bench evaluates ML, deep learning and foundation models for autism spectrum disorder screening from AQ-10… | Biomedicine Healthcare |
| B | ATLASATLAS measures long-context language model capabilities across eight dimensions with length-aware scoring over… | Long Context |
| B | Auto-CompAuto-Comp is an automated synthetic pipeline and set of benchmarks probing compositional reasoning (color… | Multimodal Understanding |
| B | AutoBenchAutoBench is a dynamic LLM benchmark that generates virtual agentic business environments with multiple LLMs… | Agentic |
| B | BabyAIBabyAI is a gridworld platform with levels that test how sample-efficiently agents learn to understand and… | Agentic |
| B | BacktestBenchBacktestBench measures how well LLMs automate quantitative trading strategy backtesting across metrics… | Finance |
| B | BalrogBALROG measures the agentic reasoning and decision-making capabilities of LLMs and VLMs across a diverse set… | Knowledge |
| B | BehaviorBenchBehaviorBench evaluates foundation models on behavioral science tasks (behavior prediction and simulation,… | Behavioral |
| B | BenchCADBenchCAD measures how well multimodal models understand and generate executable parametric CadQuery programs… | Coding |
| B | BIG-benchBIG-bench is a collaborative benchmark of more than 200 tasks probing large language models' broad… | Knowledge |
| B | BIRD-SQLBIRD-SQL evaluates text-to-SQL systems on converting natural language questions into SQL over large… | Data |
| B | BizFinBenchBizFinBench measures how well LLMs handle real-world, business-driven financial tasks such as anomalous event… | Finance Business |
| B | BLINKGBLINKG measures how well LLMs map heterogeneous data source schemas to ontology terms to construct knowledge… | Data |
| B | C3-Bench: A Context-Aware Change Captioning BenchmarkC3-Bench measures how well change captioning models and large multimodal models describe changes between image… | Vision |
| B | CADGenBenchCADGenBench measures how well AI systems generate and edit valid, geometrically correct 3D mechanical part… | Multimodal |
| B | CanLegalRAGBenchCanLegalRAGBench measures retrieval-augmented generation systems on Canadian legal question answering, using… | Legal |
| B | CardioLensCardioLens evaluates multimodal LLMs on multi-sequence cardiac MRI interpretation, covering image… | Biomedicine Healthcare |
| B | CFLUECFLUE evaluates large language models on Chinese financial language understanding through knowledge assessment… | Finance |
| B | ChartographerChartographer evaluates vision-language models' visual reasoning on chart question answering using… | Reasoning |
| B | CheckMIABenchCheckMIABench evaluates membership inference attacks against language models, using intermediate checkpoints… | Safety |
| B | ChemBenchChemBench evaluates the chemical knowledge, reasoning, and intuition of language and multimodal models against… | Science |
| B | CHI-BenchCHI-Bench evaluates long-horizon, policy-rich U.S. healthcare workflow agents across provider prior… | Agentic Healthcare |
| B | Chinese Labor Law Large Language Model BenchmarkLabourLawBench measures how well LLMs handle Chinese labor-law tasks such as provision citation, legal QA,… | Legal |
| B | CiteVQACiteVQA measures whether multimodal LLMs answer questions about long real-world PDFs correctly while citing… | Document Understanding |
| B | ClaimDBClaimDB measures how well LLMs verify factual claims whose evidence comes from compositions of millions of… | Factuality |
| B | ClassEvalClassEval measures how well LLMs generate class-level Python code, scored by class- and method-level test pass… | Coding |
| B | ClawBenchClawBench measures whether AI browser agents can complete everyday online tasks such as purchases, bookings… | Agentic |
| B | ClawProBenchClawProBench measures how models perform as OpenClaw agents when they must reason, plan, use tools, and stay… | Agentic |
| B | CLUBenchCLUBench evaluates 24 clustering algorithms, spanning conventional, deep learning-based and foundation… | Machine Learning |
| B | CommunityBenchCommunityBench evaluates how well large language models model community-specific preferences and values,… | Alignment |
| B | COMPL-AICOMPL-AI is an open-source benchmarking suite that scores LLMs against technical requirements derived from the… | Safety |
| B | ComplexMCPComplexMCP measures how well LLM agents complete interdependent commercial software automation tasks using… | Tool Use Business |
| B | ConStory-BenchConStory-Bench measures narrative consistency errors (contradictions in facts, characters, timeline, world… | Long Context |
| B | ContextEchoContextEcho measures persona drift in frontier language models over long, tool-using agentic-coding sessions… | Agentic |
| B | Continual Model Routing in Evolving Model HubsCMRBench measures how well routing methods select the right pre-trained model from a continually expanding hub… | Machine Learning |
| B | ContractBenchContractBench measures whether tool-augmented LLM agents preserve observation contracts, i.e., the temporal… | Agentic |
| B | ControBenchControBench measures how well graph neural networks, pretrained language models and LLMs predict users'… | Domain |
| B | Convex Coding Evals — No GuidelinesMeasures whether language models can write working Convex backends and client integrations without… | Coding |
| B | CorpBench Work 1.3CorpBench Work 1.3 evaluates AI agents on 100 deterministic business workflows across finance, marketing,… | Agentic Business |
| B | CTFBenchCTFBench evaluates AI-powered smart contract auditors on test contracts each containing one vulnerability,… | Generalization |
| B | CursorBench 3.1CursorBench 3.1 measures coding agents on ambiguous, multi-file tasks sourced from real Cursor sessions,… | Coding |
| B | CV-ArenaCV-Arena evaluates instruction-guided image editing on 12K high-resolution real-image instruction pairs across… | Vision |
| B | Data Agent BenchmarkUC Berkeley EPIC Data Lab benchmark for data agents answering complex real-world data tasks across 12… | Agentic |
| B | DataComp-VLMDataComp-VLM is a data-curation benchmark where participants curate training data for vision-language models,… | Multimodal |
| B | Deep20BenchDeep20Bench measures how many yes/no-style questions language models need to identify a hidden subject in a… | Game Reasoning+1 |
| B | DeepResearchEvalDeepResearchEval scores deep research systems on persona-driven, multi-source web research tasks, measuring… | Agentic |
| B | DentalGPT Dental Eval SuiteMeasures multimodal models' understanding of dental images, including panoramic X-ray understanding, dental… | Healthcare |
| B | DocScopeDocScope measures whether multimodal LLMs can produce verifiable reasoning trajectories (evidence pages,… | Document Understanding |
| B | Dr-CiKDr-CiK measures whether agents can retrieve forecasting-relevant context from a noisy document corpus, filter… | Agentic |
| B | DrVD-BenchDrVD-Bench measures whether vision-language models reason like clinicians in medical image diagnosis across… | Biomedicine Healthcare |
| B | DuelLab GameBench 2DuelLab GameBench 2 measures how well AI models write game-playing programs, which are compiled and run… | Game |
| B | EbisuEbisu measures how well LLMs understand native Japanese financial language, via implicit commitment/refusal… | Finance |
| B | EgoCoT-BenchEgoCoT-Bench measures how well multimodal LLMs perform fine-grained, grounded and verifiable operation-centric… | Reasoning |
| B | EHRBenchEHRBench measures LLM clinical decision-making on nearly 1M EHR-grounded QA items covering diagnosis,… | Biomedicine Healthcare |
| B | EmoHarborEmoHarbor evaluates how well LLMs provide personalized emotional support by simulating users' internal worlds… | Behavioral |
| B | EscherVerseEscherVerse measures teleo-spatial reasoning in vision-language models: reasoning about physical dynamics,… | Spatial Reasoning |
| B | EvalPlusEvalPlus measures the functional correctness of LLM-generated code using test-augmented benchmarks HumanEval+… | Coding |
| B | ExploitBench v8-benchExploitBench v8-bench measures how far AI agents progress in exploiting 41 patched V8 bugs, graded across 16… | Cybersecurity |
| B | FAM-BenchFAM-Bench measures whether language and vision-language models can judge and rank dishes for suitability to… | Healthcare |
| B | FCMBenchFCMBench evaluates vision-language models on perception and credit-specific reasoning tasks over financial… | Finance |
| B | FEV Benchfev-bench measures time series forecasting models (pretrained, statistical, baseline) on 100 real-world… | Data |
| B | Fin-BiasFin-Bias measures how LLM investment decisions (Bullish/Neutral/Bearish ratings) from long firm analyst… | Finance |
| B | Finetuning with Scientific Data Increases HallucinationsSciFactCheck measures factuality hallucinations (unverifiability, overclaim, attribution) of LLMs, including… | Factuality |
| B | FinVerBenchFinVerBench measures whether LLMs can correctly verify the numerical consistency of corporate financial… | Finance |
| B | Flat-Pack BenchFlat-Pack Bench measures fine-grained spatio-temporal video understanding of large vision-language models on… | Video |
| B | FollowTableFollowTable measures how well retrieval models follow fine-grained content and schema instructions when… | Retrieval |
| B | ForecastBench-SimForecastBench-Sim is a contamination-free, simulated-world benchmark that scores LLMs' distributional… | Finance |
| B | FormalRewardBenchFormalRewardBench measures how well reward models distinguish correct from incorrect Lean 4 formal proofs… | Math |
| B | FrontierCodeFrontierCode measures whether model-generated code changes on real open-source repositories meet maintainers'… | Coding |
| B | Hindsight LLM Memory LeaderboardA leaderboard ranking LLMs, rerankers and embedding models on Hindsight agent-memory operations (retain fact… | Agentic |
| B | ImpermanentImpermanent measures how well time-series forecasting models generalize over time by scoring their forecasts… | Data |
| B | IneqMathIneqMath measures LLMs' ability to solve Olympiad-level inequality problems via bound estimation and relation… | Math |
| B | InvisibleBenchInvisibleBench evaluates AI models in multi-turn caregiver-support conversations for safety (crisis detection,… | Safety Healthcare |
| B | LatencyGridLatencyGrid measures latency of voice AI infrastructure across providers, including LLM time-to-first-token,… | Audio |
| B | LEXamLEXam measures LLM legal reasoning on 7,537 English and German law exam questions, both long-form open… | Legal |
| B | LiveSecBenchLiveSecBench is a dynamic safety leaderboard that scores LLMs, mainly in Chinese-language applications, on… | Safety |
| B | LiveSQLBenchLiveSQLBench measures how well LLMs and agents solve real-world text-to-SQL tasks, including… | Text TO Sql |
| B | MageBench Season 1MageBench Season 1 rates LLMs by playing Magic: The Gathering games across formats, reporting ratings, blunder… | Game |
| B | MBPP+MBPP+ measures LLM ability to synthesize short Python programs from natural language descriptions, scored by… | Coding |
| B | McEvalMcEval measures code LLMs on code generation, completion, and understanding tasks across 40 programming… | Coding |
| B | MCP-UniverseMCP-Universe measures how well LLMs and agents complete realistic, long-horizon tasks by interacting with… | Agentic |
| B | MCP-BenchMCP-Bench evaluates LLM agents on realistic multi-step tool-use tasks across 28 live MCP servers, scoring… | Agentic |
| B | MCPMarkMCPMark measures how well AI models use MCP servers to complete challenging, verifiable tasks across Notion,… | Agentic |
| B | MedAraBenchMedAraBench measures how well large language models answer Arabic multiple-choice medical questions across 19… | Biomedicine |
| B | Medical STT BenchmarkMeasures speech-to-text accuracy and latency on medical audio, including medical word error rate, drug-term… | Audio |
| B | METR Task-Completion Time Horizons 1.1METR Time Horizon 1.1 measures the human-expert task duration at which AI agents are predicted to succeed with… | Agentic Autonomy |
| B | MIR-SafetyBenchMIR-SafetyBench measures how vulnerable multimodal large language models are to safety attacks built on… | Safety |
| B | MMAUMMAU measures how well audio-language models understand and reason about speech, environmental sounds, and… | Multimodal |
| B | MMBench-CNMMBench-CN is the Chinese-translated version of MMBench, scoring vision-language models on multiple-choice… | Knowledge |
| B | MMSI-BenchMMSI-Bench measures multimodal LLMs' multi-image spatial reasoning through 1,000 human-designed… | Multimodal |
| B | MolecularIQMolecularIQ measures LLMs' chemical reasoning over molecular graphs via symbolically verifiable counting,… | Chemistry |
| B | MoralChoiceMoralChoice measures how language models handle moral decision scenarios, scored by accuracy and… | Generalization |
| B | NanoGPT Speedrun FrontierA leaderboard measuring how much autonomous coding agents can improve the nanoGPT speedrun training recipe,… | Agentic |
| B | OK-VQAOK-VQA measures visual question answering on images where answering requires outside knowledge beyond the… | Knowledge |
| B | Omni-MATHOmni-MATH measures LLMs' olympiad-level mathematical reasoning on 4428 competition problems across 33… | Math |
| B | PolyBenchPolyBench measures how well language models reason about polymer design tasks, from simple to complex… | Materials |
| B | POPEPOPE measures object hallucination in large vision-language models by polling yes/no questions about whether… | Knowledge |
| B | Professional Reasoning Bench - LegalProfessional Reasoning Bench Legal evaluates frontier LLMs on complex legal reasoning tasks drawn from… | Legal |
| B | ProductWebGenProductWebGen measures how well multimodal generative models create product showcase webpages (renderable HTML… | Multimodal Business |
| B | PutnamBenchPutnamBench measures how well theorem-proving systems can formally prove Putnam competition math problems in… | Math |
| B | AgentWorldBenchAgentWorldBench measures how faithfully language world models predict the next environment observation after… | Agentic |
| B | RealToxicityPromptsRealToxicityPrompts measures how likely language models are to generate toxic continuations (neural toxic… | Generalization |
| B | RedSage-BenchRedSage-Bench measures LLM cybersecurity knowledge, offensive skills, and security tool expertise through 30K… | Cybersecurity |
| B | RefusalBenchRefusalBench measures frontier LLM refusal behaviour and safety calibration on matched biological research… | Safety |
| B | RepairBenchRepairBench is a leaderboard that scores frontier models on zero-shot program repair of 574 single-function… | Coding |
| B | RuVerBenchRuVerBench measures how reliably LLM judges verify whether long agentic outputs (deep research reports and… | Agentic |
| B | SaaS-BenchSaaS-Bench measures whether computer-use agents can complete long-horizon professional workflows by driving a… | Software Engineering |
| B | scBenchscBench measures AI agents' ability to perform single-cell RNA-seq analysis tasks on real data snapshots,… | Science |
| B | SciKnowEvalSciKnowEval measures large language models' multi-level scientific knowledge (memory, comprehension,… | Science |
| B | SecCodeBenchSecCodeBench measures the security of code generated and repaired by AI models and agentic coding tools, with… | Cybersecurity |
| B | SEED-Bench-2SEED-Bench-2 measures the hierarchical capabilities of multimodal large language models, including image-text… | Knowledge |
| B | SimpleBenchSimpleBench measures LLMs on over 200 multiple-choice questions covering spatio-temporal reasoning, social… | Reasoning |
| B | SingGuard-BenchSingGuard-Bench measures how well multimodal guardrail models classify safety risks across 80+ risk types in… | Uncategorized |
| B | SPACENUM: Revisiting Spatial Numerical Understanding in VLMsSpaceNum measures whether vision-language models can ground numerical values (action magnitudes, coordinates)… | Spatial Reasoning |
| B | SpatialBenchSpatialBench measures whether AI agents can extract biological insight from real-world spatial transcriptomics… | Science |
| B | SpeakerSleuthSpeakerSleuth measures whether large audio-language models can reliably judge speaker consistency across… | Audio |
| B | SpiderSpider measures cross-domain text-to-SQL semantic parsing, scoring how well models generate complex SQL… | Data |
| B | StoryAlignStoryAlign (with its StoryRMB benchmark) measures how well reward models select human-preferred stories, using… | Writing |
| B | Structured Output BenchmarkSOB measures how accurately language models produce schema-compliant, value-correct JSON from text-normalized… | Structured Output |
| B | SuperGLUESuperGLUE measures model performance on a set of difficult English language understanding tasks via a… | Knowledge |
| B | SustainBenchSustainBench measures machine learning models on satellite and street-level imagery tasks for monitoring the… | Geospatial |
| B | SWT-BenchSWT-Bench measures whether models or agents can write tests that reproduce real-world user-reported GitHub… | Coding |
| B | TableBenchTableBench measures large language models' table question answering across fact checking, numerical reasoning,… | Knowledge |
| B | TAT-QATAT-QA measures question answering with numerical reasoning over hybrid tabular and textual content from… | Data |
| B | TaxBenchTaxBench measures how accurately and reliably AI models handle real-world tax tasks—tax knowledge and… | Finance Business |
| B | TDC AMESTDC AMES is a leaderboard scoring molecular machine learning models by AUROC on binary prediction of Ames… | Science |
| B | TDC BBB_MartinsTDC BBB_Martins is a Therapeutics Data Commons leaderboard scoring molecular models by AUROC on binary… | Science |
| B | TDC Bioavailability_MaTDC Bioavailability_Ma is a leaderboard that scores molecular models by AUROC on binary prediction of oral… | Science |
| B | TDC Caco2_WangTDC Caco2_Wang is a leaderboard benchmark scoring molecular models on regression of Caco-2 cell effective… | Science |
| B | TDC Clearance_Hepatocyte_AZA Therapeutics Data Commons ADMET leaderboard scoring molecular models by Spearman correlation on predicting… | Science |
| B | TDC Clearance_Microsome_AZA TDC ADMET leaderboard scoring molecular models by Spearman correlation on predicting microsomal clearance of… | Science |
| B | TDC CYP2C9_Substrate_CarbonMangelsA Therapeutics Data Commons ADMET leaderboard task scoring molecular models by AUPRC on predicting whether… | Science |
| B | TDC CYP2C9_VeithTDC CYP2C9_Veith is a Therapeutics Data Commons ADMET leaderboard scoring models by AUPRC on binary prediction… | Science |
| B | TDC CYP2D6_Substrate_CarbonMangelsA Therapeutics Data Commons ADMET leaderboard task scoring models by AUPRC on binary prediction of whether… | Science |
| B | TDC CYP2D6_VeithTDC CYP2D6_Veith is a Therapeutics Data Commons ADMET leaderboard scoring molecular models by AUPRC on binary… | Science |
| B | TDC CYP3A4_Substrate_CarbonMangelsA Therapeutics Data Commons ADMET leaderboard scoring molecular models by AUROC on binary prediction of… | Science |
| B | TDC CYP3A4_VeithTDC CYP3A4_Veith is a Therapeutics Data Commons leaderboard scoring molecular models by AUPRC on binary… | Science |
| B | TDC DILITDC DILI is a Therapeutics Data Commons leaderboard that scores molecular models by AUROC on binary prediction… | Science |
| B | TDC DRD3 DockingA molecule generation benchmark that scores methods on docking scores against the DRD3 target under limited… | Science |
| B | TDC Half_Life_ObachTDC Half_Life_Obach is a leaderboard benchmark scoring molecular models on regression of drug half-life… | Science |
| B | TDC hERGTDC hERG is a Therapeutics Data Commons leaderboard scoring molecular models by AUROC on binary prediction of… | Science |
| B | TDC HIA_HouTDC HIA_Hou is a leaderboard benchmark scoring molecular machine learning models by AUROC on binary prediction… | Science |
| B | TDC LD50_ZhuTDC LD50_Zhu is a Therapeutics Data Commons leaderboard that scores molecular models on regression of acute… | Science |
| B | TDC Lipophilicity_AstraZenecaA Therapeutics Data Commons ADMET leaderboard scoring molecular models by MAE on regressing lipophilicity… | Science |
| B | TDC Pgp_BroccatelliTDC Pgp_Broccatelli is a Therapeutics Data Commons leaderboard that scores molecular models on binary… | Science |
| B | TDC PPBR_AZTDC PPBR_AZ is a Therapeutics Data Commons leaderboard that scores models by mean absolute error when… | Science |
| B | TDC Solubility_AqSolDBTDC Solubility_AqSolDB is a Therapeutics Data Commons ADMET leaderboard that scores molecular models by MAE on… | Science |
| B | TDC VDss_LombardoTDC VDss_Lombardo is a Therapeutics Data Commons leaderboard scoring molecular models on regression of drug… | Science |
| B | TextArena Balanced SubsetTextArena Balanced Subset is a leaderboard slice that ranks LLMs by TrueSkill ratings from competitive play… | Game |
| B | TRLawBenchTRLawBench measures how well large language models handle Turkish law across foundational multiple-choice,… | Legal |
| B | VariantBenchVariantBench measures how well AI agents perform genetic variant discovery and interpretation across 118… | Science Biology+1 |
| B | Visual Aesthetic BenchmarkVisual Aesthetic Benchmark measures whether multimodal models can match expert aesthetic judgment by selecting… | Multimodal |
| B | VitaBenchVitaBench measures how well LLM agents complete multi-turn, tool-heavy consumer-service tasks across food… | Agentic |
| B | VQAv2VQAv2 measures how well models answer open-ended natural-language questions about images, using a balanced… | Knowledge |
| B | VTBVisualToolBench measures how well multimodal LLMs use image-manipulation and other tools to reason with visual… | Multimodal |
| B | WildClawBenchWildClawBench measures how well autonomous AI agents complete 60 hand-crafted real-world tasks inside a live… | Agentic |
| B | WolfBenchWolfBench measures how consistently AI models and coding-agent combinations solve Terminal-Bench 2.0 tasks,… | Agentic |
| B | Workspace-BenchWorkspace-Bench measures how well AI agents operate in realistic workspaces with large-scale file… | Agentic Business |
| B | AGC-BenchAGC-Bench measures the creativity of LLMs across 78 datasets spanning brainstorming, problem solving, STEM,… | |
| B | KernelBench CUDAKernelBench CUDA scores coding agents on writing fast, correct CUDA GPU kernels for hard tasks like fused MoE,… | |
| B | KernelBench MegaKernelBench Mega measures whether coding agents can write a single fused GPU megakernel (e.g., Kimi-Linear… | |
| B | SPIKE-BenchSPIKE-Bench measures whether LLMs comply with toxin-design prompts and generate biologically plausible,… | |
| B | URSAURSA benchmarks retrosynthesis systems and LLMs on synthesis planning for novel drug-like target molecules,… | |
| B | ACEBenchACEBench measures how well large language models use tools across normal, special (ambiguous or incomplete… | Tool Use |
| B | Advancing ESG IntelligenceA three-level benchmark built from 310 corporate sustainability reports that evaluates LLMs and agents on ESG… | Finance |
| B | AfroBench-LiteAfroBench measures how well large language models perform across 64 African languages on understanding,… | Multilingual |
| B | Agent IslandAgent Island measures language-model agents' skill in a multiplayer winner-take-all game of cooperation,… | Agentic |
| B | AgentCIBenchAgentCIBench measures whether computer-use agents leak contextually inappropriate personal information across… | Agentic |
| B | Agentic Skills Evaluation FrameworkEvaluates how well LLM agent-model configurations use agent skills, scoring instruction-following and goal… | Agentic |
| B | AgentRxAgentRx evaluates single- and multi-agent LLM systems on multimodal clinical risk prediction tasks using… | Healthcare |
| B | AGL1KAGL1K measures how well audio-language models can infer the geographic location (continent, country, city,… | Audio |
| B | AICD BenchAICD Bench measures how well detectors identify AI-generated source code, covering robust binary… | Coding |
| B | AlignBenchAlignBench measures how well large language models are aligned in Chinese, using 683 real-scenario queries… | General Knowledge |
| B | ALL Bench LLMALL Bench LLM is a composite leaderboard that aggregates and cross-verifies LLM scores across knowledge,… | General Knowledge |
| B | Altered RiddlesAltered Riddles measures how often language models fall back on memorized answers (Conditioned Override Rate)… | Knowledge |
| B | ArchSIBenchArchSIBench measures the architectural spatial intelligence of vision-language models across perception,… | Spatial Reasoning |
| B | ASPECTSIMASPECTSIM evaluates how well LLMs and embedding models measure document similarity conditioned on an… | Language-understanding |
| B | AsyncToolAsyncTool measures how well LLM-based agents perform asynchronous function calling, coordinating multiple… | Agentic |
| B | AuAuAuAu measures the risk that LLMs generate responses with authoritarian tendencies, using psychometric… | Alignment |
| B | Audio Reasoning ChallengeThe Audio Reasoning Challenge scores single models and agents on deep audio reasoning over speech, sound,… | Audio |
| B | BABILongBABILong measures how well language models extract and reason over facts distributed within extremely long… | Long Context |
| B | BanglaMedVQABanglaMedVQA measures how well LLMs and vision-language models answer clinically validated Bangla medical… | Biomedicine Healthcare |
| B | Benchmarks are Not EnoughRAMP measures LLM software engineering agents on a long-horizon, serially dependent compiler-construction… | Agentic |
| B | Beyond QuestionsBeQu measures the parametric knowledge LLMs surface in response to open-ended elicitation prompts about 10,000… | Factuality |
| B | BIG-Bench Extra HardBIG-Bench Extra Hard (BBEH) measures broad general reasoning in LLMs with harder replacements for each… | Knowledge |
| B | BigCodeBench-HardBigCodeBench-Hard measures LLM code generation on a 148-task subset of BigCodeBench aligned with real-world… | Coding |
| B | BLINKBLINK measures core visual perception abilities of multimodal LLMs via 3,807 multiple-choice questions… | Multimodal |
| B | BLURBBLURB measures biomedical natural language understanding and reasoning across thirteen PubMed-based datasets… | Healthcare |
| B | BOLDBOLD measures the tendency of language models to produce biased completions across demographic groups. | Generalization |
| B | BTZSCBTZSC measures zero-shot text classification performance (macro F1) of cross-encoders, embedding models,… | Classification |
| B | C4STYLIC4STYLI measures LLMs' recognition and generation of culturally resonant aesthetic stylistics in translated… | Language |
| B | CalArena: A Large-Scale Post-Hoc Calibration BenchmarkCalArena is a standardized benchmark of nearly 2000 experiments that compares post-hoc calibration methods for… | Classification |
| B | CareQA-VisionCareQA-Vision measures vision-language models on medical and nursing image-based questions drawn from Spain's… | Healthcare |
| B | CC-OCR V2CC-OCR V2 measures how well large multimodal models handle real-world document OCR tasks: text recognition,… | Ocr |
| B | CFMMECFMME evaluates large vision-language models on perception, understanding, reasoning and cognition across… | Finance |
| B | CG-BenchCG-Bench measures multimodal LLMs' long-video understanding through clue-grounded question answering, scoring… | Video |
| B | ChartBenchChartBench measures multimodal LLMs' chart comprehension and data reliability through complex visual reasoning… | Knowledge |
| B | Clembench Text v3.0Clembench Text v3.0 scores chat-optimized language models as conversational agents playing text games,… | Agentic |
| B | ClinicalMCClinicalMC measures LLM performance on multi-course clinical decision-making as a patient's condition evolves… | Healthcare |
| B | ComplexFuncBenchComplexFuncBench measures LLMs' ability to perform multi-step and constrained function calling across five… | Tool Use |
| B | CrossViewBenchCrossViewBench measures multimodal LLMs' cross-view spatial understanding across correspondence,… | Spatial Reasoning |
| B | CSimpleQAChinese SimpleQA measures the factuality of language models answering short Chinese factual questions across 6… | General Knowledge |
| B | CULTURE-MTCULTURE-MT measures how well models translate social media user-generated content while preserving cultural… | Translation |
| B | CURE-MedCUREMED-BENCH measures the logical correctness and language consistency of LLMs on open-ended, verifiable… | Healthcare |
| B | CVGEvalKitCVGEvalKit evaluates language models on criminal court view generation from case facts and charge prediction… | Legal |
| B | DataBenchDataBench measures how well LLMs answer questions over real-world tabular data, using 80 datasets and 1,822… | Data |
| B | DatBenchDatBench measures vision-language model capabilities across nine areas (charts, counting, documents, general… | Multimodal Understanding |
| B | DeepTumorVQADeepTumorVQA measures how well medical vision-language models and tool-augmented agents answer 3D abdominal CT… | Biomedicine Healthcare |
| B | Defects4JDefects4J is a collection of 854 reproducible real-world Java bugs with supporting infrastructure, used to… | Coding |
| B | DermoBenchDermoBench evaluates multimodal LLMs on 11 dermatology tasks across morphology, diagnosis, reasoning, and… | Healthcare |
| B | DiffCap-BenchDiffCap-Bench evaluates multimodal large language models on image difference captioning, describing… | Vision |
| B | DisastQADisastQA measures how accurately LLMs answer multiple-choice and open-ended disaster-management questions… | Question Answering |
| B | DIYHealth SuiteDIYHealth Suite includes DIYHealthBench, a benchmark evaluating foundation models on 11 home-based health care… | Healthcare |
| B | DocAtlasDocAtlas-Bench measures multilingual document OCR and understanding (text, tables, formulas, reading order)… | Document Understanding |
| B | EComStageEComStage evaluates LLMs as e-commerce agents across stage-wise reasoning (perception, planning, action) on… | Business |
| B | EgoMemReasonEgoMemReason measures how well multimodal models and agentic frameworks reason over week-long egocentric video… | Reasoning |
| B | EgoSchemaEgoSchema measures very long-form video understanding through multiple-choice questions on three-minute… | Knowledge |
| B | ENAMELENAMEL measures how efficient LLM-generated code is, using HumanEval-derived problems, expert reference… | Coding |
| B | EnterpriseArenaEnterpriseArena measures LLM agents' ability to perform long-horizon resource allocation under uncertainty as… | Finance Business |
| B | EpiQALEpiQAL measures how well LLMs answer epidemiological questions grounded in research articles, testing factual… | Healthcare |
| B | EstGraphEstGraph measures LLMs' reasoning ability to estimate properties of very large, partially accessible graphs… | Reasoning |
| B | Facet-ProbeFacet-Probe audits whether multimodal large language models change their answers when order-irrelevant… | Multimodal |
| B | Fiction.LiveBenchFiction.LiveBench measures how well models comprehend and reason over fiction/narrative text, with models… | Long Context |
| B | Fin-RATEFin-RATE measures how well LLMs perform financial analyst tasks over U.S. SEC filings, including detail… | Finance |
| B | FinChainFinChain evaluates models' financial chain-of-thought reasoning, scoring them with ChainEval plus ROUGE and… | Finance |
| B | FinEvalFinEval measures large language models' Chinese financial domain knowledge and practical abilities across… | Finance |
| B | FraudBenchFraudBench measures how well MLLMs, AI-generated image detectors, and humans detect AI-generated fake damage… | Multimodal |
| B | FreshQAFreshQA is a regularly updated question-answering benchmark that measures the factuality of LLM answers on… | Generalization |
| B | FullStackBench enFullStack Bench measures LLMs' full-stack coding ability across application domains and 16 programming… | Coding |
| B | FullStackBench zhFullStackBench zh is the Chinese subset of FullStack Bench, which evaluates LLMs on full-stack programming… | Coding |
| B | FutureOmniFutureOmni measures how well multimodal LLMs forecast future events from audio-visual video context via… | Multimodal Understanding |
| B | GenAI-BenchGenAI-Bench measures how well multimodal LLMs judge AI-generated images, image edits and videos, scored by… | Multimodal |
| B | GeneBenchGeneBench measures whether AI agents can carry out realistic multi-stage scientific data analysis in genetics… | Science |
| B | GEO-Bench 2GEO-Bench-2 evaluates geospatial foundation models on Earth Observation tasks including classification,… | Vision |
| B | GeoCode LeaderboardA leaderboard scoring large language models on geospatial code generation for Google Earth Engine using… | Geospatial |
| B | GeoSym-BenchGeoSym-Bench is an expert-curated set of 511 complex multimodal geometry problems for rigorously evaluating… | Math |
| B | GIFT-EvalGIFT-Eval measures zero-shot time series forecasting performance of foundation and other models across 23… | Data |
| B | GPF-LiveNewsGPF-LiveNews is a streaming evaluation protocol and benchmark that audits group-conditioned framing bias in… | Factuality |
| B | CORE-Bench HardCORE-Bench Hard measures whether AI agents can computationally reproduce published scientific papers' results… | Coding |
| B | Online Mind2WebOnline Mind2Web measures how well web agents complete 300 realistic tasks on 136 live websites, scored with… | Agentic |
| B | Hallucinations LeaderboardThe Hallucinations Leaderboard evaluates LLMs on hallucination-related benchmarks covering factuality and… | Behavioral |
| B | HalluScoreHalluScore is an Arabic question answering benchmark of 827 questions that evaluates hallucination behavior in… | Factuality |
| B | HardMTBenchHardMTBench measures bidirectional Chinese-English machine translation quality on difficult,… | Translation |
| B | HardSecBenchHardSecBench measures whether LLMs generate secure hardware code, scoring functional and security correctness… | Cybersecurity |
| B | HarmVideoBenchHarmVideoBench measures how well large multimodal models understand harmful videos across observable evidence,… | Safety |
| B | Harvey LAB-AAHarvey LAB-AA measures how well AI agents complete real-world legal work across 120 private tasks in 24… | Legal |
| B | HealthBench-BR and PCDT-QAHealthBench-BR and PCDT-QA measure LLM recall of Brazilian SUS official clinical guideline knowledge via… | Biomedicine Healthcare |
| B | HEIM: Holistic Evaluation of Text-to-Image ModelsHEIM evaluates text-to-image generation models across 12 aspects including image-text alignment, quality,… | Creative Media Image Generation |
| B | HMMT February 2026Measures LLM performance on final-answer math problems from the February 2026 Harvard-MIT Mathematics… | Math |
| B | HREFHREF measures how well language models follow instructions, using human response-guided automatic evaluation… | Instruction Following |
| B | HUMAINEHUMAINE is a leaderboard ranking LLMs based on detailed human feedback about their performance in real-world… | Human-preference |
| B | Hunt Instead of WaitHunt Instead of Wait introduces DDR-Bench, a checklist-based benchmark measuring how well LLM agents… | Agentic |
| B | HyperTrackHyperTrack evaluates vision-language model agents on offline mobile GUI navigation across over 16000… | GUI |
| B | IdioLinkIdioLink measures whether embedding/retrieval models can link idiomatic expressions to conceptually equivalent… | Retrieval |
| B | IFMTBenchIFMTBench measures how well models follow translation instructions (format preservation, glossaries, context,… | Multilingual |
| B | IHBenchIHBench measures how well voice agents recover after user interruptions in multi-step, state-machine-driven… | Speech Healthcare+1 |
| B | INCLUDE-base-44 European LanguagesMeasures multilingual LLMs' regional knowledge and reasoning via multiple-choice questions from local exams,… | Multilingual |
| B | IndoBiasIndoBias measures social bias and stereotypes of LLMs in Indonesian and three local languages (Javanese,… | Safety |
| B | IndustryBenchIndustryBench measures LLM knowledge and safety-aware correctness on industrial procurement QA grounded in… | Domain Business |
| B | InfiniteBM Heads-Up No-Limit Hold'emA head-to-head LLM arena ladder that rates models with Bradley-Terry Elo on heads-up no-limit hold'em poker… | Game |
| B | InfographicVQAInfographicVQA measures models' ability to answer natural language questions about infographic images,… | Knowledge |
| B | IntentGraspIntentGrasp measures how well LLMs understand the intent behind speech, conversation, and writing, using test… | Knowledge |
| B | Inverse Turing BenchInverse Turing Bench measures how accurately models can tell which of two paired multi-turn dialogue… | Classification |
| B | ITBench-AAITBench-AA measures AI agents' ability to perform Kubernetes incident root-cause analysis (SRE) from offline… | Agentic |
| B | J1-ENVSJ1-ENVS measures how well LLM agents perform interactive Chinese legal tasks, from consultation and document… | Legal |
| B | JMed48kJMed48k measures how well vision-language models answer Japanese national healthcare licensing exam questions,… | Biomedicine Healthcare |
| B | K-12EduBenchK-12EduBench evaluates LLMs' subject-level knowledge, subject-specific problem solving, and educational goal… | Education |
| B | Korean Canonical Legal BenchmarkKCL measures language models' Korean legal reasoning independently of domain knowledge, using multiple-choice… | Legal |
| B | Ko-WideSearchKo-WideSearch measures how well web agents exhaustively enumerate the members of a closed set from Korean web… | Agentic |
| B | LegalCiteBenchLegalCiteBench measures how reliably language models recover, complete, verify and match U.S. case law… | Legal |
| B | LingLanMiDianLingLanMiDian (LingLan) evaluates large language models on Traditional Chinese Medicine knowledge recall,… | Healthcare |
| B | LingOly-TOOLingOly-TOO measures language models' reasoning on obfuscated Linguistics Olympiad problems, separating… | Knowledge |
| B | LiveMedBenchLiveMedBench evaluates large language models on time-stamped real-world medical consultation cases using… | Healthcare |
| B | LLM-WikiRaceLLM-WikiRace measures how well language models plan and reason over world knowledge by navigating Wikipedia… | Agentic |
| B | LOGICAL-COMMONSENSEQALOGICAL-COMMONSENSEQA measures whether language models can judge the joint plausibility of pairs of… | Reasoning |
| B | LogicScoreLogicScore evaluates the global logical reasoning integrity (completeness, essentiality, determinateness) of… | Reasoning |
| B | LongBench ProLongBench Pro measures long-context understanding of LLMs on 1,500 natural bilingual (English and Chinese)… | Long Context |
| B | LongVideoBenchLongVideoBench measures how well large multimodal models retrieve and reason over detailed information in… | Long Context |
| B | M3MAD-BenchM3MAD-Bench evaluates multi-agent debate methods across knowledge, mathematics, medicine, natural sciences and… | Agentic |
| B | MacAgentBenchMacAgentBench measures how well computer use agents complete 676 real-world tasks across 25 macOS… | GUI |
| B | Magis-BenchMagis-Bench measures LLMs on magistrate-level legal writing tasks, including discursive legal analysis and… | Legal |
| B | Math-VRMath-VR measures how well VLMs, unified models and LLMs solve bilingual math problems requiring visual… | Multimodal |
| B | MechVQAMechVQA measures how well multimodal LLMs understand mechanical engineering drawings across recognition,… | Document Understanding |
| B | Med-RewardBenchMed-RewardBench measures how well reward models and judges evaluate medical multimodal LLM responses across… | Biomedicine Healthcare |
| B | MedAgentGymMedAgentGym measures LLM agents' code-centric reasoning on biomedical data science tasks in executable sandbox… | Biomedicine |
| B | MedConsultBenchMedConsultBench evaluates medical consultation agents across the full online consultation cycle, from history… | Healthcare |
| B | MedCTAMedCTA measures how well multimodal medical tool agents select and use clinical tools across 107… | Healthcare |
| B | MedGUIDEMedGUIDE measures whether LLMs can make clinical decisions consistent with NCCN cancer guideline decision… | Healthcare |
| B | MedLayXPlainMedLayXPlain measures whether medical vision-language models can generate accurate expert and… | Biomedicine Healthcare |
| B | Memory Architectures for Multi-Turn Text-to-SQLEnterpriseMem-Bench measures how well models with different memory architectures answer multi-turn Text-to-SQL… | Text TO Sql Business |
| B | Merge-BenchMerge-Bench measures how well language models resolve real-world version control merge conflicts, using 7938… | Coding |
| B | MHGraphBenchMHGraphBench measures how well LLMs recognize mental-health entities, judge relations, and perform two-hop… | Biomedicine Healthcare |
| B | MindEdit-BenchMindEdit-Bench measures whether vision-language models can perform object-level counterfactual spatial… | Spatial Reasoning |
| B | MineExplorerMineExplorer measures how well multimodal LLM agents sustain open-world exploration in Minecraft, using… | Game |
| B | MINTEvalMINTEval measures how well memory-augmented agents and long-context LLMs recall and aggregate information over… | Agentic |
| B | MixReaMixRea measures whether LLMs attend to subtle implicit contextual cues while following explicit task… | Reasoning |
| B | MMBench-VideoMMBench-Video measures large vision-language models' video understanding, including temporal reasoning, using… | Multimodal |
| B | MMDocBenchMMDocBench measures fine-grained visual perception, reasoning and grounding of large vision-language models on… | Document Understanding |
| B | MMEBMMEB measures universal multimodal embedding models across classification, visual question answering,… | Multimodal |
| B | MMGistMMGist is a curated vision-language benchmark of 7,262 items across seven capability dimensions that measures… | Multimodal |
| B | MMTUMMTU measures how well models understand, reason over, and manipulate real tables across 25 expert-level table… | Data |
| B | MolDeToxMolDeTox evaluates LLMs and VLMs on stepwise fragment-level editing of molecules to reduce toxicity (molecular… | Biomedicine |
| B | MolRecBench-WildMolRecBench-Wild measures how well models translate real-world molecular diagrams from chemistry papers into… | Chemistry |
| B | MPDocBench-ParseMPDocBench-Parse measures how well models parse multi-page, visually rich documents in English and Chinese… | Document-ai |
| B | MRBenchMRBench measures grain blindness in universal multimodal retrieval models, scoring how well they handle… | Retrieval |
| B | MSRVTT-QAMSRVTT-QA measures how well models answer natural-language questions about the content of short videos drawn… | Knowledge |
| B | MT-BenchMT-Bench measures chat assistants' quality on challenging multi-turn open-ended questions, scored by strong… | Generalization |
| B | MT-JailBenchMT-JailBench is a modular evaluation framework that benchmarks multi-turn jailbreak attacks on large language… | Safety |
| B | MTRefSeg-21KMTRefSeg-21K measures how well vision-language models segment language-described temporal changes across… | Vision |
| B | MuirBenchMuirBench measures robust multi-image understanding of multimodal LLMs via 2,600 multiple-choice questions… | Multimodal |
| B | MulTaBenchMulTaBench measures predictive performance of tabular learners on 40 multimodal tabular datasets combining… | Tabular |
| B | Multi-LCBMulti-LCB measures LLM code-generation ability on contamination-aware competitive programming tasks across… | Coding |
| B | MultiCaptionMultiCaption measures how well models detect contradictions between multilingual claims about the same image… | Multimodal Understanding |
| B | MultiSoc-4DMultiSoc-4D is a Bengali social media benchmark of 58K+ comments annotated for category, sentiment, hate… | Human-preference |
| B | MuseBenchMuseBench measures how well multimodal LLMs understand creative intent in audiovisual arts (cinema, visual… | Creative Media |
| B | Natural2CodeNatural2Code is a code-generation benchmark that scores models on writing code from natural-language prompts;… | Coding |
| B | NeuroAtlasNeuroAtlas measures how well EEG and time-series foundation models perform on clinical EEG (epilepsy, sleep,… | Biomedicine Healthcare |
| B | OccuBenchOccuBench measures how well AI agents complete professional occupational tasks across 10 industries and 65… | Domain |
| B | OCR-RobustOCR-Robust measures how robust vision-language models' OCR reasoning is under controlled visual perturbations… | Ocr |
| B | OCRBenchOCRBench measures the OCR capabilities of large multimodal models across text recognition, scene-text VQA,… | Ocr |
| B | OCRROCRR measures how quickly classification systems recover novel-class accuracy and retain original-distribution… | Classification |
| B | OJBenchOJBench measures competitive-level code reasoning of LLMs using 232 programming competition problems from NOI… | Knowledge |
| B | OmniGAIAOmniGAIA measures how well omni-modal AI agents solve multi-hop tasks requiring deep reasoning and multi-turn… | Agentic |
| B | OPT-BENCHOPT-BENCH measures whether LLM agents can iteratively self-improve solutions from environmental feedback… | Self Improvement |
| B | OR-SpaceOR-Space measures whether LLM agents can build, revise, and explain industrial operations research… | Agentic Business |
| B | P3B3P3B3 measures LLM bias toward European versus Brazilian Portuguese varieties and controllability of variety in… | Multilingual |
| B | PAGE BenchPAGE Bench measures GUI agents' ability to perform point-precise geometric construction actions in continuous… | GUI |
| B | PALM-BenchPALM-Bench evaluates large audio-language models on personalized tasks: recognizing personal concepts (e.g.,… | Audio |
| B | PandaBenchPandaBench measures LLM safety against jailbreak attacks, evaluating attack success rates across 49 LLMs under… | Safety |
| B | Patient-Similarity Cohort Reasoning in Clinical Text-to-SQLCLINSQL measures how well language models generate executable SQL for clinical questions over MIMIC-IV EHR… | Text TO Sql |
| B | PEC-HomePEC-Home is a simulated smart-home dataset that scores how well LLM home assistants interpret progressively… | Language-understanding |
| B | Perception or PrejudiceMeasures whether multimodal LLMs perceive Big Five personality from videos through grounded behavioral… | Behavioral |
| B | From Perception to ActionCHAIN is an interactive 3D physics-driven benchmark that evaluates whether vision-language models can… | Embodied |
| B | PerfCodeBenchPerfCodeBench measures whether LLMs can produce correct and efficient system-level high-performance code… | Coding |
| B | PersistBenchPersistBench measures safety risks of long-term memory in LLM assistants, namely cross-domain leakage and… | Safety |
| B | PersonaArenaPersonaArena evaluates LLMs' persona-level role-playing through multi-turn simulated social interactions… | Agentic |
| B | Physical AI Bench GenerationPhysical AI Bench's generation track evaluates video generative models on predicting physically plausible… | Multimodal |
| B | Physical AI Bench UnderstandingPhysical AI Bench (PAI-Bench) evaluates multimodal LLMs and video generative models on perception and… | Multimodal |
| B | PhysicsFinalsPHYSICS (PhysicsFinals) measures foundation models' ability to solve 1,297 expert-annotated university-level… | Science |
| B | PIQAPIQA measures whether language models can answer physical commonsense questions, choosing the right solution… | Language |
| B | PL-MTEBPL-MTEB evaluates text embedding models on Polish-language classification, clustering, pair classification,… | Language |
| B | PlanningBenchPlanningBench measures LLMs' ability to produce complete, verifiable plans for realistic planning problems… | Reasoning |
| B | PlantMarkerBenchPlantMarkerBench measures how well language models interpret literature-grounded evidence for… | Biology |
| B | PLawBenchPLawBench evaluates LLMs on realistic legal practice tasks (legal consultation, case analysis, document… | Legal |
| B | PodBenchPodBench measures how well LLMs generate podcast scripts from long source documents under complex… | Audio |
| B | POLAR-BenchPOLAR-Bench measures privacy-utility trade-offs of LLM agents that must follow a user's privacy policy while… | Agentic |
| B | PolySpeech-100PolySpeech-100 measures native-level speech comprehension of Speech-LLMs across 110 languages and dialects,… | Speech |
| B | ProactBenchProactBench measures conversational proactivity: whether LLM assistants notice and act on user needs that are… | Agentic |
| B | PROBE Binding AffinityPROBE Binding Affinity measures how well protein representation methods estimate protein-protein binding… | Science |
| B | PROBE Function PredictionPROBE Function Prediction measures how well protein representation models predict ontology-based protein… | Science |
| B | ProsaProsa measures LLM quality on 1,000 real multi-turn Brazilian Portuguese user chats from WildChat, scored with… | Language |
| B | PTEBPTEB (Paraphrasing Text Embedding Benchmark) evaluates the robustness of sentence embedding models by… | Language |
| B | PubMedQAPubMedQA measures a model's ability to answer biomedical research questions with yes/no/maybe using the… | Biomedicine Healthcare |
| B | Qiskit QuantumKatasQiskit QuantumKatas measures how well LLMs write Qiskit quantum computing code across 350 tasks spanning… | Coding |
| B | RankJudgeRankJudge evaluates and ranks LLM judges on their ability to identify the worse of paired multi-turn,… | Agentic |
| B | RealityTestRealityTest measures whether AI text and speech models disclose that they are AI when users probe their… | Safety |
| B | RealStreamEvalRealStreamEval is a frame-level multi-turn evaluation protocol that scores video-language models on streaming… | Video |
| B | ReasonTabQAReasonTabQA measures LLM table question answering and reasoning over complex industrial tables with… | Tabular |
| B | RedactionBenchRedactionBench measures how well models redact personally identifiable information in context, across 200… | Safety |
| B | ReEfBench: Quantifying the Reasoning Efficiency of LLMsReEfBench evaluates the reasoning efficiency of LLMs via a neuro-symbolic, process-centric evaluation of… | Efficiency |
| B | RepoQARepoQA measures LLMs' long-context code understanding by having them find functions in repositories from… | Coding |
| B | RIFT-Bench: Dynamic Red-teaming For Agentic AI SystemsRIFT-Bench evaluates the security of agentic AI systems through automated dynamic red-teaming, discovering… | Agentic |
| B | RPC-BenchRPC-Bench measures how well foundation models comprehend computer science research papers through 15K… | Document Understanding |
| B | RTEBRTEB measures the retrieval accuracy of embedding models on open and private datasets across enterprise… | Retrieval |
| B | RTL-OPTRTL-OPT measures how well LLMs optimize RTL hardware code for power, performance, and area while preserving… | Coding |
| B | RWGBenchRWGBench measures how well systems generate related work sections for computer science papers, scoring… | Writing |
| B | RxEvalRxEval measures LLM inpatient prescribing capability via multiple-choice questions requiring selection of… | Healthcare |
| B | SaaSBenchSaaS-Bench measures whether computer-use agents can complete long-horizon professional workflows by driving a… | Software Engineering Business |
| B | SABER-MathSABER-Math measures how well retrieval and embedding models rerank documents by fine-grained mathematical… | Retrieval |
| B | SaLADSaLAD measures whether multimodal LLMs give safe, informative safety-warning responses to 2,013 real-world… | Safety |
| B | SceneFunRISceneFunRI measures whether vision-language models can infer the 2D locations of invisible (occluded)… | Spatial Reasoning |
| B | SciHorizon-GENESciHorizon-GENE measures how well LLMs reason from gene-level knowledge to functional understanding,… | Biology |
| B | ScreenSpotScreenSpot measures GUI grounding: locating the screen element matching a natural-language instruction across… | Agentic |
| B | Script SensitivityScript Sensitivity benchmarks open-source language models via perplexity on Unicode, Romanized and… | Multilingual |
| B | SDGBiasBenchSDGBiasBench measures decision-level and estimation-level biases of vision-language models in Sustainable… | Reasoning |
| B | Seal-0Seal-0 measures factual accuracy and reasoning of search-augmented language models on the hardest fact-seeking… | Search |
| B | SeePhys ProSeePhys Pro measures whether multimodal models keep the same physics reasoning ability as critical problem… | Reasoning |
| B | Sefaria Rabbinic Embedding LeaderboardMeasures how well embedding models retrieve and align Rabbinic Hebrew/Aramaic texts with their English… | Retrieval |
| B | SenseBenchSenseBench measures how well vision-language models perceive and describe low-level degradations in remote… | Vision |
| B | ShoppingMMLUShopping MMLU measures how well LLMs act as online shop assistants across 57 Amazon-derived tasks covering… | Business |
| B | SiT-BenchSiT-Bench measures the spatial intelligence of LLMs from coordinate-aware textual scene descriptions, without… | Spatial Reasoning |
| B | SkillRetSkillRet measures how well retrievers select the right skill from large libraries of public agent skills given… | Retrieval |
| B | SonicBenchSonicBench measures whether large audio language models can perceive 12 fundamental physical audio attributes… | Audio |
| B | SpaceDGSpaceDG measures the robustness of multimodal LLMs' spatial reasoning on indoor-scene visual questions under… | Spatial Reasoning |
| B | SpatialUAVSpatialUAV measures spatial intelligence of vision-language models on real low-altitude UAV imagery, covering… | Spatial Reasoning |
| B | Spider 2.0Spider 2.0 measures how well language models and agents solve real-world enterprise text-to-SQL workflow… | Data Business |
| B | StakeBenchStakeBench evaluates whether LLMs understand language grounded in market commitment, using prediction-market… | Finance |
| B | STALESTALE measures whether LLM agents detect and act on implicit conflicts where later observations invalidate… | Agentic |
| B | StemBindStemBind is a diagnostic benchmark that measures where multimodal LLMs fail in abstract visual reasoning by… | Reasoning |
| B | STEMVerseSTEMVerse is a diagnostic evaluation framework that scores LLMs' STEM reasoning across over 20,000 problems… | Reasoning |
| B | StereoTalesStereoTales measures social bias and harmful stereotypes emerging in open-ended LLM story generation across 10… | Safety |
| B | Stick To Your Role!Stick To Your Role! measures how stably LLMs maintain simulated personas' personal values across many… | Alignment |
| B | STT-ArenaSTT-Arena measures whether LLM agents can detect mid-task spatio-temporal disruptions and adaptively replan in… | Agentic |
| B | SwordBenchSwordBench evaluates methods for steering image representations of vision models in concept removal tasks,… | Vision |
| B | SycoEval-EMSycoEval-EM measures whether LLMs acquiesce to adversarial patient persuasion for guideline-discordant care in… | Healthcare |
| B | TableVistaTableVista measures foundation models' multimodal table reasoning across 30,000 rendered table images that… | Reasoning |
| B | TADDLE BenchmarkAn expert-annotated benchmark of 1,800 LLM-generated peer reviews on 50 ICLR 2025 papers, scoring systems on… | Classification |
| B | Tau2 AirlineTau2 Airline is the airline customer-service domain of τ2-bench, which evaluates conversational agents that… | Agentic Business |
| B | TAU3-Benchτ³-Bench evaluates conversational customer-support agents completing multi-step tasks with tools, including… | Agentic |
| B | TheoremQATheoremQA measures how well AI models apply university-level theorems to solve challenging problems in math,… | Math |
| B | TIDETIDE is an agent-agnostic, environment-agnostic diagnostic evaluation framework that scores LLM agents'… | Agentic |
| B | TOBenchTOBench (also called MM-ToolBench) measures omni-modal tool-using agents on 100 executable MCP-based tasks in… | Tool Use Business |
| B | Trip+Trip+ measures how well language-model agents plan and revise personalized, minute-level travel itineraries… | Agentic |
| B | TriViewBenchTriViewBench measures how multimodal LLMs handle multi-view structural visual reasoning (local decisions,… | Reasoning |
| B | Trust No ToolTRUST-Bench measures whether LLM agents and defenses can detect and avoid harmful final actions when a… | Tool Use |
| B | TuRTLe Code CompletionTuRTLe Code Completion evaluates LLMs on completing RTL (Verilog) hardware code, scoring syntax,… | Coding |
| B | TuRTLe Line CompletionTuRTLe Line Completion scores LLMs on completing lines of RTL (Verilog) hardware design code, as one task in… | Coding |
| B | TuRTLe Module CompletionTuRTLe Module Completion scores LLMs on completing RTL hardware modules drawn from over 1000 real-world Tiny… | Coding |
| B | TuRTLe Spec-to-RTLTuRTLe Spec-to-RTL scores LLMs on generating hardware RTL (Verilog) from specifications, measuring syntax,… | Coding |
| B | UAVBenchUAVBench measures LLM reasoning for autonomous UAV systems using 50,000 validated flight scenarios and 50,000… | Agentic |
| B | UbuntuGuardUbuntuGuard measures how well guardian models and LLMs enforce culturally grounded, context-specific safety… | Safety |
| B | Uncertainty Quantification for Computer-Use AgentsArgus benchmarks post-hoc uncertainty quantification methods for vision-language computer-use agents on… | GUI |
| B | URIAL BenchURIAL Bench scores base (untuned) language models aligned only through in-context prompting on multi-turn… | Instruction Following |
| B | Poker AgentPoker Agent measures how well language models play 10-seat no-limit Texas Hold'em against each other, ranked… | Agentic |
| B | VAREX-BenchVAREX measures how accurately multimodal foundation models extract structured JSON data from government forms… | Document Understanding |
| B | VerdictBenchVerdictBench measures how well models adjudicate e-commerce transaction disputes from multi-round, multimodal… | Agentic |
| B | VibeCodingBenchVibeCodingBench evaluates AI coding agents on production-oriented full-stack development tasks, scoring… | Coding |
| B | Video-HolmesVideo-Holmes measures multimodal LLMs' complex video reasoning by requiring them to locate and connect visual… | Multimodal |
| B | Video Understanding Reward ModelingVURB measures how well reward models judge preference pairs of responses to video understanding tasks,… | Video |
| B | VisualWebBenchVisualWebBench measures how well multimodal LLMs understand and ground web pages across seven tasks built from… | Coding |
| B | VitaBench 2.0VitaBench 2.0 measures how well LLM agents infer, use and update user preferences and act proactively across… | Agentic |
| B | VLMbench PerformanceVLMbench Performance measures inference throughput and latency (tokens/sec, TTFT, TPOT) of vision-language… | Efficiency |
| B | VoiceBenchVoiceBench measures LLM-based voice assistants on spoken instructions across knowledge, instruction following,… | Speech |
| B | VRBenchVRBench evaluates large models' multi-step reasoning over long narrative videos, scoring both final… | Multimodal |
| B | What Users Leave UnsaidHAERAE-Vision measures how well vision-language models answer real-world, under-specified visual questions… | Multimodal Understanding |
| B | When Seekers Are Hard to HelpEvaluates emotional support dialogue systems under worst-case interactions with hard-to-help simulated… | Reasoning |
| B | When Simulation LiesRobustBench-TC (from the paper When Simulation Lies) measures the robustness of tool-use language agents under… | Tool Use |
| B | WikiSQLWikiSQL measures how accurately models translate natural language questions into SQL queries over relational… | Knowledge |
| B | WikiVQABenchWikiVQABench measures how well vision-language models answer knowledge-grounded multiple-choice visual… | Multimodal |
| B | WildRoadBenchWildRoadBench measures how well vision-language models and autonomous LLM agents can localise road damage in… | Vision |
| B | WMT24++WMT24++ measures machine translation quality of MT systems and LLMs from English into 55 languages and… | Translation |
| B | Your One-Stop Solution for AI-Generated Video DetectionAIGVDBench measures how well detectors and vision-language models can distinguish AI-generated videos from… | Video |
| B | ZEROBench-SubZEROBench-Sub is a subset of ZeroBench, a difficult visual reasoning benchmark that tests large multimodal… | Multimodal |
| B | Avalon-ToM-BenchAvalon-ToM-Bench measures fine-grained Theory of Mind in language models through perspective-constrained… | |
| B | ClawTrackClawTrack evaluates LLM-based autonomous agents on 320 multi-step tasks across 8 domains, scoring both task… | |
| B | EASELEASEL measures dexterous visual tool use in multimodal agents, chiefly via reference-guided painting of a… | |
| B | Gestalt Grouping behavioral batteryA behavioral battery that scores vision models against human perception data on four Gestalt grouping tasks… | |
| B | MedPRESSMedPRESS measures whether LLMs abandon safe medical advice and sycophantically agree with patients under… | |
| B | MemeBenchMemeBench measures how well large vision-language models interpret culture-dependent Chinese and English… | |
| B | MMJailBenchMMJailBench measures the jailbreak vulnerability of multimodal large language models by systematically varying… | |
| B | quantum-api-driftMeasures whether LLMs generate quantum SDK (Qiskit) code that executes correctly on the specific requested SDK… | |
| B | RRS-10KRRS-10K measures how well vision-language models interpret rare, military-related remote sensing images across… | |
| B | From Sounds to ScenesMeasures whether large audio language models can understand auditory scenes by integrating speech, acoustic… | Audio |
| B | FINESSE-BenchFINESSE-Bench evaluates LLMs' financial domain knowledge and technical analysis across eight hierarchical… | Finance |
| B | Overview of the PsyDefDetect Shared Task at BioNLP 2026PsyDefDetect is a shared task that scores systems on classifying help-seeker utterances in emotional support… | Biomedicine |
| B | PhysicsMindPhysicsMind measures whether multimodal LLMs and video world models reason about and generate motion… | Science |
| C | Terminal-Bench 2.0 | Long-Horizon |
| C | SWE-Bench Pro v1 (2025-09-19) and v2SWE-Bench Pro measures whether AI agents can solve long-horizon software engineering tasks. | Coding Computer Use+2 |
| C | WebArenaWebArena measures whether autonomous language agents can complete diverse, long-horizon tasks on self-hosted,… | Agentic |
| C | MMLUMMLU measures a text model's multitask accuracy across 57 multiple-choice subjects, testing world knowledge… | Knowledge |
| C | Multi-SWE-BenchMulti-SWE-bench measures how well LLMs resolve real software issues by generating patches to codebases across… | Coding |
| C | GDP.xlsx | Enterprise Workflows Long-Horizon |
| C | AIME 2024AIME 2024 measures a model's ability to solve challenging competition mathematics problems from the 2024… | Math |
| C | MMLU-ReduxMMLU-Redux measures broad multitask knowledge of language models using 5,700 manually re-annotated MMLU… | Knowledge |
| C | FrontierMathFrontierMath tests AI models on unpublished, highly challenging mathematics problems ranging from… | Math |
| C | MAGIMAGI measures how well AI agents complete ambiguous, long-horizon enterprise work tasks that interleave GUI… | Computer Use |
| C | SWE-bench LiveSWE-bench Live measures how well LLMs and agents resolve real, recent GitHub issues by generating patches that… | Coding |
| C | SWE-bench FullSWE-bench Full measures whether AI systems can resolve 2,294 real GitHub issues from 12 popular Python… | Coding |
| C | DrugDiscoveryBench with PhyloDrugDiscoveryBench evaluates whether coding agents can assist with early-stage drug discovery tasks. | Coding Computer Use+2 |
| C | READY or Not enterprise agent deployment benchmarksREADY measures whether AI agents can be reliably deployed on enterprise workflows by jointly assessing… | Coding Computer Use+2 |
| C | RSI BenchRSI Bench evaluates whether AI agents can autonomously perform AI R&D tasks such as post-training, alignment,… | Coding Computer Use+2 |
| C | PRBenchPRBench measures high-stakes professional reasoning of language models in legal and finance domains using… | Coding Computer Use+2 |
| C | DROPDROP measures English reading comprehension requiring discrete reasoning (addition, counting, sorting) over… | Reasoning |
| C | BrowseComp Long Context 128kMeasures the ability of AI browsing agents to persistently navigate the internet and locate hard-to-find,… | Search |
| C | BrowseComp Long Context 256kBrowseComp Long Context 256k is a variant of BrowseComp, which measures how well AI agents can browse the web… | Search |
| C | CrafterCrafter is an open-world 2D survival game benchmark that scores agents by success rates on 22 achievements and… | Creative Media |
| C | FrontierMath Tier 4FrontierMath Tier 4 measures model accuracy on 43 exceptionally difficult, research-level mathematics problems… | Math |
| C | SWE-bench Verified MiniSWE-bench Verified Mini measures AI agents' ability to resolve real-world GitHub issues in Python… | Coding |
| C | TAU-bench AirlineTAU-bench Airline evaluates AI agents on airline customer-service tasks, such as changing or finding flights,… | Agentic |
| C | HumanEval-MulHumanEval-Mul measures functional correctness of code generated by models across multiple programming… | Coding |
| C | LiveCodeBench-PlusLiveCodeBench-Plus is a 91-problem coding benchmark of evolved and difficult original LiveCodeBench v6 tasks… | Coding |
| C | MMLU Medical GeneticsMMLU Medical Genetics is the medical genetics subject of the MMLU test, scoring models on multiple-choice… | Healthcare |
| C | MMLU-STEMMMLU-STEM is the STEM subset of MMLU, a multiple-choice test of a text model's multitask accuracy and… | General Knowledge |
| C | MultiMedia-TerminalBenchMMTB evaluates terminal agents on 105 tasks across 5 meta-categories that require directly operating on audio… | Agentic |
| C | OSWorld-MCPOSWorld-MCP measures computer-use agents' tool invocation via MCP, GUI operation, and decision-making on real… | Agentic |
| C | Westworld v1 simulated-web environments (2025-11-23): | Computer Use Enterprise Workflows |
| C | BrowserBench (2025-11-22): | Computer Use Enterprise Workflows |
| C | CyberStrikeCyberStrike measures whether frontier agents can complete realistic long-horizon cybersecurity work—defensive… | Coding Enterprise Workflows+1 |
| C | SciCode++SciCode++ measures models on computationally intensive scientific coding tasks across physics, materials… | Coding Enterprise Workflows+1 |
| C | Open-MM-RLOpen-MM-RL measures PhD-level multimodal STEM reasoning over figures, diagrams and graphs with single… | Coding Enterprise Workflows+1 |
| C | SDLCbenchSDLCbench evaluates models on long-horizon software-engineering work on private enterprise codebases. | Coding Enterprise Workflows+1 |
| C | CompanyBenchCompanyBench evaluates enterprise financial-analyst agents working on a simulated consumer-fintech neobank. | Coding Enterprise Workflows+1 |
| C | EKWbenchEKWbench measures models on multi-hour operational knowledge work such as sourcing reviews, cost roll-ups,… | Coding Enterprise Workflows+1 |
| C | Medical BenchMedical Bench measures longitudinal clinical reasoning over a single dermatology patient's chart read at nine… | Coding Enterprise Workflows+1 |
| C | OdysseysMegaOdysseysMega measures computer-use agents on long-horizon browser tasks on the live web, judged rubric by… | Coding Enterprise Workflows+1 |
| C | SWE-Bench Pro - Private DatasetSWE-Bench Pro Private Dataset measures AI agents' resolve rate on long-horizon software engineering tasks… | Coding Long-Horizon+1 |
| C | gba-eval-attempts and per-model gba-eval-attempt-* reposGBA Eval measures whether frontier AI coding agents can write a complete GBA emulator with WebAssembly support… | Coding Private Codebases |
| C | Legora BARLegora BAR (Benchmark for Agentic Reasoning) evaluates AI agents on end-to-end legal tasks drawn from real… | Coding Computer Use+1 |
| C | AutoResearchExamAutoResearchExam measures how quickly AI agents can improve and generalize hidden-test scores on 29 open-ended… | Long-Horizon |
| C | data-eng-bench | Long-Horizon |
| C | CWE-bench v1 | Coding Computer Use+2 |
| C | TraitBasis / Impatient Users Confuse AI AgentsTraitBasis/τ-Trait measures how robust conversational AI agents are when simulated users show traits like… | Coding Computer Use+2 |
| C | Vending-Bench 2 / Vending-Bench ArenaVending-Bench 2 measures how well AI models stay coherent while running a simulated vending machine business… | Computer Use Long-Horizon |
| C | Blueprint-Bench 2, 2026-05Blueprint-Bench 2 measures AI agents' spatial reasoning by having them convert ~20 apartment interior photos… | Computer Use Long-Horizon |
| C | Drone-Bench, with Anthropic Project Pilot, 2026-07Drone-Bench measures how well AI models can write code to perform a simple surveillance task (reconstruction,… | Computer Use Long-Horizon |
| C | ClawsBenchClawsBench measures the task success and safety of LLM productivity agents working across mock Gmail, Slack,… | Coding Computer Use+1 |
| C | Robo UseRobo Use measures how well coding agents solve robot benchmark tasks by driving a simulated robot from the… | Coding Computer Use+1 |
| C | FrontierPhysics v0.1FrontierPhysics evaluates AI agents on end-to-end frontier physics research tasks. | Coding Computer Use+1 |
| C | Benchmarking Agents in Insurance Underwriting EnvironmentsUNDERWRITE is an expert-designed, multi-turn insurance underwriting benchmark that evaluates AI agents using… | Coding Computer Use+2 |
| C | BigLaw Bench: ResearchBigLaw Bench: Research measures whether AI models can use search tools to research US case law and produce… | Coding Computer Use+2 |
| C | Measuring Autonomous AI ResearchMeasures how well frontier models can autonomously conduct AI research by running multi-day agent runs on the… | Coding Enterprise Workflows+2 |
| C | General Agent synthetic environmentA self-evolving synthetic agent environment of 4,504 stateful database tool-use tasks across 1,040 domains,… | Coding Enterprise Workflows+2 |
| C | Realm: Legal reasoning benchmarkRealm evaluates the legal reasoning ability of AI systems, scoring models on a Best@3 leaderboard. | Enterprise Workflows |
| C | Realm: Financial reasoning benchmarkRealm Financial measures how well AI agents complete 103 expert-authored real-world finance tasks (IFRS… | Enterprise Workflows |
| C | Democratizing Diplomacy harness paperAn evaluation harness that lets any LLM play full-press Diplomacy without fine-tuning, measuring strategic… | Long-Horizon |
| C | Humor / Bad Cards Arena (LOL Arena) leaderboardMeasures humor alignment: how often a model picks the same funniest card as the human judge across thousands… | Long-Horizon |
| C | DAYJOB (Healthcare/Finance) - arXiv:2610.01306DAYJOB measures whether AI agents can complete long-horizon professional healthcare and finance tasks in… | Enterprise Workflows Long-Horizon |
| C | ComplexConstraints and BeyondComplexConstraints is an expert-curated instruction-following benchmark of 75 public prompts scored against… | Enterprise Workflows Long-Horizon |
| C | EnterpriseBench CoreCraft - arXiv:2602.16179EnterpriseBench CoreCraft measures whether AI agents can perform multi-step, domain-specific customer support… | Enterprise Workflows Long-Horizon |
| C | The Hierarchy of Agentic Capabilities - arXiv:2601.09032Evaluates frontier AI models on 150 workplace tasks in a realistic e-commerce RL environment, analyzing… | Enterprise Workflows Long-Horizon |
| C | Antidote and Hemingway-bench leaderboards | Enterprise Workflows Long-Horizon |
| C | OfficeQAOfficeQA measures how well models and agents perform grounded reasoning by finding and combining figures from… | Multimodal |
| C | Westworld Finance Diligence Bench (2026-08-11): | Computer Use Enterprise Workflows |
| C | DealTrace Bench (2026-05/06): | Computer Use Enterprise Workflows |
| C | VAmoS Pro Bench / 'VAmoS Part Deux'VAmoS Pro Bench / 'VAmoS Part Deux' (arXiv 2609.38512, Sep 2026) - https://arxiv.org/abs/2609.38512 | Enterprise Workflows |
| C | VAmoS Bench: Voice Agent Simulation BenchVAmoS Bench: Voice Agent Simulation Bench (arXiv 2607.27453, Jul 2026) - https://arxiv.org/abs/2607.27453 | Enterprise Workflows |
| C | Public leaderboards VAmoS and VAmoS Pro | Enterprise Workflows |
| C | BankingBench | Enterprise Workflows |
| C | SantaBench | Enterprise Workflows |
| C | KellyBenchKellyBench measures how well language-model agents make long-horizon sequential decisions by betting on a… | Coding Long-Horizon |
| C | VADERVADER measures how well LLMs assess, detect, explain, and remediate real-world software vulnerabilities, with… | Coding Computer Use+1 |
| C | Blueprint-BenchBlueprint-Bench measures spatial reasoning of AI models by having them convert apartment photographs into… | Computer Use Long-Horizon |
| C | Butter-BenchButter-Bench measures the practical intelligence of LLMs acting as high-level controllers of robots in the… | Computer Use Long-Horizon |
| C | LibraryDesignBenchLibraryDesignBench measures how well coding agents design full-featured software libraries from… | Coding Computer Use+2 |
| C | SlopCodeBenchSlopCodeBench is a community benchmark measuring code erosion in AI-generated code. | Coding Computer Use+2 |
| C | SWE-Marathon-ExtSWE-Marathon-Ext measures whether AI models can build complete SaaS product clones from scratch given a… | Coding Enterprise Workflows+1 |
| C | Realm: Tax reasoning benchmarkRealm: Tax measures whether AI agents can prepare complete, accurate federal Form 1040 tax returns with… | Enterprise Workflows |
| C | CortexRetrievalBenchCortexRetrievalBench evaluates how well AI retrieval systems find and rank authoritative sources for financial… | Enterprise Workflows |
| C | LongExtractionBenchLongExtractionBench measures how accurately, completely and robustly document-extraction systems and frontier… | Enterprise Workflows |
| C | Diplomacy Arena leaderboardDiplomacy Arena scores models on multi-agent strategic negotiation by having each play 20 games of AI… | Long-Horizon |
| C | AI_Diplomacy open-source harnessAI_Diplomacy is an open-source harness in which AI models play full games of Diplomacy against each other,… | Long-Horizon |
| C | Chartography - arXiv:2608.10677Chartography measures whether frontier multimodal models can correctly read and reason over professional,… | Enterprise Workflows Long-Horizon |
| C | HANDBOOK.md - arXiv:2607.25398HANDBOOK.md measures whether LLM agents follow long, binding company policy documents (20-124 page SOPs) while… | Enterprise Workflows Long-Horizon |
| C | GDP.pdf - arXiv:2607.11192GDP.pdf measures whether multimodal models can answer realistic, expert-authored questions grounded in… | Enterprise Workflows Long-Horizon |
| C | Riemann-Bench - arXiv:2604.06802Riemann-Bench measures AI systems on private, expert-curated research-level mathematics problems with… | Enterprise Workflows Long-Horizon |
| C | Assemble Bench (robotics | Computer Use Enterprise Workflows |
| C | SheetBench-50 (with Sepal AI) - 2025-10-01SheetBench-50 evaluates agents on 50 financial analyst spreadsheet tasks performed in a browser-based… | Computer Use Enterprise Workflows |
| C | Cua-Bench EDA Leaderboard | Computer Use |
| C | Gemini 3.5 Flash computer-use evaluation on Cua-BenchEvaluates Gemini 3.5 Flash and other frontier models as computer-use agents on Cua-Bench's KiCad EDA suite of… | Computer Use |
| C | Windows Agent ArenaWindows Agent Arena measures how well multi-modal agents complete 150+ realistic tasks by operating… | Computer Use |
| C | IPI ArenaIPI Arena measures how vulnerable LLM agents are to concealed indirect prompt injection attacks across tool… | |
| C | Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration TestingEvaluates AI agents against human cybersecurity professionals on penetration testing of a live university… | |
| C | D-REX deceptive reasoning benchmarkD-REX evaluates whether models and safety mechanisms can detect deceptive reasoning, where a model's internal… | |
| C | Agent Red Teaming (ART) benchmarkART measures how robust LLM-powered AI agents are against curated high-impact prompt-injection attacks that… | |
| C | NIST CAISI blog on the UK AISI x Gray Swan agent red-teaming competitionA large-scale public agent red-teaming competition measuring how often attacks (direct and indirect prompt… | |
| C | BlenderBench | Coding Computer Use+1 |
| C | InsureBench: insurance underwriting and claims benchmark | Coding Computer Use+3 |
| C | CorpLaw (measured 2026-08-18; access on request): | Coding Enterprise Workflows+1 |
| C | AUDITSAUDITS measures how robustly image manipulation detection and localization methods detect diffusion-based… | Vision |
| C | ConvertedCoding agents increasingly act as codebase-scale collaborators that can assist with codebase conversion, but… | Coding |
| C | EnterpriseBenchEnterpriseBench measures how well LLM agents complete 500 tasks in a simulated enterprise environment spanning… | Agentic Business |
| C | Frontier-BenchFrontier-Bench (presented as Terminal-Bench) measures frontier AI agents on a diverse, difficult, evolving set… | Agentic |
| C | FrontierFinanceFrontierFinance measures how well AI systems answer 220 open-ended financial research queries across the… | Finance |
| C | ActBenchActBench measures behavioral safety of cowork agents by scoring execution trajectories on matched benign and… | |
| C | CoREBCoREB measures code retrieval and reranking models across text-to-code, code-to-text, and code-to-code search… | Retrieval |
| C | FinQAFinQA measures models' ability to answer expert-written numerical reasoning questions over financial reports… | Finance |
| C | FRAMESFRAMES measures LLMs' ability in end-to-end retrieval-augmented generation: factual responses, retrieval, and… | Search |
| C | IndicaIndica measures LLMs' knowledge of region-specific cultural commonsense across five Indian regions and their… | General Knowledge |
| C | MOREMORE measures how well OCR and document parsing models recover text, formulas, tables, code blocks, catalogs… | Document Understanding |
| C | NESTNEST measures how well vision-language models understand narrative event structure in full-length movies, via… | Video |
| C | PARSEPARSE measures reasoning-focused open-domain question answering by LLMs in Persian across 10,800 Boolean,… | Reasoning |
| C | PlantLarge Language Model (LLM) agents remain vulnerable to safety threats from the external environment, where… | Safety |
| C | PortBenchPortBench measures how well LLMs perform correlation-aware portfolio management across six asset classes, via… | Finance |
| C | PRIMEPRIME evaluates how LLMs resolve conflicting meta-instructions across response length, output format, and… | Instruction Following |
| C | REALREAL evaluates LLMs' memory, comprehension, reasoning and hallucination in housing transactions and real… | Business |
| C | ROSEROSE measures whether multimodal LLMs can turn fixed visual evidence into context-specific actions via coupled… | Reasoning |
| C | SODESODE evaluates the social dynamics of LLM agents in behavioral game theory settings: direct reciprocity,… | Behavioral |
| C | STAGESTAGE measures models' narrative understanding over full-length movie screenplays via knowledge graph… | Long Context |
| C | TASTETASTE is a designer-annotated multi-dimensional preference dataset for AI-generated graphic design, used to… | Creative |
| C | RFT case study with Savi Security, Qwen3 4B scam detection | Enterprise Workflows |
| C | DDBenchDDBench measures how well LLM coding agents repair historical bugs in open-source distributed systems, with… | |
| C | FinReportBenchFinReportBench measures whether LLMs can generate institution-grade bilingual financial research reports,… | |
| C | ToolFailBenchToolFailBench measures tool-use failure modes of LLM agents (tool-skip, result-ignore, output fabrication,… | |
| C | TREKTREK measures whether tool-using LLM agents can synthesize feasible travel itineraries that are… | |
| C | CybenchCybench measures language model agents' cybersecurity capabilities on 40 professional-level Capture the Flag… | Agentic |
| C | ExploitGymExploitGym measures whether AI agents can extend vulnerability-triggering inputs into working exploits across… | Agentic |
| C | EdgeBenchEdgeBench measures how autonomous AI agents learn from real-world environment feedback over 12+ hour runs… | Agentic |
| C | ExploitBenchMeasures how far AI agents progress along the Chromium V8 exploitation ladder, scoring 16 exploit capability… | Agentic |
| C | CorpusQA 1MCorpusQA 1M measures a model's accuracy at answering questions over roughly million-token long-context… | Reasoning |
| C | CountBenchCountBench is an image-text benchmark that evaluates whether vision-language models understand object counts… | Multimodal |
| C | Design2CodeDesign2Code measures how well multimodal LLMs convert screenshots of real-world webpages into front-end code… | Multimodal |
| C | ContractNLIContractNLI measures document-level natural language inference on contracts: classifying whether hypotheses… | Decision Models |
| C | BelebeleBelebele measures multilingual machine reading comprehension via four-option multiple-choice questions on… | Decision Models |
| C | WebVoyagerWebVoyager measures how well multimodal web agents complete open-ended user tasks end-to-end on 15 popular… | Agentic |
| C | ZClawBenchZClawBench measures how well models perform as general-purpose agents completing end-to-end OpenClaw-style… | Agentic |
| C | SWE-sweepSWE-sweep measures whether coding agents can autonomously discover and fix as many unspecified bugs as… | Agentic |
| C | InferenceBenchInferenceBench measures whether autonomous coding agents can build and optimize an OpenAI-compatible LLM… | Agentic |
| C | MobileWorldMobileWorld measures how well autonomous mobile agents complete long-horizon, cross-application tasks on… | Agentic |
| C | GAIAGAIA measures how well AI assistants answer real-world questions that are simple for humans but require… | Agentic |
| C | Time Horizon Index: KSPMeasures how far an AI agent gets through a fixed ladder of 30 Kerbal Space Program missions when given five… | Agentic |
| C | SRE BenchSRE-Bench measures how well AI agents reverse engineer compiled binaries of private, real-world-scale programs… | Agentic |
| C | Public Benefits Bench v1Public Benefits Bench v1 measures how accurately AI models answer SNAP benefits navigation questions,… | Agentic |
| C | DeepResearch Bench IIDeepResearch Bench II evaluates deep research systems on producing long-form research reports from web… | Agentic |
| C | FLTEvalFLTEval scores models on Lean 4 formal proof tasks derived from the Fermat's Last Theorem project, applying… | Coding |
| C | VIBE-ProVIBE-Pro is a repo-level code generation benchmark that scores models on end-to-end full project delivery… | Coding |
| C | KernelBenchKernelBench measures how well language models write correct, fast GPU kernels (CUDA/Triton), scoring speedup… | Coding |
| C | MRCRv2MRCR v2 measures long-context reasoning by asking models to reproduce a specific instance of assistant output… | Reasoning |
| C | Graphwalks Parents 128KGraphwalks Parents measures whether a language model can, given a long list of directed graph edges in… | Reasoning |
| C | MRCR 1MMRCR 1M measures a model's ability to retrieve and resolve multi-round coreferences across contexts of about… | Reasoning |
| C | ODINW13A visual detection and grounding benchmark slice used to compare zero-shot object understanding across diverse… | Multimodal |
| C | VideoMMMUVideo-MMMU measures large multimodal models' ability to acquire and apply knowledge from expert-level… | Multimodal |
| C | Vision2WebVision2Web measures whether multimodal coding agents can build static webpages, interactive frontends and… | Multimodal |
| C | SimpleVQASimpleVQA measures the factuality of multimodal large language models answering short natural-language… | Multimodal |
| C | AudioMC – Audio OutputAudioMultiChallenge Audio Output evaluates spoken dialogue systems producing native speech on multi-turn… | Multimodal |
| C | ChartQAChartQA measures models' ability to answer complex questions about charts requiring visual and logical… | Multimodal |
| C | InfoVQAInfographicVQA measures visual question answering over infographic images, requiring joint reasoning over… | Multimodal |
| C | TextVQATextVQA measures whether models can read and reason about text in images to answer visual questions. | Multimodal |
| C | AGIEvalAGIEval measures foundation models on human-centric standardized exams such as college entrance exams, law… | Knowledge |
| C | TruthfulQATruthfulQA measures whether language models answer 817 questions across 38 categories truthfully rather than… | Knowledge |
| C | TriviaQATriviaQA measures reading comprehension and question answering on trivia questions authored by enthusiasts,… | Knowledge |
| C | NOVA-63NOVA-63 is a multilingual benchmark evaluating LLM capability across 63 languages. | Multilingual |
| C | PolyMathPolyMATH evaluates multi-modal LLMs' cognitive and mathematical reasoning on 5,000 image-based textual and… | Multilingual |
| C | MAXIFEMaXIFE measures LLMs' instruction-following capabilities in multilingual and cross-lingual settings across 23… | Multilingual |
| C | TabFactTabFact measures whether models can verify if natural-language statements are entailed or refuted by Wikipedia… | Decision Models |
| C | HANDBOOK.md AgentsHANDBOOK.md Agents measures whether agents can follow a long (100-page) company handbook while completing… | Enterprise Workflows Tool-use+1 |
| C | A^3-BenchA3-Bench measures memory-driven scientific reasoning in LLMs, scoring accuracy and anchor/attractor memory… | Science |
| C | A2RBenchA2RBench measures LLMs' abstract reasoning ability, their capacity to extract and apply abstract rules, using… | Reasoning |
| C | ABC-BenchABC-Bench measures whether coding agents can explore real repositories, edit code, configure environments,… | Coding |
| C | AcademiClaw: When Students Set Challenges for AI AgentsAcademiClaw measures how well AI agents complete 80 complex, long-horizon, bilingual academic tasks sourced… | Agentic |
| C | ActiveSciBenchActiveSciBench measures closed-loop scientific discovery with active, budget-constrained experiment selection,… | Science |
| C | AcuityBenchAcuityBench measures whether language models identify the appropriate urgency of medical care from user health… | Healthcare |
| C | AEQ-BenchAEQ-Bench measures the empathy of omni-modal large models, both generating empathetic responses from audio and… | Audio |
| C | AfriEconQAAfriEconQA measures how well LLM and RAG systems answer complex African economic questions requiring numerical… | Finance |
| C | AfriScience-MTAfriScience-MT measures machine translation quality of scientific plain-language summaries from English into… | Translation |
| C | Afrispeech SemanticsAfrispeech Semantics measures how well audio language models perform semantic and paralinguistic reasoning… | Audio |
| C | AfroBenchAfroBench measures how well large language models perform across 64 African languages on 15 tasks spanning… | Multilingual |
| C | AgencyBenchAgencyBench measures autonomous LLM agents on 138 long-horizon real-world tasks across 32 scenarios spanning… | Agentic |
| C | Agent Security LeagueAgent Security League measures whether AI coding agent and model combinations produce functionally correct and… | Coding |
| C | Agent-ValueBenchAgent-ValueBench evaluates the values steering autonomous agents' behavior through value-conflict tasks in… | Alignment |
| C | AgentBoardAgentBoard evaluates LLMs as generalist agents across 9 diverse multi-turn, partially-observable tasks,… | Agentic |
| C | AgentClinicAgentClinic measures how well LLM agents perform sequential diagnostic decision-making in simulated multimodal… | Healthcare |
| C | AgentHijackAgentHijack measures how robust multimodal computer-use agents are on desktop tasks when exposed to nine… | Agentic |
| C | AgentickAgentick measures sequential decision-making agents (RL, LLM, VLM, hybrid, human) on 37 procedurally generated… | Agentic |
| C | AgenticVBenchAgenticVBench measures how well multimodal AI agents complete 100 expert-authored real-world video… | Domain |
| C | AgentLeakAgentLeak measures privacy leakage of sensitive data across internal channels (inter-agent messages, shared… | Safety |
| C | Agentset Reranker LeaderboardAgentset's Reranker Leaderboard evaluates reranking models in a RAG pipeline across six retrieval datasets… | Retrieval |
| C | AgingBenchAgingBench measures how the reliability of deployed, memory-based AI agents degrades over many sessions across… | Agentic |
| C | AgroBenchAgroBench evaluates vision-language models on seven expert-annotated agricultural question-answering tasks… | Science |
| C | AgroVGAgroVG measures how well vision-language models localize agricultural targets (crops, weeds, fruit, pests,… | Vision |
| C | AI Snitches Get GlitchesMeasures whether LLM agents covertly monitor and report users (agentic surveillance) across over 300… | Agentic |
| C | AI2DAI2D measures diagram understanding through multiple-choice question answering over grade-school science… | Knowledge |
| C | Aider Refactoring BenchmarkMeasures an LLM's ability to refactor 89 large methods from large Python classes, outputting long code chunks… | Coding |
| C | AIM-BenchAIM-Bench measures the inventory decision-making quality and behavioral biases of LLM agents acting as… | Agentic Business |
| C | AIR-BenchAIR-Bench evaluates large audio-language models on understanding speech, natural sounds and music via… | Knowledge |
| C | AlexandriaAlexandria measures how well machine translation systems and LLMs translate between English and 13 regional… | Translation |
| C | Ambig-DSAmbig-DS measures whether data-science agents recognize and handle underspecified task framings (ambiguous… | Agentic |
| C | Android Control High_EMMeasures how accurately agents complete high-level natural-language tasks in Android apps, scored by exact… | Agentic |
| C | Android Control Low_EMAndroid Control Low_EM measures exact-match accuracy of UI control agents executing low-level human… | Agentic |
| C | Animation2CodeAnimation2Code measures whether vision-language models can reconstruct executable HTML/CSS/JavaScript web… | Coding |
| C | ANLIANLI measures natural language understanding via adversarially collected natural language inference… | Language |
| C | AppellateGenAppellateGen measures how well models draft second-instance (appellate) legal judgments by reasoning over… | Legal |
| C | ARC ChallengeARC Challenge measures models' ability to answer difficult grade-school multiple-choice science questions that… | Knowledge |
| C | ARC EasyARC Easy measures a model's ability to answer grade-school multiple-choice science questions, forming the… | Knowledge |
| C | ArchBench Serverless Component GenerationArchBench's serverless component generation task measures how well LLMs and coding agents generate serverless… | Coding |
| C | Are LLM Decisions Faithful to Verbal Confidence?RiskEval evaluates whether LLMs adjust their abstention decisions in line with their verbal confidence under… | Safety |
| C | ArtifactsBenchArtifactsBench measures LLMs' ability to generate visual and interactive code artifacts, scoring rendered… | Coding |
| C | Artificial Analysis Speech-to-TextA leaderboard measuring speech-to-text transcription accuracy (word error rate), speed factor and price across… | Audio |
| C | ASPIASPI measures how much more vulnerable LLM agents are to prompt injection attacks when they enter a… | Safety |
| C | ASR-FairBenchASR-FairBench is a leaderboard that scores automatic speech recognition models on accuracy (WER) and fairness… | Audio |
| C | AssertLLM2AssertLLM2 measures how well LLMs generate formal SystemVerilog Assertions from hardware design… | Coding |
| C | AstroMindAstroMind measures how well LLMs reason about spacecraft behavior, covering intent inference, maneuver… | Aerospace |
| C | AtelierEvalAtelierEval measures the prompting proficiency of humans and multimodal LLMs as text-to-image prompters across… | Multimodal |
| C | ATOM-BenchATOM-Bench evaluates real-world robot manipulation policies on atomic motor and instruction skills and on… | Robotics |
| C | AudioAgentBenchAudioAgentBench measures voice agent models on six multi-turn spoken workflows (appointment, assistant,… | Speech |
| C | AudioBenchAudioBench evaluates audio large language models on speech, sound, music, ASR, audio QA, translation, and… | Knowledge |
| C | AutoEval-VideoAutoEval-Video measures how well large vision-language models answer open-ended questions about videos across… | Multimodal |
| C | AutoMedBenchAutoMedBench measures how well autonomous coding agents run medical AI research pipelines (plan, setup,… | Healthcare |
| C | AVBenchAVBench measures the quality of human-centric audio-video generative models across ten dimensions covering… | Creative Media |
| C | Bayesian Adaptation GymBayesian Adaptation Gym measures how well Bayesian low-rank adaptation methods for multi-modal language models… | Machine Learning |
| C | BCoughBenchBCoughBench evaluates respiratory acoustic foundation models on cough-based disease, demographic and age tasks… | Audio Healthcare |
| C | Benchmarking Autonomous Agents against TemporalA3S-Bench measures how vulnerable LLM-based autonomous agents are to temporal, spatial, and semantic evasion… | Agentic |
| C | Benchmarking Robot Memory Under InterferenceMeasures how well robot memory systems (memory-augmented VLAs) retain relevant information across sessions as… | Robotics |
| C | Beyond ContinuityMeasures whether LLMs detect topic pivots versus refinements in multi-turn dialogue and shortlist relevant… | Reasoning |
| C | Beyond the All-in-One AgentEntCollabBench measures how well role-specialized LLM agents collaborate in a simulated permission-isolated… | Agentic Business |
| C | Beyond Value BenchmarksMeasures how closely LLMs' value structures align with human reference structures by having models and humans… | Alignment |
| C | BFCL v2BFCL v2 measures how accurately large language models call functions (tools), using live enterprise and… | Tool Use |
| C | BIM-EditBIM-Edit measures how well LLMs perform natural-language editing of IFC-format Building Information Models,… | Document Understanding |
| C | BioCoderBioCoder measures how well large language models generate bioinformatics-specific code in Python, Java and… | Biology |
| C | BioMedHopBioMedHop measures LLM biomedical multi-hop reasoning over multi-source evidence (knowledge graphs, documents,… | Biomedicine |
| C | Biopharma BenchBiopharma Bench measures whether AI agents can complete pharmaceutical regulatory and CMC assignments inside… | Biomedicine Agentic+1 |
| C | BioSecBench-FunctionIt measures whether AI agents can analyze real experimental data (biophysical assays, structures, sequences)… | Biology Safety |
| C | BioSecBench-SurveillanceBioSecBench-Surveillance measures whether AI agents can turn raw pathogen sequencing data into correct genomic… | Biology Agentic+1 |
| C | BlueFin: Benchmarking LLM Agents on Financial SpreadsheetsBlueFin measures how well LLM agents perform synthesis, manipulation, and comprehension tasks over spreadsheet… | Finance Business |
| C | Boiling the FrogBoiling the Frog measures whether tool-using AI agents in corporate and office settings can be led into unsafe… | Agentic |
| C | BrowserARTBrowserART measures whether LLM-based browser agents attempt 100 harmful browser-related behaviors on… | Safety |
| C | CadEvalCadEval scores models on CAD and technical code generation tasks. | Coding |
| C | CalBenchCalBench measures how well multiple LLM agents coordinate calendar scheduling with private calendars, scoring… | Agentic |
| C | CaseHOLDCaseHOLD measures legal reasoning by asking models to pick the correct holding of a cited case from five… | Legal |
| C | CaseReportBenchCaseReportBench measures how well LLMs perform dense information extraction from clinical case reports on… | Healthcare |
| C | Cattle TradeCattle Trade measures LLM agents' strategic reasoning in a long multi-agent economic card game combining… | Game |
| C | CCBenchCCBench measures how well AI coding agents implement features in small, private, human-written codebases not… | Coding |
| C | CEO-Bench: Can Agents Play the Long Game?CEO-Bench measures whether language model agents can sustain long-horizon, adaptive decision-making by… | Agentic Business |
| C | ChaosBenchChaosBench evaluates ML weather emulators on subseasonal-to-seasonal climate prediction using reanalysis data… | Climate |
| C | ChartFIChartFI measures how faithfully and insightfully multimodal large language models describe complex charts,… | Document Understanding |
| C | Checkup2ActionCheckup2Action measures how well language models generate safe, prioritised, patient-oriented structured… | Healthcare |
| C | Chess-World-ModelChess-World-Model measures whether models can track state exactly, predicting the board state reached after a… | Game |
| C | CheXperceptCheXpercept measures expert-level lesion perception of vision-language models on chest X-rays across coarse… | Biomedicine Healthcare |
| C | ChildAgentEvalChildAgentEval is a WISC-inspired interactive benchmark that compares the reasoning of multimodal LLM-based… | Agentic |
| C | ChLogicChLogic measures whether language models preserve logical reasoning performance when the same logical… | Reasoning |
| C | ChronoMedKGChronoMedKG is a temporal biomedical knowledge graph released with the ChronoTQA benchmark, which scores LLMs… | Biomedicine Healthcare |
| C | CIPHERCIPHER measures how often LLM-generated Python code contains cryptographic vulnerabilities under insecure,… | Cybersecurity |
| C | Cited but Not VerifiedEvaluates the source attribution of LLM deep research agents by parsing inline citations in generated reports… | Factuality |
| C | ClawForgeClawForge measures how well command-line agents complete executable, multi-step workflows over persistent… | Agentic |
| C | ClinConsensusClinConsensus measures how well LLMs cover physician-authored, case-specific rubric criteria in open-ended… | Healthcare |
| C | Clotho-AQAClotho-AQA measures audio question answering: answering natural language questions about environmental audio… | Knowledge |
| C | CLQTCLQT is a closed-loop, cost-aware trading environment that diagnoses LLM portfolio-management agents'… | Finance |
| C | CoffeeBenchCoffeeBench measures how much net income an LLM agent can generate as a coffee roaster over a 90-day simulated… | Business |
| C | CogScaleCogScale is a benchmark of 14 scalable synthetic tasks that evaluates the cognitive and memory abilities of… | Reasoning |
| C | CogSense-BenchCogSense-Bench is a visual question answering benchmark that scores multimodal LLMs on five cognitive… | Multimodal Understanding |
| C | CommonsenseQACommonsenseQA measures how accurately models answer multiple-choice questions that require commonsense world… | Knowledge |
| C | CommonWhyCommonWhy measures LLMs' entity-based causal commonsense reasoning by having them answer 15,000 why questions… | Reasoning |
| C | Composite-BenchComposite-Bench measures whether models can complete long-horizon enterprise workflows in deterministic… | Agentic Business |
| C | Confirming CorrectEffective tutoring requires distinguishing optimal, valid but suboptimal, and incorrect student solutions, a… | Education |
| C | ConsumerSimBenchConsumerSimBench measures whether LLMs can reconstruct the concrete reaction patterns real consumers express… | Behavioral |
| C | Contextual StereoSetContextual StereoSet measures how robust LLM stereotype-bias evaluations are by holding stereotype content… | Safety |
| C | CooperBenchCooperBench measures how well pairs of coding agents coordinate to implement separate, potentially conflicting… | Agentic |
| C | Counter Turing TestCounter Turing Test (CT2) is a shared-task benchmark that scores systems on detecting AI-generated text… | Factuality |
| C | CRMArenaCRMArena evaluates LLM agents on realistic customer relationship management tasks in a sandboxed Salesforce… | Agentic Business |
| C | CrossCult-KIBenchCrossCult-KIBench measures how well multimodal LLMs can have cultural knowledge inserted for a target culture… | Multimodal Understanding |
| C | CrystalXRD-BenchCrystalXRD-Bench measures whether vision-language models can read powder XRD pattern images and identify the… | Materials |
| C | CUADCUAD measures how well models highlight salient clauses in commercial legal contracts for contract review,… | Legal |
| C | CUDABeaver: Benchmarking LLM-Based Automated CUDA DebuggingCUDABeaver measures whether LLMs can truly repair failing CUDA code from real failing workspaces while… | Coding |
| C | Cultural Fidelity in English-to-Hindi TranslationMeasures whether English-to-Hindi translation systems preserve the recoverability of explicitly encoded gender… | Translation |
| C | CulturaQACulturaQA is an LRM-generated, human-curated Greek question answering dataset used to train and evaluate LLMs… | Question Answering |
| C | CutVerseCutVerse measures how well autonomous GUI agents complete long-horizon media post-production editing tasks… | GUI |
| C | CyberSecEvalCyberSecEval measures LLM cybersecurity risks, including propensity to generate insecure code and compliance… | Generalization |
| C | DBESDBES is a diagnostic benchmark and metric suite that measures expert specialization in Mixture-of-Experts… | Machine Learning |
| C | DEEPSYNTHDEEPSYNTH measures how well LLM agents gather information from multiple web sources, synthesize it, and reason… | Agentic |
| C | DeluluDelulu measures whether code LLMs produce hallucinated completions (invented methods, invalid parameters,… | Coding |
| C | DevBenchDevBench evaluates LLMs across the full software development lifecycle: software design, environment setup,… | Coding |
| C | DexHoldemDexHoldem is a real-world benchmark evaluating dexterous ShadowHand manipulation policies, agentic perception… | Embodied |
| C | DiagFlowBenchDiagFlowBench measures how language models grounded in industrial diagnostic flowcharts recognise and handle… | Instruction Following |
| C | DialDeferDialDefer measures dialogic deference: how LLM judges shift verdicts on identical claims when framed as a… | Behavioral |
| C | DiffSpotDiffSpot measures whether vision-language models can spot and describe fine-grained visual differences between… | Vision |
| C | DiscoverPhysicsDiscoverPhysics measures whether LLM agents can discover the altered laws of motion of simulated N-body worlds… | Reasoning |
| C | Do Coding Agents Understand Least-Privilege Authorization?AuthBench measures whether coding agents can infer least-privilege file-level read/write/execute permission… | Agentic |
| C | DocOSDocOS measures whether GUI agents can autonomously search a web browser for online documentation, understand… | GUI |
| C | DR-ArenaDR-Arena is an automated, dynamic evaluation framework that ranks deep research agents via Elo battles on… | Agentic |
| C | Drift-BenchDrift-Bench measures how LLM agents handle faulty user inputs (implicit intent, missing parameters, false… | Agentic |
| C | Drive-P2DDrive-P2D evaluates vision-language models on autonomous driving through 6,650 questions that progress from… | Embodied |
| C | DriveBenchDriveBench evaluates the reliability and visual grounding of vision-language models on autonomous driving… | Embodied |
| C | DSAEvalDSAEval measures how well LLM-based data science agents solve 641 real-world data science problems over… | Data |
| C | Dynamic-SUPERBDynamic-SUPERB is a leaderboard benchmark that scores speech and audio language models on… | Speech |
| C | EgoPro-BenchEgoPro-Bench evaluates multimodal LLMs on personalized proactive interaction in streaming egocentric videos,… | Video |
| C | EHR-ComplexEHR-Complex measures how well medical agents perform interactive clinical database reasoning over MIMIC-IV… | Biomedicine Healthcare |
| C | EmoSEmoS measures how well multimodal language models recognize fine-grained emotions in bilingual (Chinese and… | Multimodal Understanding |
| C | Enhancing Pathological VLMs with Cross-scale ReasoningScale-VQA measures pathology vision-language models' cross-scale reasoning via 4,685 multiple-choice questions… | Reasoning Healthcare |
| C | EpiBenchEpiBench evaluates LLMs on five epilepsy clinical reasoning tasks: clinical decision QA, EEG report… | Healthcare |
| C | EQUITRIAGEEQUITRIAGE audits gender bias in LLM-based emergency department triage by measuring how often models change… | Healthcare |
| C | ERP-BenchERP-Bench measures how well AI agents complete 300 long-horizon procurement and manufacturing workflow tasks… | Agentic Business |
| C | EUDAIMONIA: Evaluating Undesirable Dynamics in AIEUDAIMONIA measures whether LLMs align with user welfare in social interactions, checking for violations such… | Safety |
| C | EVA-BenchEVA-Bench evaluates voice agents in simulated spoken multi-turn conversations across enterprise domains,… | Agentic Business |
| C | EvoCode-BenchEvoCode-Bench measures whether coding agents can keep their own codebase working across multi-round evolving… | Software Engineering |
| C | Faithfulness Metrics Don't Measure FaithfulnessBonaFide measures how well chain-of-thought faithfulness metrics detect unfaithful reasoning, using 3,066 CoTs… | Alignment |
| C | FBHMFBHM measures vision-language models' ability to detect hateful memes across 25 rhetorical functionalities and… | Safety |
| C | FinBenFinBen is an evaluation benchmark that scores large language models on financial understanding and prediction… | Finance |
| C | FineBenchFineBench measures how well vision-language models understand fine-grained human activity in long-form videos,… | Video |
| C | FORTIS: Benchmarking Over-Privilege in Agent SkillsFORTIS measures over-privilege in LLM agent skills: whether models select the minimally sufficient skill from… | Safety |
| C | FraudSMSWalkerFraudSMSWalker measures whether web agents can accurately and evidence-groundedly judge fraud in bilingual… | Safety |
| C | From Laboratory to Real-World ApplicationsRepoReason measures LLM agents' repository-level code reasoning via white-box abductive assertion verification… | Software Engineering |
| C | From Scenes to ElementsGranuVistaVQA, introduced in the paper From Scenes to Elements, measures how well multimodal RAG systems… | Multimodal Understanding |
| C | FrontierChallengeFrontierChallenge measures whether AI agents can complete end-to-end scientific workflows across six domains… | Agentic Science |
| C | FullHomeFullHome evaluates whether household agents can infer executable task structure from a complete household… | Robotics |
| C | GameCraft-BenchGameCraft-Bench measures whether coding agents can turn natural-language game specifications into complete,… | Game |
| C | GavelGavel evaluates LLMs on long-context multi-document legal case summarization (32K-512K tokens) using a 26-item… | Legal |
| C | GitTaskBenchGitTaskBench measures how well code agents solve 54 real-world tasks by leveraging GitHub code repositories,… | Coding |
| C | Global PIQAGlobal PIQA evaluates culturally specific physical commonsense reasoning of language models across over 100… | Knowledge |
| C | GPTNTGPTNT measures real-time, asymmetric collaboration between multimodal agents who must communicate to defuse… | Agentic |
| C | Hack-Verifiable TextArenaHack-Verifiable TextArena measures whether and how language model agents exploit reward hacking opportunities… | Agentic |
| C | HarmActionsEvalHarmActionsEval measures how often LLM agents avoid executing harmful tool actions under adversarial pressure,… | Safety |
| C | Hex DataBenchHex DataBench measures how well AI agents perform realistic, often vague agentic analytics tasks in a messy… | Agentic Data+1 |
| C | HiMed: Incentivizing Hindi Reasoning in Medical LLMsHiMed is a Hindi medical reasoning benchmark suite evaluating LLMs on Western and Indian systems of medicine… | Healthcare |
| C | HLV-1KHLV-1K measures how well multimodal models understand hour-long videos through time-specific QA and… | Multimodal |
| C | ITSMBenchITSMBench measures whether AI agents can resolve real enterprise IT service management tickets end-to-end… | Agentic |
| C | LexGLUELexGLUE measures model performance on seven English legal language understanding tasks, including court case… | Legal |
| C | LLM-AggreFactLLM-AggreFact measures how well models detect grounded factuality errors (hallucinations) by checking claims… | Safety |
| C | LLMEval-LogicLLMEval-Logic measures LLMs' natural-language logical reasoning on Chinese propositional and first-order logic… | Reasoning |
| C | Look-Ahead-BenchLook-Ahead-Bench measures look-ahead bias in LLMs used in practical financial trading workflows by comparing… | Finance |
| C | MAS-PromptBenchMAS-PromptBench measures how much system-prompt optimization improves multi-agent LLM systems across tasks… | Uncategorized |
| C | MedAgentBenchMedAgentBench measures how well LLM agents complete 300 physician-written clinical tasks by interacting with a… | Healthcare |
| C | MedHELMMedHELM evaluates large language models on 121 clinical tasks across 35 benchmarks spanning healthcare… | Healthcare |
| C | Minerva-EgoMinerva-Ego measures complex multi-step egocentric video reasoning via multiple choice questions over HD-EPIC… | Video |
| C | MiniWoB++MiniWoB++ is a collection of over 100 web interaction environments testing agents' ability to complete simple… | Agentic |
| C | MLAgentBenchMLAgentBench measures how well language agents can autonomously perform machine learning experimentation,… | Agentic |
| C | MMBenchMMBench evaluates multimodal (vision-language) models on image understanding and cross-modal reasoning, with… | Knowledge |
| C | MMErroRMMErroR measures whether vision-language models can detect erroneous reasoning chains and classify the error… | Multimodal Understanding |
| C | MMStarMMStar measures vision-indispensable multimodal capabilities of large vision-language models across 1,500… | Multimodal |
| C | MultiMedQAMultiMedQA measures large language models' medical question answering across professional medical exams,… | Healthcare |
| C | NeedleBenchNeedleBench measures LLM retrieval and reasoning in bilingual long-context tasks with key information embedded… | Generalization |
| C | ODinWODinW measures how well object detection models transfer zero-shot and few-shot to 13 diverse real-world… | Vision |
| C | OdysseyBenchOdysseyBench measures how well LLM agents complete long-horizon office workflows across Word, Excel, PDF,… | Agentic Business |
| C | Omi SOAP Note Safety BenchmarkMeasures how safely models generate clinical SOAP notes from doctor-patient dialogues, scoring unsupported… | Healthcare |
| C | OpenEarthMapOpenEarthMap is a benchmark for global sub-meter level land cover mapping (semantic segmentation of… | Geospatial |
| C | OpenHuEvalOpenHuEval measures how well large language models handle Hungarian language and Hungary-specific knowledge… | Language |
| C | PaintBenchPaintBench measures how precisely multimodal image editing models perform single-answer visual edits across 20… | Multimodal |
| C | PassBenchPassBench measures how well LLMs generate graph compiler passes that speed up long-tail fusible subgraphs,… | Coding |
| C | PeerRankPeerRank is an autonomous LLM evaluation framework where models generate questions, answer them with web… | Meta |
| C | PersonalAlignPersonalAlign (evaluated via its AndroidIntent benchmark) measures how well GUI agents resolve vague mobile… | GUI |
| C | PhyGroundPhyGround measures whether video generation models produce videos that follow 13 physical laws across… | Video |
| C | PhysicianBenchPhysicianBench measures how well LLM agents complete long-horizon physician tasks in a FHIR-based EHR… | Healthcare |
| C | PRISMPRISM is a name shared by two benchmarks: one scores language models on generating spatially correct… | Reasoning |
| C | PRiSM: Benchmarking Phone Realization in Speech ModelsPRiSM benchmarks phone recognition systems and speech models on phonetic perception via intrinsic… | Speech |
| C | PrivacyGLUEPrivacyGLUE measures language models' general language understanding of privacy policies across 7… | Legal |
| C | Progress-BenchProgress-Bench measures whether vision-language models can estimate how far a robot manipulation task has… | Multimodal Understanding |
| C | QASCQASC measures multi-hop question answering by composing facts to answer 8-way multiple-choice grade school… | Knowledge |
| C | QFBenchQFBench measures whether AI agents can write and execute correct quantitative finance code (derivatives… | Finance |
| C | R2MEDR2MED measures how well retrieval systems find medical evidence that requires reasoning, across Q&A reference… | Healthcare |
| C | Recovering Policy-Induced ErrorsGUI-RobustEval measures GUI agents' ability to recover from their own policy-induced errors across 1,216… | GUI |
| C | RedBenchRedBench measures LLM robustness against adversarial red-teaming attack prompts and refusal behavior across 22… | Safety |
| C | ReMMDReMMDBench measures how well models and agents detect multimodal misinformation in real-world multilingual,… | Factuality |
| C | RoboBenchRoboBench evaluates multimodal large language models as embodied brains for robot manipulation across… | Embodied |
| C | RoboWitsRoboWits measures robot policies' cognitive reasoning, creative tool use, and robustness to unexpected… | Robotics |
| C | Rogo Big Finance BenchBig Finance Bench measures how frontier AI agents perform on 928 rubric-graded finance workflow questions such… | Finance |
| C | RTSGameBenchRTSGameBench measures strategic reasoning of vision-language models through gameplay and diagnostic mini-games… | Game |
| C | RXL-RADSetRXL-RADSet measures how validly and accurately language models assign RADS risk categories from 1,600… | Healthcare |
| C | SALT-KGSALT-KG evaluates how well models perform semantics-aware prediction on multi-table enterprise transactional… | Tabular |
| C | SCICONVBENCHSCICONVBENCH measures how well LLMs conduct multi-turn clarification dialogues to formulate well-posed… | Science |
| C | SciFig-BenchSciFig-Bench measures how well systems generate methodology figures for scientific papers from methodology… | Creative Media |
| C | SemEval-2026 Task 7SemEval-2026 Task 7 evaluates how well LLMs and NLP systems answer everyday cultural knowledge questions… | Multilingual |
| C | SIN-BenchSIN-Bench measures whether multimodal LLMs can understand long-form scientific papers by constructing explicit… | Document Understanding |
| C | SparkleSparkle-Bench evaluates instruction-guided video background replacement models on instruction following,… | Video |
| C | SpeechEQSpeechEQ measures the emotional intelligence of speech-language models by testing whether they select socially… | Speech |
| C | SpeedrunBench: SuperTux 1-1SpeedrunBench: SuperTux 1-1 measures how quickly LLM agents can complete the first level of the platformer… | Agentic Multimodal |
| C | StrategyQAStrategyQA measures a model's ability to answer yes/no questions that require inferring implicit multi-step… | Knowledge |
| C | StructEvalStructEval measures LLMs' ability to generate and convert between structured output formats, both… | Structured Output |
| C | SWE-TogetherSWE-Together measures how well coding agents complete repository-level tasks in multi-turn interactive… | Software Engineering |
| C | TAU2-bench RetailTAU2-bench Retail measures how well conversational agents complete multi-turn retail customer-service tasks… | Agentic Business |
| C | TDC BindingDB Patent DTI-DGMeasures how well models predict drug-target binding affinity on BindingDB patent data under domain… | Science |
| C | TDC TCHard TCR-EpitopeMeasures how well models predict TCR-epitope binding interactions, scored by AUPRC on leaderboards under NA… | Science |
| C | TeleCom-BenchTeleCom-Bench measures how well LLMs understand telecom knowledge (3GPP, 5G, proprietary product docs) and… | Domain |
| C | TerminalWorldTerminalWorld measures how well AI agents complete real-world terminal tasks reverse-engineered from… | Agentic |
| C | TinyGrafixBenchTinyGrafixBench measures how precisely image editing models can perform data visualization edits (bar,… | Multimodal |
| C | TriBench-KoTriBench-Ko is a Korean benchmark that evaluates LLM task performance and deployment risks (inaccuracy, bias,… | Legal |
| C | Video-MME v2Video-MME-v2 evaluates multimodal models' video understanding across a tri-level hierarchy of visual… | Multimodal |
| C | VideoDRVideoDR measures multimodal models' ability to answer video-conditioned open-domain questions by extracting… | Multimodal |
| C | VoiceAssistant-EvalVoiceAssistant-Eval measures how well AI voice assistants handle listening, speaking, and viewing tasks across… | Speech |
| C | VoxParadoxVoxParadox measures whether audio LLMs understand paralinguistic speech cues (age, gender, emotion, pitch,… | Audio |
| C | WBenchWBench evaluates interactive video world models over multi-turn cases covering navigation, subject action,… | Video |
| C | WebLINXWebLINX measures how well agents perform conversational web navigation, controlling a browser to follow… | Agentic |
| C | WebWorld-BenchWebWorld-Bench measures how well language world models predict browser state transitions on real-web… | Agentic |
| C | WorkArena-L1WorkArena-L1 measures how well LLM-based web agents complete atomic knowledge-work tasks in the ServiceNow… | Agentic Business |
| C | WorkArena-L2WorkArena-L2 measures web agents' ability to complete compositional, realistic knowledge-work workflows on the… | Agentic Business |
| C | YouCook2YouCook2 scores video-language models on cooking instructional videos via text-to-video retrieval, video… | Knowledge |
| C | AnyGroundBenchAnyGroundBench measures how well vision-language models perform and adapt at spatio-temporal video grounding… | |
| C | EduArtEduArt measures multimodal LLMs' art-historical knowledge and visual reasoning using 871 human-authored… | |
| C | EgoArgusEgoArgus measures whether vision-language models can act as egocentric daily assistants, arbitrating between… | |
| C | S-EMBERS-EMBER measures streaming episodic memory retrieval in long-form egocentric smart-glasses video, requiring… | |
| C | VWE-BENCHVWE-BENCH measures how well multimodal agents construct and refine interactive 3D open worlds from user… | |
| C | 1GC-7RC1GC-7RC measures how well autonomous AI coding agents can design, implement, and train ML models from scratch… | Agentic |
| C | 3DCodeBench3DCodeBench measures how well vision-language model agents translate text and image references into procedural… | Coding |
| C | ABRA: Agent Benchmark for Radiology ApplicationsABRA measures how well (V)LLM agents operate a medical imaging viewer and DICOM server via function-calling… | Biomedicine Healthcare |
| C | Absurd WorldAbsurd World is a benchmarking framework that tests whether LLMs reason logically in altered, absurd versions… | Reasoning |
| C | ACC-OOD benchmarkACC-OOD evaluates frozen noisy-label-learning image classifier checkpoints on both closed-set accuracy and… | Machine Learning |
| C | ActTraitBenchActTraitBench measures personality consistency in LLMs, quantifying the gap between explicit persona… | Behavioral |
| C | ADBenchAD-Bench measures how well LLM agents answer real-world advertising and marketing analytics requests using… | Agentic Business |
| C | AdversaRiskQAAdversaRiskQA measures whether LLMs detect and resist confidently framed misinformation injected into prompts… | Factuality |
| C | AfriVox-v2AfriVox-v2 measures speech recognition accuracy of speech models on in-the-wild African-language audio across… | Speech |
| C | AgentRewardBenchAgentRewardBench measures how well LLM judges and automatic evaluators assess the success, side effects, and… | Agentic |
| C | AgentEHRAgentEHR measures how well LLM agents perform complex clinical decision-making tasks such as diagnosis and… | Healthcare |
| C | AgentEvalBenchAgentEvalBench measures how well AI systems can automatically generate agent evaluations (metrics, executable… | Agentic |
| C | AgentMeterAgentMeter measures how well model-CLI configurations perform on local command-line tasks, using a… | Agentic |
| C | AgentTrapAgentTrap measures whether LLM agents using third-party skills in a sandbox resist malicious runtime behavior… | Agentic |
| C | AGGBenchAGGBench evaluates whether systems can answer entity-level aggregation queries over large unstructured text… | Retrieval |
| C | AGORAAGORA measures how well LLM agents locate sparse evidence across large archives of workplace documents,… | Agentic |
| C | AHa-BenchAHa-Bench measures auditory hallucinations in large audio-language models across semantic, acoustic, and… | Audio |
| C | AIPatient ArenaAIPatient Arena evaluates LLMs in multi-turn, EHR-grounded clinical consultations across eight dimensions of… | Biomedicine Healthcare |
| C | AirNavAirNav measures how well UAV agents follow natural-language instructions to navigate realistic urban aerial… | Embodied |
| C | AIRTBenchAIRTBench measures language models' ability to autonomously discover and exploit AI/ML security… | Cybersecurity |
| C | AITZ_EMAITZ_EM scores GUI agents on Android smartphone tasks by checking whether their predicted screen actions… | Agentic |
| C | ALARBALARB evaluates LLM reasoning in the Arabic legal domain using Saudi commercial court cases, with tasks such… | Legal |
| C | ALFREDALFRED measures how well models map natural language instructions and egocentric vision to action sequences… | Embodied |
| C | ALFWorldALFWorld measures how well agents complete household tasks in interactive text environments aligned with… | Agentic |
| C | ambiguous emotion recognition benchmark for ALMs with TTSA benchmark evaluating how well audio-language models recognize ambiguous emotions in speech under various… | Audio |
| C | AMVICCAMVICC measures visual reasoning failure modes of multimodal LLMs (via VQA) and image generation models (via… | Multimodal Understanding |
| C | AOR-BenchAOR-Bench measures whether large audio language models over-refuse benign pseudo-harmful audio queries, using… | Audio |
| C | API-BankAPI-Bank evaluates tool-augmented LLMs on planning, retrieving, and calling APIs across 314 annotated tool-use… | Tool Use |
| C | ArguAnaArguAna measures how well embedding and retrieval models retrieve the best counterargument for a given… | Retrieval |
| C | Artificial Intelligence Quotient (AIQ) BenchmarkThe AIQ Benchmark measures psychometric cognitive abilities of generative AI models (verbal, quantitative,… | Knowledge |
| C | ASR Code-Switch BenchmarkMeasures commercial automatic speech recognition accuracy (WER and BERTScore) on code-switched speech across… | Speech |
| C | ASTRA-QAASTRA-QA measures how well RAG systems answer abstract questions that require synthesizing information across… | Question Answering |
| C | AstroAlertBenchAstroAlertBench evaluates multimodal LLMs on reviewing real astronomical transient alerts from ZTF through… | Domain |
| C | AstroReason-BenchAstroReason-Bench evaluates LLM agents' planning on physics-constrained space planning problems such as ground… | Aerospace |
| C | ATC Safety-Oriented Language Understanding EvalMeasures how reliably LLMs interpret air traffic control utterances, using consequence-aware scoring that… | Aerospace |
| C | ATRBenchATRBench measures whether long-lived LLM agents proactively ask for reusable user preferences that a later… | Agentic |
| C | AttaQAttaQ measures how vulnerable large language models are to adversarial questions designed to provoke harmful… | Safety |
| C | AttuneBenchAttuneBench measures LLM emotional intelligence in real multi-turn human-model conversations, scoring emotion… | Behavioral |
| C | AutoLogiAutoLogi is a bilingual benchmark of automatically generated open-ended logic puzzles with program-based… | Knowledge |
| C | AutoRO-BenchAutoRO-Bench evaluates how well LLMs reformulate uncertain robust optimization models into tractable… | Math |
| C | AVGen-BenchAVGen-Bench scores text-to-audio-video generation models on uni-modal quality, cross-modal alignment, and… | Video |
| C | AWARE-USAWARE-US measures whether tool-calling conversational agents querying a structured car database can… | Tool Use |
| C | BabelJudgeBabelJudge measures the reliability of LLM-as-a-judge models across languages and agent trajectories, scoring… | Multilingual |
| C | BenGERBenGER evaluates LLM systems on subsumption-based legal reasoning in German law using exam-style free-text… | Legal |
| C | BenHalluEvalBenHalluEval measures how well LLMs detect hallucinations in Bengali across generative QA, Bangla-English… | Factuality |
| C | Beyond Binary SuccessBeyond Binary Success (MetaFine) is a diagnostic meta-evaluation framework that scores vision-language-action… | Embodied |
| C | Beyond Function CallingBeyond Function Calling (ToolBench-X) measures how well tool-using LLM agents complete executable multi-step… | Tool Use |
| C | Beyond Static DialoguesRHELM (Beyond Static Dialogues) measures LLM long-term memory over realistic, evolving dialogues integrated… | Long Context |
| C | BilliardPhys-BenchBilliardPhys-Bench measures multimodal LLMs' intuitive physical reasoning in synthetic billiards scenes,… | Reasoning |
| C | BioSecBench-RefusalBioSecBench-Refusal measures whether AI agents refuse legitimate life-science research tasks and whether they… | Safety |
| C | BIRD-PythonBIRD-Python measures how reliably models translate natural-language data questions into executable… | Coding |
| C | BizBenchBizBench measures LLMs' quantitative reasoning on realistic business and financial problems, including… | Finance |
| C | BLUEX v2BLUEX v2 measures LLM performance in Portuguese on open-ended, free-form questions from second-phase Brazilian… | Multilingual |
| C | BossesCommunities can sustainably manage shared resources (commons) through self-governance and cooperative norms, a… | Agentic |
| C | BUILD-AND-FINDBUILD-AND-FIND evaluates whether downstream agents can recover intended design choices from agent-generated… | Software Engineering |
| C | CADBenchCADBench measures how well models generate editable CAD programs (CadQuery code) from meshes or images,… | Multimodal |
| C | CAM-BenchCAM-Bench measures how well language models and formalization agents prove 1,000 Lean 4 theorems in… | Math |
| C | Can Large Language Models UnderstandCodeMixQA measures how well LLMs understand, reason over, and answer questions in code-switched… | Multilingual |
| C | CanItEditCanItEdit measures how well LLMs perform instructional code editing, updating Python programs given natural… | Coding |
| C | CapRiCorn-1KCapRiCorn-1K measures how accurately and comprehensively models caption long videos (15 s to 10 min) while… | Video |
| C | CAR-benchCAR-bench measures the consistency, uncertainty handling, and limit-awareness of multi-turn, tool-using LLM… | Agentic |
| C | CAREBenchCAREBench evaluates LLMs' emotion understanding through cognitive appraisal reasoning, appraisal ratings, and… | Behavioral |
| C | Causal Sensitivity ScoreCSS measures whether clinical LLMs and agents correctly update oncology tumor-board recommendations when… | Biomedicine Healthcare |
| C | CBT-AudioCBT-Audio measures how well audio language models estimate turn-level patient distress intensity from spoken… | Audio Healthcare |
| C | CFEvalCFEval is a coding benchmark that scores models on competitive programming problems, reported as an Elo-style… | Coding |
| C | ChaosBench-Logic v2ChaosBench-Logic v2 measures LLM logical reasoning about dynamical systems, with 40,886 binary questions over… | Reasoning |
| C | Chart data extraction benchmarkMeasures how accurately multimodal LLMs extract data tables from real-world chart images that lack visible… | Multimodal Understanding |
| C | ChatRAG BenchChatRAG Bench evaluates a model's conversational question answering over documents or retrieved context across… | Retrieval |
| C | ChiEngMixBenchChiEngMixBench evaluates how spontaneously and naturally large language models produce Chinese-English… | Multilingual |
| C | ChronosBenchChronosBench measures how well LLM agents maintain long-term user intents and proactively follow up on… | Agentic |
| C | Citation GroundingCitation Grounding measures how accurately LLMs cite Ukrainian legal provisions, scoring citation precision,… | Legal |
| C | CITYREPCityRep evaluates urban representation learning models across 8 cities and 8 tasks (regression,… | Geospatial |
| C | Claw-AnythingClaw-Anything measures how well LLM agents act as always-on personal assistants over long activity histories,… | Agentic |
| C | Claw-Eval-LiveClaw-Eval-Live measures AI agents on 105 real-world enterprise workflow tasks across 17 families, such as… | Agentic Business |
| C | Clinical Reasoning GraphsClinical Reasoning Graphs evaluates LLM diagnostic reasoning on NEJM clinicopathological cases by extracting… | Healthcare |
| C | ClinicalBenchClinicalBench is a 400-question test over 43 MIMIC-IV patients that measures assertion-aware retrieval and… | Biomedicine Healthcare |
| C | ClinPivotClinPivot measures whether clinical models correctly change treatment decisions when pivoted patient contexts… | Biomedicine Healthcare |
| C | CNMO 2024CNMO 2024 measures models' ability to solve challenging competition mathematics problems from the 2024 Chinese… | Math |
| C | CoCoVideoCoCoVideo measures how well detectors distinguish real videos from high-quality AI-generated videos produced… | Video |
| C | Code-Guided Reasoning for Small Language ModelsCGR is an evaluation protocol measuring when executable Python reasoning scaffolds generated by code improve… | Reasoning |
| C | CodeClinicCodeClinic measures whether LLM agents can synthesize and compose reusable code-based clinical skills over… | Healthcare |
| C | CodeGolf BenchCodeGolf Bench measures LLMs' ability to generate concise, minimal-length code across up to 60 programming… | Coding |
| C | CodeGuard: Improving LLM Guardrails in CS EducationCodeGuard is a benchmark dataset of 8,000 prompts for evaluating how well LLM guardrails classify irrelevant,… | Safety |
| C | CogCAPTCHA30CogCAPTCHA30 is a battery of 30 cognitive tasks that measures whether process-level behavioral features… | Behavioral |
| C | CogRailCogRail measures how well vision-language models perceive railway intrusion risks through spatial position… | Spatial Reasoning |
| C | COLLIECOLLIE measures how well language models generate text satisfying rich, compositional constraints at word,… | Writing |
| C | ColorConceptBenchColorConceptBench measures how well text-to-image models associate colors with implicit concepts such as… | Creative Media |
| C | CombEvalCombEval measures how well large language models solve generated natural-language combinatorial counting… | Reasoning |
| C | CommunityFactCommunityFact measures LLMs' ability to verify claims and detect misinformation in the wild across five… | Factuality |
| C | COMPASSCOMPASS evaluates whether LLMs comply with organization-specific allowlist and denylist policies across eight… | Alignment |
| C | Connecting the DotsRefMem-Bench (Connecting the Dots) measures reflective memory in long-horizon dialogue, requiring models to… | Reasoning |
| C | ConvFinQAConvFinQA measures models' ability to perform chains of numerical reasoning in conversational question… | Finance |
| C | Cookie-BenchCookie-Bench measures LLMs' ability to generate front-end web applications, scored reference-free by an agent… | Coding |
| C | COPACOPA measures open-domain commonsense causal reasoning by asking models to pick which of two alternatives more… | Knowledge |
| C | CorpusQACorpusQA measures how well LLMs and systems reason over entire document corpora up to 10 million tokens,… | Long Context |
| C | Correction Suppression BenchmarkMeasures how often LLMs fail to correct false premises when those claims are embedded in task-oriented… | Factuality |
| C | CounterCountCounterCount measures whether vision-language models count objects from visual evidence rather than prior… | Reasoning |
| C | CPCD-BenchCPCD-Bench evaluates models' long-horizon campus psychological counseling abilities in Chinese across… | Healthcare |
| C | CRAGCRAG measures how accurately LLMs and retrieval-augmented generation systems answer factual questions across… | Search |
| C | CSSBenchCSSBench measures the safety of lightweight LLMs against Chinese-specific adversarial patterns such as… | Safety |
| C | CSVFidelity-BenchCSVFidelity-Bench measures the fidelity of knowledge graphs that LLM pipelines build from statistical CSV… | Data |
| C | CUDAHerculesCUDAHercules measures whether LLMs can write expert-level, hardware-aware optimized CUDA code compared against… | Coding |
| C | CulMindCulMind measures multimodal LLMs' understanding and reasoning on Chinese Cultural Heritage tasks, scoring both… | Reasoning |
| C | CurveBenchCurveBench measures whether vision-language models can recover the exact rooted containment tree of nested,… | Spatial Reasoning |
| C | CUSPCUSP measures whether AI systems can forecast scientific progress across 4,760 events via feasibility… | Science |
| C | CVE-BenchCVE-Bench measures AI agents' ability to autonomously exploit 40 critical real-world web vulnerabilities… | Cybersecurity |
| C | CVerifBenchAn evaluation framework of 500 C program verification tasks built on SV-COMP 2025 that scores models on… | Coding |
| C | CyberChainBenchCyberChainBench measures how well LLM-based agents detect vulnerabilities, generate exploits, and synthesize… | Cybersecurity |
| C | CyBiasBenchCyBiasBench measures attack-selection bias of LLM agents in offensive cybersecurity scenarios across targets,… | Cybersecurity |
| C | DACompDAComp measures how well data agents handle enterprise data engineering (building and evolving multi-stage SQL… | Data |
| C | DataClawBenchDataClawBench measures how well autonomous agents perform exploratory analysis over noisy, real-world,… | Finance Business |
| C | DecisionBenchDecisionBench measures how well AI agents delegate subtasks to peer models in long-horizon agentic workflows,… | Agentic |
| C | DecodingTrust-Agent Platform (DTap)AI agents are increasingly deployed across diverse domains to automate complex workflows through long-horizon… | Safety |
| C | DeepResearch BenchDeepResearch Bench measures how well deep research agents produce citation-rich research reports on 100… | Agentic |
| C | DeepSynth-EvalDeepSynth-Eval measures how well LLMs and agents consolidate information from hundreds of reference papers… | Summarization |
| C | DeepWeb-BenchDeepWeb-Bench measures how well deep research agents search the open web, collect and reconcile cross-source… | Reasoning |
| C | DeFAbDeFAb measures whether foundation models can perform defeasible abduction, constructing hypotheses that… | Reasoning |
| C | DisasterBenchDisasterBench measures whether LLMs can plan executable multi-step disaster-response workflows by composing… | Agentic |
| C | DistractionBenchDistractionBench evaluates whether video large language models can robustly link subjects to events when… | Video |
| C | DMV-BenchDMV-Bench measures whether multimodal agents retain visual memory across chains of autonomous web shopping… | Agentic |
| C | Do Joint Audio-Video Generation Models Understand Physics?AV-Phys Bench measures whether joint audio-video generation models produce outputs consistent with… | Video |
| C | Doc-PPDoc-PP measures whether large vision-language models adhere to user-defined non-disclosure policies when… | Document Understanding |
| C | Don't Judge a Book by its CoverLogiQAte, introduced in the paper Don't Judge a Book by its Cover, measures whether LLMs keep reasoning… | Reasoning |
| C | DPBenchDPBench measures how well multi-agent LLM systems coordinate under simultaneous resource contention, using the… | Agentic |
| C | Dr. DocBenchDr. DocBench measures how well vision-language models and document parsers handle expert-level, difficult… | Document Understanding |
| C | DRBenchDRBench measures dense-scene visual reasoning in vision-language models, with 14,573 questions over 2,943… | Vision |
| C | DRInQDRInQ measures language models' pragmatic reasoning about conversational implicature in questions, varying… | Reasoning |
| C | DrugAuditDrugAudit is a 3,772-item authority-aware benchmark for drug-information question answering that scores source… | Biomedicine Healthcare |
| C | DSGymDSGym evaluates LLM data science agents on executing end-to-end data analyses, bioinformatics tasks and… | Data |
| C | E2V-BenchE2V-Bench measures whether text-to-image models can generate pedagogically meaningful visuals from arithmetic… | Education |
| C | ECLeKTicECLeKTic measures LLMs' cross-lingual knowledge transfer via closed-book fact-seeking questions across 12… | Knowledge |
| C | EComAgentBenchEComAgentBench measures how well LLM shopping agents uncover hidden user intent across query, profile and… | Agentic |
| C | EDIREDIR is a fine-grained composed image retrieval benchmark of 5,000 queries across five categories and fifteen… | Retrieval |
| C | EduBenchEduBench evaluates large language models on 9 student- and teacher-oriented educational scenarios, scoring… | Education |
| C | EduGuardBenchEduGuardBench evaluates LLMs acting as simulated teachers on pedagogical fidelity (teaching harm) and… | Education |
| C | EduResearchBenchEduResearchBench evaluates LLMs on educational academic writing across six research modules and 24… | Education |
| C | Ego2WorldEgo2World measures how well embodied agents plan under partial observation in executable symbolic household… | Embodied |
| C | EgoBenchEgoBench measures how well multimodal video-LLM agents combine egocentric visual perception, tool-augmented… | Agentic |
| C | EmoMMEmoMM measures how multimodal large language models recognize emotions from video, audio and text when… | Multimodal |
| C | EnactToMEnactToM measures functional Theory of Mind in embodied multi-agent agents, scoring whether models can act on… | Reasoning |
| C | EnterpriseClawBenchEnterpriseClawBench measures how well harness-model agent systems complete realistic enterprise workplace… | Agentic Business |
| C | EnvSimBenchEnvSimBench measures how accurately LLMs can simulate environment feedback and state transitions for agent… | Agentic |
| C | EnvTrustBenchEnvTrustBench measures whether LLM agents remain grounded in the true environment state or overtrust stale,… | Agentic |
| C | EO-GymEO-Gym evaluates multimodal, tool-using vision-language agents on interactive Earth Observation analysis tasks… | Geospatial |
| C | ERGeoBenchERGeoBench measures how well multimodal LLMs perform embodied geo-localization from street-view imagery across… | Embodied |
| C | ES-MemEvalES-MemEval measures conversational agents' long-term memory capabilities (information extraction, temporal… | Long Context |
| C | ETHICSETHICS measures language models' knowledge of basic moral concepts across justice, deontology, virtue ethics,… | Generalization |
| C | Every Act Has Its PriceMoral Trolley Arena, introduced in the paper Every Act Has Its Price, is a two-stage blind ELO benchmark… | Safety |
| C | EvoEnvEvoEnv is a dynamic evaluation environment that scores agents in workplace scenarios on context-aware… | Agentic |
| C | EvolMemEvolMem measures multi-session dialogue memory capabilities of LLMs and agent systems across declarative and… | Long Context |
| C | EvoMemBenchEvoMemBench measures how well memory mechanisms in LLM-based agents store, update, and retrieve information… | Agentic |
| C | Executable GamesMeasures interactive multi-turn reasoning of LLMs across 474 executable games where models query a hidden… | Game |
| C | Eye-QEye-Q measures vision-language models' ability to solve cue-implicit visual word puzzles by inferring a target… | Multimodal Understanding |
| C | Factorio Learning EnvironmentFactorio Learning Environment scores LLM agents on building automated factories in the game Factorio, measured… | Game |
| C | FastKernelsFastKernels measures how well LLM-based agents generate optimized GPU kernels that are correct and faster than… | Coding |
| C | FINAL Bench MetacognitiveFINAL Bench measures functional metacognition in language models: whether they recognize uncertainty and… | Knowledge |
| C | FinREDFinRED evaluates the safety of financial LLMs against finance-specific red-teaming prompts covering fraud,… | Finance |
| C | FinRetrievalFinRetrieval measures how accurately AI agents retrieve specific numeric financial values from structured… | Finance |
| C | FinSafetyBenchFinSafetyBench is a bilingual English-Chinese red-teaming benchmark that measures whether LLMs refuse requests… | Finance |
| C | FinSearchCompFinSearchComp measures how well search-enabled LLM agents retrieve time-sensitive, simple historical, and… | Finance |
| C | FlenQAFlenQA measures how increasing input length, via padding of different lengths, types and locations, degrades… | Long Context |
| C | FLEURSFLEURS is a 102-language parallel speech benchmark evaluating few-shot speech tasks such as ASR, speech… | Speech |
| C | FML-BenchFML-Bench measures how well AI research agents iteratively improve baseline codebases on 18 fundamental… | Machine Learning |
| C | FORCEBENCHFORCEBENCH is a contrastive stress test measuring whether citation evaluators and model judges score… | Retrieval |
| C | FormInvFormInv measures whether models give consistent answers to semantically equivalent paraphrases of… | Math |
| C | FounderBenchFounderBench measures whether LLM agents can make repeated, consequential startup-like business decisions… | Agentic |
| C | From 0-Order Selection to 2-Order JudgmentLogiHard measures LLM logical reasoning on multiple-choice exam questions combinatorially transformed from… | Reasoning |
| C | From Hallucination to GroundingCRISP Bench evaluates vision-language models' visual spatial intelligence by checking consistency between… | Spatial Reasoning |
| C | From Knowing to DoingKTD-Fin measures whether LLM trading agents can generate genuine stock-selection alpha in end-to-end… | Finance |
| C | From Prompt Injection to Persistent ControlClawTrojan measures whether LLM agents or defenses in local agentic harnesses can stop multi-step persistent… | Cybersecurity |
| C | From Text to VoiceEvaluates tool-calling ability of omni-modal voice agents on audio-converted versions of text tool-calling… | Speech |
| C | FrontierORFrontierOR measures whether LLMs can design scalable, efficient algorithms for realistic large-scale… | Math |
| C | FRTR-BenchFRTR-Bench measures how well LLMs reason over large multimodal enterprise Excel workbooks with linked sheets… | Spreadsheets |
| C | G-IdiomAlignG-IdiomAlign measures how well LLMs align and translate idioms across languages, using English Wiktionary… | Multilingual |
| C | Gaokao-BenchGAOKAO-Bench evaluates large language models' language understanding and logical reasoning using 2,811… | Knowledge |
| C | GateMemGateMem measures whether memory-augmented LLM agents in multi-principal shared-memory settings stay useful,… | Agentic |
| C | GAVEL: Grounded Caption Error Verification and LocalizationGAVEL measures whether vision-language models can verify image-caption misalignment, explain the discrepancy,… | Vision |
| C | Gaze4HRIGaze4HRI is a large-scale video dataset and benchmark that evaluates zero-shot appearance-based 3D gaze… | Robotics |
| C | Gen-ArenaGen-Arena is a human-annotated benchmark measuring how faithfully text-to-image models realize complex visual… | Vision |
| C | Generalization SpectrumGeneralization Spectrum is an evaluation framework that measures how far learning from individual competitive… | Generalization |
| C | Generalized Zero-Shot Benchmark for Facial Age EstimationMeasures how well facial age estimation models trained only on adult (18-59) face images generalize zero-shot… | Generalization |
| C | GENSTRATGENSTRAT measures LLM strategic reasoning by evaluating models head-to-head on procedurally generated… | Reasoning |
| C | Geo-EvalGeo-Eval is a domain-specific benchmark that scores language models on specialized geological reasoning, such… | Science |
| C | GeoRCGeoRC evaluates vision-language models on geolocation reasoning from images, scoring reasoning chains by… | Geospatial |
| C | German Statutory QA BenchmarkMeasures how well LLMs answer time-sensitive German statutory law questions, testing post-cutoff staleness and… | Legal |
| C | GlobalDentBenchGlobalDentBench measures LLM clinical reasoning and safety in dentistry with 8,978 expert-validated questions… | Biomedicine Healthcare |
| C | Gorilla Benchmark API BenchAPIBench measures how accurately language models generate API calls for HuggingFace, TorchHub, and TensorHub… | Tool Use |
| C | GovReportGovReport evaluates long document summarization of government agency reports with significantly longer… | Summarization |
| C | GRACEGRACE measures step-level faithfulness of chain-of-thought reasoning over input context, with human… | Reasoning |
| C | GraphARCGraphARC measures abstract few-shot reasoning by asking models to infer a graph transformation rule from… | Reasoning |
| C | GraphIP-BenchGraphIP-Bench measures how easily graph neural networks can be stolen through model-extraction attacks and how… | Cybersecurity |
| C | GroundUI-1KGroundUI evaluates GUI grounding: whether models can locate the right on-screen element or coordinates for an… | Multimodal |
| C | GroupMemBenchGroupMemBench measures how well LLM agent memory systems extract, retrieve and apply information in… | Agentic |
| C | GroupTravelBenchGroupTravelBench measures LLM agents on multi-user, multi-turn travel planning, testing preference… | Agentic |
| C | GrowLoopGrowLoop is a self-evolving evaluation system and benchmark that measures human-likeness of large language… | Human-preference |
| C | GRUFF: LLM Pronoun Fidelity, Reasoning, and Biases in GermanGRUFF measures LLMs' pronoun fidelity, referential reasoning, and gender biases in German across gender… | Reasoning |
| C | GSM-SEMGSM-SEM measures the robustness of LLM mathematical reasoning on semantically perturbed variants of… | Reasoning |
| C | GSO-BenchGSO-Bench is a frontier benchmark, described as a general science and observation benchmark, that scores… | Science |
| C | GTEvalGTEval is an evaluation pipeline that measures whether graph-tokenizing LLMs understand graph tokens, using… | Reasoning |
| C | GUI Agents for Continual Game GenerationPlaytestArena, introduced in this paper, measures whether models can generate playable browser-based games: a… | Game |
| C | GUI vs. CLIA matched benchmark of 440 desktop tasks across 18 applications comparing screen-only GUI agents and… | Agentic |
| C | gwBenchmarksgwBenchmarks measures whether LLM coding agents can perform end-to-end, high-precision gravitational wave… | Science |
| C | ScienceAgentBenchScienceAgentBench measures how well language agents can generate self-contained Python programs that complete… | Agentic |
| C | USACOUSACO measures whether language models and agents can solve 307 competitive programming problems from the USA… | Coding |
| C | HalluHard: A Hard Multi-Turn Hallucination BenchmarkHalluHard measures how often language models hallucinate ungrounded factual claims in multi-turn conversations… | Truthfulness |
| C | HallusionBenchHallusionBench measures how large vision-language models handle image-context reasoning, diagnosing language… | Vision |
| C | Halluverse-M3Halluverse-M3 measures how well language models detect fine-grained (entity, relation, sentence-level)… | Safety |
| C | HalluWorldHalluWorld measures language model hallucination against fully specified reference worlds across gridworlds,… | Factuality |
| C | Harness-BenchHarness-Bench measures how agent harness configurations, across multiple model backends, affect completion,… | Agentic |
| C | HarnessAudit-BenchHarnessAudit-Bench measures whether LLM agent harnesses respect user intent, permission boundaries and… | Safety |
| C | HealthAdminBenchHealthAdminBench measures how well AI agents complete administrative healthcare tasks (prior authorization,… | Healthcare |
| C | HearSay Benchmark: Do Audio LLMs Leak What They Hear?HearSay measures whether audio LLMs infer and leak private user attributes (age, gender, health, income, etc.)… | Audio |
| C | HerculeanHerculean measures whether AI agents can carry out end-to-end financial professional workflows (Trading,… | Finance |
| C | HIDBenchHIDBench measures how well large language models perform host-based intrusion detection over large, noisy,… | Cybersecurity |
| C | HOLMES: Evaluating Higher-Order Logical Reasoning in LLMsHOLMES measures LLMs' higher-order symbolic logical reasoning over rules, predicates and constraints using… | Reasoning |
| C | HoloFairHoloFair measures multidimensional demographic bias (gender, age, race) in text-to-image models under neutral… | Safety |
| C | Honeyquest for LLMsHoneyquest for LLMs evaluates how often LLM attackers fall for cyber deception traps across 174 reconnaissance… | Cybersecurity |
| C | HoneyvalHoneyval evaluates LLMs acting as HTTP honeypots against AI hacking agent attackers across 16 backend… | Cybersecurity |
| C | How Well Do Models Follow Their Constitutions?An audit pipeline that measures how often models violate their developers' published behavioral specifications… | Alignment |
| C | HPR-BenchHPR-Bench measures how well LLMs perform holistic persona reasoning from simulated user behavioral histories,… | Behavioral |
| C | HRBenchHRBench is a unified evaluation framework that compares thinking-mode switching strategies in hybrid-reasoning… | Reasoning |
| C | htkgh-polecathtkgh-polecat measures how well LLMs forecast geopolitical events, via relation prediction on a… | Geospatial |
| C | IB-benchIB-bench measures whether large language models can perform investment banking analyst tasks, scored across… | Domain Business |
| C | ICU-BenchICU-Bench measures how well unlearning methods let multimodal large language models continually forget… | Biomedicine Healthcare |
| C | IDEAFixIDEAFix evaluates LLMs' divergent thinking and originality in open-ended idea generation for short design… | Creative |
| C | IFStruct v1.0IFStruct measures whether a model can follow instructions to produce valid JSON or YAML output that matches a… | Instruction Following |
| C | IMAVBIMAVB measures whether omnimodal LLMs detect textual premises that contradict the video or audio of long-form… | Multimodal |
| C | IMCBenchIMCBench measures how well multimodal LLMs handle image-grounded, multi-turn medical conversations, scoring… | Healthcare |
| C | IMLogicIMLogic measures how well LLM memory systems retrieve implicit, logically relevant memories with limited… | Retrieval |
| C | Inductive Generalization for Robotic ManipulationMeasures inductive generalization of robotic manipulation (visuomotor) policies on progressively harder,… | Robotics |
| C | InfiniBenchInfiniBench measures how well large multi-modal models understand very long videos (movies and TV shows)… | Multimodal |
| C | InfiniteBenchInfiniteBench measures language models' ability to process, understand, and reason over contexts longer than… | Generalization |
| C | InfiniteBM Settlers of CatanA head-to-head arena that ranks LLMs by Bradley-Terry Elo ratings from matches of Settlers of Catan involving… | Game |
| C | InfoVQAtestInfographicVQA measures visual question answering over infographic images, requiring joint reasoning over… | Multimodal |
| C | InkBench OCRInkBench OCR ranks OCR models on InkBench document images using pairwise VLM-as-judge comparisons scored with… | Multimodal |
| C | InteractiveBench MathInteractiveBench Math evaluates LLMs' ability to solve HLE math problems through budgeted multi-turn… | Math |
| C | InteractiveBench Texas Hold'emInteractiveBench Texas Hold'em evaluates LLMs' strategic reasoning by having them play thousands of no-limit… | Game |
| C | InteractiveBench Trust GameInteractiveBench Trust Game evaluates LLMs' strategic reasoning in a round-robin Iterated Prisoner's Dilemma… | Game |
| C | InterLV-SearchInterLV-Search measures how well multimodal agents perform interleaved language-vision agentic search, where… | Agentic |
| C | INVESTORBENCHInvestorBench evaluates LLM-based agents on financial decision-making tasks such as trading stocks,… | Finance |
| C | IPO Finance AgentIPO Finance Agent measures how well LLM agents perform IPO due diligence by answering questions over long SEC… | Finance |
| C | IslamicLegalBenchIslamicLegalBench measures LLM knowledge and reasoning of Islamic law across seven schools of jurisprudence,… | Legal |
| C | iWorld-BenchiWorld-Bench evaluates interactive world models on interaction abilities such as visual generation, trajectory… | Video |
| C | JamSet/JamBenchJamBench measures models' ability to generate and complete project-level game code on the Godot engine, scored… | Game |
| C | JEEBenchJEEBench measures LLM problem-solving on 515 challenging mathematics, physics and chemistry problems from the… | Math |
| C | JMMMU-ProJMMMU-Pro measures how well large multimodal models understand Japanese multi-discipline questions presented… | Multimodal |
| C | JourneyBenchJourneyBench measures how well LLM customer-support agents adhere to multi-step business policies across 703… | Agentic Business |
| C | Judge Model for Large-scale Multimodality BenchmarksA multimodal benchmark spanning text, audio, image and video tasks in which a judge model scores MLLM outputs… | Multimodal |
| C | JudgmentBenchJudgmentBench measures how well rubric scoring versus pairwise preference judgments, from expert attorneys and… | Legal |
| C | JuICEJuICE measures how well LLM judges detect span-level cultural and linguistic errors in long-form LLM responses… | Alignment |
| C | K-FinHalluK-FinHallu measures how well LLMs detect hallucinations, including justified abstention, in multi-turn Korean… | Finance |
| C | K12-BenchK12-Bench measures LLMs' curriculum cognition in K-12 education via 23,640 multi-select questions derived from… | Education |
| C | KARMA-MVKARMA-MV measures models' causal audio-visual reasoning about how visual dynamics drive musical structure in… | Video |
| C | KernelBenchXKernelBenchX measures how well LLM-generated Triton GPU kernels compile, match reference outputs numerically,… | Coding |
| C | Kinship Data Benchmark for Multi-hop ReasoningKinshipQA measures LLMs' multi-hop reasoning over kinship relations in generated, culture-specific family… | Reasoning |
| C | Knowledge-Intensive Video GenerationKIVI-Bench measures the factuality and helpfulness of videos that text-to-video models generate from short… | Factuality |
| C | KOCO-BENCHKOCO-BENCH measures how well LLMs and coding agents acquire and apply domain knowledge from curated corpora to… | Coding |
| C | KramaBenchKramaBench measures how well AI systems and agents can design and execute end-to-end data-to-insight pipelines… | Data Agentic+1 |
| C | KSAFE-MMKSAFE-MM measures the safety of multimodal large language models against general and Korean culture-specific… | Safety |
| C | KVoiceBench, KOpenAudioBench, KMMAUKorean speech benchmarks that evaluate speech language models on Korean spoken question answering… | Speech |
| C | LaborBenchLaborBench is a question-and-answer benchmark that evaluates how accurately AI systems, including RAG… | Legal |
| C | LabOSBenchLabOSBench measures how well multimodal GUI agents operate web-based scientific-instrument simulators across… | GUI |
| C | LAMBADALAMBADA measures language modeling and broad context understanding by scoring models on predicting the final… | Language |
| C | LaViSALaViSA measures how well vision-language models resolve structural (syntactic) ambiguity in sentences by using… | Multimodal |
| C | LEAF: A Living Benchmark for Event-Augmented ForecastingLEAF is a living benchmark that evaluates LLMs on event-augmented forecasting tasks, including future event… | Reasoning |
| C | LegalWorldLegalWorld is an interactive environment modeling Chinese civil litigation across five causally connected… | Legal |
| C | LemonadeBenchLemonadeBench measures LLMs' economic intuition, long-term planning and decision-making under uncertainty by… | Business |
| C | LexGeniusLexGenius is an expert-level Chinese legal multiple-choice benchmark evaluating LLMs' legal understanding,… | Legal |
| C | LIBERTyLIBERTy benchmarks the faithfulness of concept-based explanation methods for LLMs using structural… | Machine Learning |
| C | LibEvoBenchLibEvoBench measures whether code generation models correctly use library APIs as they exist in specific… | Software Engineering |
| C | LifeAgentBenchLifeAgentBench measures how well LLM-based personal health assistants answer long-horizon, cross-dimensional,… | Healthcare |
| C | LinAlg-BenchLinAlg-Bench evaluates LLMs on structured linear algebra computation over 3x3, 4x4 and 5x5 matrices across 9… | Math |
| C | LingoQALingoQA measures how truthfully vision-language models answer questions about short autonomous-driving video… | Multimodal |
| C | LithoBenchLithoBench measures how well large multimodal models understand geological semantics in remote-sensing… | Domain |
| C | LITMUSLITMUS measures whether LLM agents operating in real OS environments can be induced via jailbreak, skill… | Safety |
| C | LiveMathematicianBenchLiveMathematicianBench measures LLMs' research-level mathematical reasoning through monthly multiple-choice… | Math |
| C | LiveBrowseCompLiveBrowseComp measures whether deep-search agents can find recently published facts on the web beyond their… | Search |
| C | LiveFMBenchLiveFMBench measures how well LLMs and agents generate correct formal ACSL specifications for C programs that… | Software Engineering |
| C | LiveK12BenchLiveK12Bench measures large multimodal models' reasoning on real, continuously updated high school exam… | Multimodal |
| C | LivePILivePI measures how often AI agents with tool access fall for indirect prompt-injection attacks embedded in… | Safety |
| C | LLM-as-a-ReviewerBenchmarks 12 LLMs as academic paper reviewers on 898 NeurIPS and ICLR papers, measuring rating calibration,… | Safety |
| C | LLM Game BenchmarkLLM Game Benchmark measures LLMs' strategic play and invalid-move behavior by having them compete against each… | Agentic |
| C | LLMEval-MedLLMEval-Med measures how well large language models handle real-world clinical medical questions across five… | Healthcare |
| C | LLMTabBenchLLMTabBench measures how well LLMs perform binary classification on tabular data in zero- and few-shot… | Tabular |
| C | LogDx-CILogDx-CI measures how well log context-reduction tools preserve evidence for LLM root-cause diagnosis of real… | Software Engineering |
| C | Long Code ArenaLong Code Arena is a suite of six benchmarks evaluating code models on tasks requiring project-wide context,… | Coding |
| C | LongActLongAct evaluates planning-level autonomy of embodied agents in long-horizon household tasks specified by… | Embodied |
| C | LongDALongDA measures how well LLM agents answer analytical queries over U.S. national survey data, where they must… | Data |
| C | LongDS-BenchLongDS-Bench measures how well LLM agents maintain, update, restore and compose evolving analytical states… | Data |
| C | LongWebBenchLongWebBench evaluates vision-language models on generating long, multi-screen webpages from visual inputs,… | Coding |
| C | LungCURELungCURE measures multimodal LLMs' clinical reasoning for lung cancer TNM staging, treatment recommendation,… | Healthcare |
| C | M3-BENCHM3-BENCH measures LLM agents' social behaviors in 24 mixed-motive games, evaluating behavioral trajectories,… | Behavioral |
| C | M3KangM3Kang measures vision-language models' multilingual, multimodal mathematical reasoning on 1,747 Kangaroo Math… | Math |
| C | MaCBenchMaCBench measures how well vision-language models handle real-world chemistry and materials science tasks… | Science |
| C | MacroLensMacroLens measures contextual financial reasoning over U.S. small- and micro-cap equities across seven tasks… | Finance |
| C | MalruleLibMalruleLib measures whether language models can infer a student's mathematical misconception from one worked… | Education |
| C | MammoExpertMammoExpert measures how accurately and interpretably models classify breast lesions in mammography images… | Biomedicine |
| C | Managing Procedural Memory in LLM AgentsAFTER measures how well procedural memory skills in LLM agents improve and transfer across 382 realistic… | Agentic Business |
| C | MANTRAMANTRA is a framework and benchmark suite of 285 tasks across 6 domains that measures whether tool-using LLM… | Tool Use |
| C | MASBENCHMASBENCH is a controlled benchmark that characterizes tasks along five axes (Depth, Horizon, Breadth,… | Agentic |
| C | MASLegalBenchMASLegalBench evaluates LLM-based multi-agent systems on deductive legal reasoning tasks grounded in GDPR… | Legal |
| C | Mat-PrefMat-Pref measures compositional chemical reasoning in inorganic materials via 10,837 DFT-grounded… | Materials |
| C | MathAtlasMathAtlas measures how well models autoformalize graduate-level mathematics (theorems, definitions, exercises,… | Math |
| C | MathConstraintMathConstraint measures LLMs' combinatorial reasoning on solver-verified constraint satisfaction problem… | Reasoning |
| C | MathlibPRMathlibPR measures whether LLMs and LLM agents can judge if Lean Mathlib4 pull requests are merge-ready,… | Software Engineering |
| C | MathTutorBenchMathTutorBench measures the open-ended pedagogical capabilities of LLM tutors in math, such as problem… | Education |
| C | MAVEN-BenchMAVEN-Bench is a stress-test benchmark scoring agents on multi-step mathematical and physical reasoning with… | Tool Use |
| C | MCGAMCGA measures how well multimodal LLMs handle Classical Chinese literary audio across six speech tasks: ASR,… | Audio |
| C | MCJudgeBenchMCJudgeBench measures how correctly and consistently LLM judges verify, constraint by constraint, whether… | Instruction Following |
| C | MCR-BenchMCR-Bench measures the reproducibility of LLM outputs across heterogeneous GPUs on mission-critical tasks. | Efficiency |
| C | MDPBenchMDPBench measures how well models parse digital and photographed document images into structured text across… | Multimodal |
| C | Med-HALTMed-HALT measures hallucination in large language models in the medical domain through reasoning-based and… | Healthcare |
| C | MedBench v5MedBench v5 evaluates clinical multimodal models (language, vision-language, agent systems) with dynamic,… | Biomedicine Healthcare |
| C | MedFrameQAMedFrameQA measures how well multimodal LLMs perform multi-image medical visual question answering requiring… | Biomedicine Healthcare |
| C | Medical Chronology LLM BenchmarkMeasures how well LLMs extract structured medical timelines (encounter dates, facilities, key fields) from… | Healthcare |
| C | Medical Information Response AuditMIRA measures whether LLMs provide comparable medical information across user phrasings varying in language,… | Healthcare |
| C | MedMCQAMedMCQA measures how accurately models answer multiple-choice questions from Indian medical entrance exams… | Healthcare |
| C | MedMemoryBenchMedMemoryBench measures how well agent memory systems store, retrieve and reason over long-horizon… | Healthcare |
| C | MedMetaMedMeta measures how well LLMs synthesize conclusions of medical meta-analyses from the abstracts of the cited… | Biomedicine Healthcare |
| C | MedMosaicMedMosaic measures how well audio and multimodal reasoning models answer medical audio questions, covering… | Biomedicine Healthcare |
| C | MedRedFlagMedRedFlag measures whether LLMs detect and redirect false premises embedded in 1100+ real-world patient… | Healthcare |
| C | MedRepBenchMedRepBench measures how well vision-language models and OCR-plus-LLM pipelines produce structured… | Healthcare |
| C | MedSafe-DxMedSafe-Dx measures whether language models safely escalate synthetic patients needing prompt attention from… | Healthcare |
| C | MedStruct-SMedStruct-S measures how well models perform key discovery, key-conditioned QA and end-to-end key-value… | Biomedicine Healthcare |
| C | MEGA MLQAMLQA measures cross-lingual extractive question answering across 7 languages using aligned Wikipedia-based QA… | Question Answering |
| C | MEGA TyDi QAMEGA TyDi QA measures information-seeking question answering across 11 typologically diverse languages. | Question Answering |
| C | MEGA UDPOSMEGA UDPOS measures how well generative language models perform part-of-speech tagging across many… | Language |
| C | MEGA XCOPAXCOPA measures causal commonsense reasoning across 11 typologically diverse languages, asking models to choose… | Knowledge |
| C | Mem-GalleryMem-Gallery measures how well multimodal LLM agents preserve, reason over, and manage long-term memory across… | Multimodal Understanding |
| C | MEME: Multi-entity & Evolving Memory EvaluationMEME evaluates how well LLM agent memory systems store, update, and reason over multi-entity, evolving… | Agentic |
| C | MemEyeMemEye evaluates whether multimodal agent memory systems preserve fine-grained visual evidence and reason… | Agentic |
| C | Memory-WorldMemory-World evaluates mobile GUI agents on long-horizon tasks that require memorizing and later retrieving… | Mobile |
| C | MEMPROBEMEMPROBE measures how well a memory-equipped LLM agent's long-term memory lets a hidden, structured user state… | Agentic |
| C | MemTraceMemTrace measures how LLM agent long-term memory systems handle individual user facts across memory age,… | Agentic |
| C | Metal-SciMetal-Sci measures how well LLMs, driving an evolutionary search loop, can write and optimize correct, fast… | Coding |
| C | MINDGAMESMINDGAMES is a live multi-game arena that rates LLM agents on social and strategic (theory-of-mind) reasoning… | Reasoning |
| C | MIRAGEMIRAGE measures anti-Muslim bias in frontier LLMs across direct completion, chain-of-thought reasoning, and… | Safety |
| C | Misaligned by RewardMeasures whether reward models prefer socially undesirable responses across bias, safety, morality, and… | Alignment |
| C | ML-BenchML-Bench measures how well guardrail models make multilingual safety judgments grounded in region-specific… | Safety |
| C | MLBMLB evaluates large language models on medical knowledge and scenario-based clinical reasoning across five… | Healthcare |
| C | MLReplicateMLReplicate measures how well autonomous research systems reproduce machine learning research from ICML 2025… | Science |
| C | MLX Benchmark V2MLX Benchmark V2 measures how well language models know Apple's MLX machine learning framework, scored by… | Coding |
| C | MM-MT-BenchMM-MT-Bench evaluates vision-language models on multi-turn, open-ended instruction following in practical… | Multimodal |
| C | MM-OptBenchMM-OptBench measures whether multimodal language models can build mathematical optimization formulations and… | Multimodal |
| C | MMCL-BenchMMCL-Bench measures whether multimodal models can learn task-local rules, procedures, and empirical patterns… | Multimodal |
| C | MMDG-BenchMMDG-Bench measures multimodal domain generalization methods across action recognition, mechanical fault… | Multimodal |
| C | MMed-Bench-IRMMed-Bench-IR measures multilingual medical information retrieval across six languages via cross-lingual… | Retrieval Healthcare |
| C | MMEVerseMMEVerse evaluates multimodal large language models on emotion recognition and reasoning from video, audio and… | Multimodal Understanding |
| C | MMVIADMMVIAD evaluates video multimodal LLMs on industrial inspection clips across anomaly detection, defect… | Domain |
| C | MobileGymMobileGym-Bench measures how well mobile GUI agents complete everyday tasks across 28 simulated mobile apps,… | Mobile |
| C | MolViBench: Evaluating LLMs on Molecular Vibe CodingMolViBench measures how well LLMs generate executable, chemically correct code for molecular tasks across 358… | Coding |
| C | MomentoMomento measures whether agents can complete persistent, tool-mediated tasks in multi-session service… | Agentic |
| C | MomentSeekerMomentSeeker measures how accurately MLLMs and video retrievers can locate key moments in long videos from… | Multimodal |
| C | MonitoringBenchMonitoringBench measures how well AI monitors detect sabotage attacks by tool-using coding agents, using 2,644… | Safety |
| C | MOSAIC-BenchMOSAIC-Bench measures whether coding agents can be induced to ship exploitable code when a malicious objective… | Agentic |
| C | MosaicLeaksMosaicLeaks measures whether deep research agents leak private enterprise information through their external… | Safety |
| C | MotionHallucMotionHalluc measures directional, attributional and temporal motion hallucinations of large multimodal models… | Video |
| C | MOV-BenchMOV-Bench measures Omni-LLMs' ability to perform multi-hop reasoning over temporally dispersed audio and… | Reasoning |
| C | MPCI-BenchMPCI-Bench measures whether multimodal language-model agents respect contextual integrity, distinguishing… | Safety |
| C | Mr DreMr Dre evaluates whether deep research agents can reliably revise long-form research reports across multiple… | Agentic |
| C | MRAGMRAG measures how well retrieval-augmented generation systems answer medical and biomedical questions in… | Biomedicine |
| C | MRI-EvalMRI-Eval measures LLM knowledge of MRI physics and GE scanner operations through tiered multiple-choice and… | Biomedicine Healthcare |
| C | MTAVG-Bench 2.0MTAVG-Bench 2.0 measures how well omni large language models diagnose high-level cinematic failure modes… | Creative Media |
| C | MTBBenchMTBBench measures multimodal LLMs on Molecular Tumor Board style oncology decision-making, requiring… | Healthcare |
| C | Multi-Legal-BenchMulti-Legal-Bench evaluates LLMs on identical legal tasks (court-type classification, judgment form, case… | Legal |
| C | MultiBreakMultiBreak measures LLM safety by scoring attack success rates of 10,389 multi-turn jailbreak prompts spanning… | Safety |
| C | MultiEmo-BenchMultiEmo-Bench measures how well multimodal large language models predict the distribution of emotions evoked… | Multimodal |
| C | Multimodal Claim Extraction for Fact-CheckingMeasures how well multimodal LLMs extract fact-checkable claims from social media posts combining text and… | Factuality |
| C | Multimodal UNcommonsenseMUN evaluates vision-language models' commonsense reasoning on visual scenes paired with surprising or… | Multimodal Understanding |
| C | MultiSessionCollabMultiSessionCollab measures how well conversational agents learn user preferences and use them to improve… | Agentic |
| C | MULTITEXTEDITMULTITEXTEDIT measures cross-lingual degradation in text-in-image editing models across 12 languages, 5 visual… | Multilingual |
| C | MuPHIMuPHI evaluates vision-language models on detecting and reasoning about implicit harm that emerges from… | Safety |
| C | MuPPETMuPPET measures whether LLM assistants leak private information inappropriately in multi-party conversations… | Safety |
| C | MusTBENCHMusTBENCH measures how well large audio-language models ground their answers about music in the correct… | Audio |
| C | MVBenchMVBench measures multi-modal large language models' video understanding across diverse temporal tasks. | Knowledge |
| C | MVRBMVRB measures how well multimodal retrieval models perform visualized information retrieval, where texts,… | Multimodal |
| C | MyPCBenchMyPCBench measures how well computer-use agents act as personal assistants on a Linux desktop with 17… | Agentic |
| C | NARRA-Gym for Evaluating Interactive Narrative AgentsNARRA-Gym measures how well LLMs act as interactive narrative agents, sustaining coherent evolving stories,… | Reasoning |
| C | NarrativeTrackNarrativeTrack measures how well multimodal LLMs understand narratives in videos by tracking entities'… | Video |
| C | NarrativeWorldBenchNarrativeWorldBench measures how well language models maintain narrative structure in long-form serialized… | Creative |
| C | Natural Language to MongoshMeasures how well language models translate natural-language questions into correct, executable mongosh… | Coding |
| C | NatureBenchNatureBench measures whether AI coding agents can match or surpass the published state of the art on 90 real… | Science |
| C | needle-1M-benchneedle-1M-bench measures long-context needle retrieval and faithfulness by having models find planted… | Long Context |
| C | "Newspaper Eat" Means "Not Tasty"Measures how well language models detect, classify, and understand coded language in real-world Chinese Google… | Language-understanding |
| C | NICENICE is a 137-item diagnostic benchmark measuring the social intelligence of LLMs across 4 categories and 11… | Behavioral |
| C | NL-PDDL-BenchNL-PDDL-Bench measures how well language models translate natural-language planning task descriptions into… | Reasoning |
| C | NL2ScratchNL2Scratch measures how well language models generate executable Scratch block-based programs from… | Coding |
| C | NLCONLCO measures LLMs' ability to solve natural-language combinatorial optimization problems end-to-end, scored… | Reasoning |
| C | NoisyBenchNoisyBench evaluates model robustness to noisy contextual distractors (random documents, irrelevant chat… | Reasoning |
| C | NoisyCausalNoisyCausal measures how well LLMs perform causal reasoning in natural language scenarios generated from… | Reasoning |
| C | Not All Proofs Are EqualProofRank measures the quality of LLM-generated mathematical proofs beyond correctness, including conciseness,… | Math |
| C | Not Truly MultilingualPuMVR measures whether vision-language models answer identical visual reasoning questions consistently across… | Multilingual |
| C | NPHardEvalNPHardEval measures LLM reasoning ability on 900 algorithmic questions spanning complexity classes up to… | Knowledge |
| C | NRITYAMNRITYAM measures language models' cultural knowledge and comprehension of global dance traditions through… | Domain |
| C | NSMQ RiddlesNSMQ Riddles measures how well LLMs answer multi-clue scientific and mathematical riddles (biology, chemistry,… | Reasoning |
| C | NuscenenuScenes evaluates 3D object detection and tracking for autonomous driving using a full sensor suite of… | Multimodal |
| C | OBLIQ-BenchOBLIQ-Bench measures how well retrieval systems surface documents for oblique queries that seek latent… | Retrieval |
| C | OCR Bench BritannicaRanks OCR models on Encyclopaedia Britannica document images using pairwise VLM-as-judge comparisons scored… | Multimodal |
| C | OctoBenchOctoBench measures how well LLM coding agents follow scaffold-specified, heterogeneous, persistent… | Instruction Following |
| C | OI-BenchOI-Bench measures how susceptible LLMs are to misleading directive options injected into multiple-choice… | Safety |
| C | OLA: Output Language Alignment in Code-Switched LLM InteractionsCode-switching, alternating between languages within a conversation, is natural for multilingual users, yet… | Multilingual |
| C | OlympiadBenchOlympiadBench measures models' olympiad-level reasoning on 8,476 bilingual (English/Chinese), partly… | Math |
| C | Omni-DeepSearchOmni-DeepSearch measures whether omni-modal models can start from audio clips, use text, image and video… | Agentic |
| C | OmniACTOmniACT measures an agent's ability to generate executable scripts that accomplish computer tasks across… | Agentic |
| C | OmniCleanOmniClean is a visually debiased evaluation set of 8,551 queries from nine omni-modal benchmarks that measures… | Multimodal |
| C | OmniCodeOmniCode evaluates LLM-powered software engineering agents on 1794 tasks across Python, Java and C++ covering… | Software Engineering |
| C | OmniEarth-BenchOmniEarth-Bench measures multimodal models on Earth-science tasks across six spheres and cross-sphere… | Geospatial |
| C | OmniMatBenchOmniMatBench measures multimodal language models' reasoning on 3,171 expert-curated QA and calculation… | Materials |
| C | OmniRef-BenchOmniRef-Bench measures how well image generation models handle complex multi-reference image generation with… | Multimodal |
| C | OmniToMOmniToM measures LLM theory of mind by requiring explicit extraction and labeling of actors' belief… | Reasoning |
| C | On Stable Long-Form GenerationVOLTBench measures the length volatility and length accuracy of LLMs' long-form text generation across… | Long Context |
| C | Oncology Report-Derived VQAMeasures zero-shot vision-language model accuracy on multiple-choice VQA questions derived from private… | Biomedicine Healthcare |
| C | OP-BenchOP-Bench measures over-personalization (irrelevance, repetition, sycophancy) in memory-augmented… | Behavioral |
| C | OpenAI-MRCR: 2 needle 128kOpenAI-MRCR 2 needle 128k measures a model's ability to distinguish between and retrieve one of two similar… | Long Context |
| C | OpenBioRQ: Unsolved Biomedical Research Questions for AgentsOpenBioRQ measures whether agentic models faithfully answer or abstain on 12,553 unsolved biomedical research… | Biomedicine |
| C | OpenClaw Arena Model LeaderboardOpenClaw Arena evaluates frontier models as personal AI agents on 100 real-world tasks, measuring task… | Agentic |
| C | OpenComputerOpenComputer measures how well computer-use agents complete verifiable desktop tasks across 33 real… | Agentic |
| C | OpenExemptOpenExempt is a diagnostic benchmark that measures language models' legal reasoning over U.S. Bankruptcy Code… | Legal |
| C | OpenLearnLM BenchmarkOpenLearnLM Benchmark evaluates LLMs as educational tools across Knowledge, Skills, and Attitude dimensions… | Education |
| C | OpenSkillEvalOpenSkillEval evaluates skill-augmented LLM agent systems and open-source skills on dynamically generated… | Agentic |
| C | OpenSTBenchOpenSTBench evaluates speech translation systems (S2TT and S2ST, offline and streaming) across translation… | Speech |
| C | Opus Magnum BenchMeasures how well AI models solve Opus Magnum campaign puzzles via a Python REPL harness, scored on sum scores… | Game |
| C | ORAgentBenchORAgentBench measures whether LLM agents can solve end-to-end operations research tasks by writing and running… | Reasoning |
| C | OrderProbeOrderProbe measures whether LLMs can reconstruct the canonical order of scrambled four-character expressions… | Language-understanding |
| C | OverEagerOverEager measures how often coding agents perform out-of-scope actions (e.g., leaking credentials, deleting… | Safety |
| C | OverEager-Gen / OverEager-BenchOverEager-Bench measures how often autonomous coding agents take out-of-scope (overeager) actions, such as… | Agentic |
| C | Overlooked Safety Vulnerability in LLMsMalOptBench measures how susceptible LLMs are to malicious intelligent optimization algorithm design requests… | Safety |
| C | P2PCLAW Innovative BenchmarkMeasures the quality of scientific research papers written by AI agents, scored by multiple LLM judges across… | Agentic |
| C | PAL-BenchPAL-Bench measures how well AI systems reconstruct owner profiles, social identities and relations, with… | Multimodal |
| C | PaliBenchPaliBench measures how well large language models translate classical Pali Buddhist texts into English, scored… | Translation |
| C | Parallel WebBenchParallel WebBench measures how well long-horizon web agents complete parallel web-exploration tasks, scoring… | Agentic |
| C | ParaPairAudioBenchParaPairAudioBench measures how reliably large audio-language models acting as judges compare speech pairs… | Audio |
| C | Paraphrase-Induced Output-Mode CollapseMeasures whether LLMs preserve the requested output format and answer consistency when prompts are rewritten… | Instruction Following |
| C | Partial Evidence BenchPartial Evidence Bench measures whether agentic systems recognize and report when material evidence lies… | Agentic |
| C | PashtoTTS-BenchPashtoTTS-Bench evaluates Pashto text-to-speech systems with automated screening: synthesis completion, ASR… | Speech |
| C | PBT-BenchPBT-Bench measures whether AI agents can write property-based tests (Hypothesis strategies) that reveal… | Coding |
| C | PCD BenchmarkMeasures how well LLMs detect causal relations in text and extract exact cause-effect phrase pairs across 12… | Reasoning |
| C | PDEAgent-BenchPDEAgent-Bench measures how well LLMs and code agents generate executable, accurate and efficient numerical… | Coding |
| C | pdfQApdfQA measures how well LLM-based QA pipelines answer human-annotated and synthetic questions over real PDF… | Document Understanding |
| C | PDP-BenchPDP-Bench measures how well LLMs predict prosecutorial decisions (prosecution or one of three non-prosecution… | Legal |
| C | PersLitEvalPersLitEval measures LLM knowledge of Persian literature through 4,514 multiple-choice questions across eight… | Multilingual |
| C | PHBenchPHBench measures how well models, including ML models and zero-shot LLMs, predict whether startups launched on… | Finance Business |
| C | PhiBenchPhiBench is an internal Microsoft benchmark that scores language models on diverse skills and reasoning… | Math |
| C | PhoneSafetyPhoneSafety measures whether phone-use agents take the safe action, the unsafe action, or fail to act at 700… | Safety |
| C | Physical AI Bench Conditional GenerationPhysical AI Bench's conditional generation track evaluates how well video generative models produce physically… | Multimodal |
| C | PinTracePinTrace measures the version-level security and compatibility risk of third-party library versions that LLMs… | Cybersecurity |
| C | PInVerifyPInVerify measures how well embodied agents actively choose viewpoints around a candidate object to verify… | Embodied |
| C | PiSARPiSAR scores how well models predict a user's action and rationale on an app screen, given a persona and… | Agentic |
| C | PitchBenchPitchBench measures how reliably audio-language models perceive absolute and relative musical pitch across… | Audio |
| C | PIVOTSPIVOTS evaluates multimodal LLMs' ability to predict bidirectional fine-grained interpersonal relationship… | Reasoning |
| C | PlaceboBenchPlaceboBench measures hallucinations in LLM answers to real pharmaceutical questions in a RAG setting grounded… | Healthcare |
| C | PlanBench-XLPlanBench-XL measures long-horizon planning of LLM tool-use agents that must retrieve and invoke tools across… | Agentic |
| C | PlotQAPlotQA measures visual question answering over scientific plots (bar, line and dot-line charts), including… | Knowledge |
| C | PluriHarmsPluriHarms measures how well AI safety models and alignment methods predict diverse, often disagreeing human… | Safety |
| C | PluRulePluRule measures whether AI models can moderate pluralistic social media communities by identifying which… | Safety |
| C | PMC-VQAPMC-VQA measures how well models answer questions about medical images, with a manually verified test set and… | Healthcare |
| C | Pocket-DentistPocket-Dentist is an efficiency-aware benchmark for dental multimodal question answering that scores… | Biomedicine Healthcare |
| C | Polaris-BenchPolaris-Bench measures whether multimodal LLMs can do visual reasoning robustly by re-formulating 53 visual… | Spatial Reasoning |
| C | POLECAT-FOR-MEPOLECAT-FOR-ME is a Middle East-focused geopolitical event forecasting benchmark where models predict the… | Geospatial |
| C | PolitNuggetsPolitNuggets measures how well agentic LLM systems discover and synthesize long-tail political facts from… | Agentic |
| C | PopQAPopQA is an open-domain QA dataset of 14k entity-centric questions that probes how well language models… | General Knowledge |
| C | PosIRPosIR measures position bias in embedding-based retrieval models, testing how retrieval quality varies with… | Retrieval |
| C | PostEDA-BenchPostEDA-Bench measures how well LLM agents handle post-tool-run EDA tasks: fixing sign-off DRC violations and… | Agentic |
| C | PrecisionMemBenchPrecisionMemBench measures the retrieval precision of LLM memory systems independently of generative models,… | Retrieval |
| C | PrepBenchPrepBench evaluates LLM agents on natural-language-driven data preparation, covering interactive… | Data |
| C | PreScamPreScam measures whether language models can predict how real-world conversational scams progress, via… | Safety |
| C | PrivacyPeekPrivacyPeek measures acquisition-stage privacy leakage of LLM-based tool-using agents, i.e. whether they… | Agentic |
| C | PrivacyQAPrivacyQA measures question answering over mobile app privacy policies, comparing model answers against… | Legal |
| C | PrivacySIMPrivacySIM measures how accurately LLMs conditioned on user personas simulate individual users' privacy… | Behavioral |
| C | ProcCtrlBenchProcCtrlBench evaluates LLM coding agent trajectories for process-level execution defects and control… | Coding |
| C | Proof of TimeProof of Time measures how well models and tool-using agents judge scientific ideas by forecasting post-cutoff… | Science |
| C | ProofNetProofNet measures autoformalization and formal theorem proving of undergraduate-level mathematics in Lean,… | Math |
| C | PseudoBenchPseudoBench measures whether agentic auto-research systems can identify and resist pseudoscientific narratives… | Safety |
| C | PsychEthicsBenchPsychEthicsBench measures LLMs' ethical knowledge and behavioral responses in mental health against Australian… | Healthcare |
| C | QMFOLBenchQMFOLBench measures LLM deductive reasoning on monadic first-order logic tasks with controllable complexity in… | Reasoning |
| C | QMSumQMSum measures query-based summarization of long multi-party meeting transcripts, where models must select and… | Summarization |
| C | QSTRBenchQSTRBench measures how well large language models reason with qualitative spatial and temporal calculi,… | Spatial Reasoning |
| C | QUACKQUACK evaluates multimodal VLM agents in a social deduction game at the level of game outcomes, behavioral… | Game |
| C | QuantEvalQuantEval measures LLM ability in quantitative finance across knowledge QA, quantitative mathematical… | Finance |
| C | QuantSightBenchQuantSightBench measures whether LLMs can produce calibrated 90% prediction intervals for 1,000 real-world… | Finance |
| C | QuestBenchQuestBench measures how well deep research systems answer 256 student-designed expert-level questions across… | Question Answering |
| C | QUIETQUIET measures LLM creative generation capability via multi-blank cascaded story cloze tasks scored… | Creative |
| C | Re$^2$MathRe2Math measures whether models can retrieve and verify applicable theorems from the mathematical literature… | Retrieval |
| C | RealDataAgentBenchRealDataAgentBench scores LLM agents on data-science tasks (EDA, feature engineering, modeling, statistical… | Agentic |
| C | ReasonAudioReasonAudio measures how well text-audio retrieval models handle reasoning-intensive queries involving… | Audio |
| C | RecoAtlasRecoAtlas measures how well LLM shopping recommendation agents produce item sets and justifications, using… | Agentic |
| C | REDACTREDACT measures how well detectors and LLMs identify personally identifiable information across 51 entity… | Data |
| C | RedVoxRedVox measures safety and fairness of speech-capable models on unsafe and stereotypical spoken requests from… | Safety |
| C | Reinforcing Human Behavior Simulation via Verbal FeedbackSOUL measures how well LLMs simulate human-like behavior across 10 tasks spanning theory of mind, role play,… | Behavioral |
| C | ReplaySCMReplaySCM measures whether language models can induce executable Boolean structural causal mechanisms from… | Reasoning |
| C | RepoBenchRepoBench measures repository-level code auto-completion, covering cross-file retrieval, next-line code… | Coding |
| C | RepoMirageRepoMirage measures whether code agents can reason over repository context, using semantics-preserving… | Software Engineering |
| C | Rethinking Continual Anomaly Detection on the EdgeA benchmark evaluating continual anomaly detection methods for industrial inspection under discrete-task and… | Vision |
| C | Rethinking Evaluation for LLM Hallucination DetectionTRIVIA+ is a RAG-based, long-context hallucination detection benchmark with human annotations and noisy label… | Factuality |
| C | Rethinking Reasoning-Intensive RetrievalBRIGHT-Pro, introduced in this paper, measures how well retrievers surface multi-aspect evidence that supports… | Retrieval |
| C | Reward Hacking BenchmarkReward Hacking Benchmark measures how often tool-using LLM agents exploit shortcuts such as skipping… | Safety |
| C | RFC BenchRFC Bench evaluates large language models on detecting financial misinformation in realistic paragraph-level… | Finance |
| C | RGBRGB measures how well large language models perform retrieval-augmented generation across noise robustness,… | Generalization |
| C | RMCBRMCB measures how well confidence estimation methods predict the correctness of long-form reasoning traces… | Reasoning |
| C | RoadmapBenchRoadmapBench measures coding agents' ability to implement long-horizon, multi-target software development… | Software Engineering |
| C | Roboflow Vision Evals - DetectionMeasures how well vision models localize objects by drawing bounding boxes from text queries, scored by… | Vision |
| C | Roboflow Vision Evals - Visual UnderstandingMeasures how well AI vision models answer real-world visual QA tasks such as reading text from photos,… | Vision |
| C | ROK-FORTRESSROK-FORTRESS measures LLM safety on national security and public safety adversarial prompts across… | Safety |
| C | RP-BenchRP-Bench measures how well LLMs perform in roleplay, scoring character consistency, user agency respect,… | Creative |
| C | RTV-BenchRTV-Bench measures multimodal LLMs' continuous perception, understanding and reasoning on real-time video… | Multimodal |
| C | RubricEvalRubricEval measures how accurately LLM judges make rubric-level judgments when evaluating… | Instruction Following |
| C | RubRIXRubRIX is a clinician-validated rubric framework that scores risks (inattention, bias, inaccuracy, uncritical… | Safety |
| C | RuleSafe-VLRuleSafe-VL measures whether vision-language models apply explicit platform moderation policy rules correctly,… | Safety |
| C | SafeClawBenchSafeClawBench measures the security of tool-using LLM agents against 600 adversarial tasks (prompt injection,… | Safety |
| C | SAKESAKE measures LLMs' software architectural knowledge with 2154 expert-curated multiple-choice questions across… | Software Engineering |
| C | Sand-GlassSand-Glass evaluates LLM translation under syllable-level duration constraints, such as for subtitling and… | Translation |
| C | SastBench: A Benchmark for Testing Agentic SAST TriageSastBench measures how well LLM agents triage SAST tool findings, distinguishing real vulnerabilities (CVEs)… | Cybersecurity |
| C | ScaleWoBScaleWoB measures how well GUI agents complete verifiable tasks in synthesized interactive app environments,… | GUI |
| C | scBench-LongscBench-Long measures whether AI agents can recover scientific conclusions from raw or near-raw single-cell… | Biology |
| C | ScholarQuestScholarQuest measures how well LLM-based search agents retrieve sets of relevant academic papers for… | Agentic |
| C | SciIntegrity-BenchSciIntegrity-Bench measures whether AI scientist systems maintain academic integrity, honestly acknowledging… | Safety |
| C | SciPaths: Forecasting Pathways to Scientific DiscoverySciPaths measures whether language models can forecast discovery pathways: identifying the enabling… | Reasoning |
| C | SearchGEOSearchGEO measures how often LLM-based web-search agents are manipulated by attacker-published web content… | Agentic |
| C | SECQUESECQUE measures LLM financial analysis capabilities with 565 expert-written questions on SEC filings covering… | Finance |
| C | Seeing Isn't KnowingSeeing Isn't Knowing (SpatialUncertain) measures whether vision-language models abstain on spatial questions… | Spatial Reasoning |
| C | Seeing vs. BelievingSeeing vs. Believing (CAIT) measures whether multimodal LLMs correctly understand 400 synthetic… | Multimodal Understanding |
| C | Seneca-TRBenchSeneca-TRBench measures LLM proficiency in Turkish using multiple-choice structural-linguistics questions and… | Language |
| C | SEQUORSEQUOR measures how well models follow user constraints throughout long multi-turn conversations, including… | Instruction Following |
| C | SGR-BenchSGR-Bench measures how well search agents find the right specialized data website and configure its… | Agentic |
| C | SkillEvolBenchSkillEvolBench measures whether LLM agents can distill episodic trajectories into reusable procedural skills,… | Agentic |
| C | SkillGenBenchSkillGenBench measures how well LLM agent skill-generation pipelines produce correct, reusable, executable… | Agentic |
| C | SkillSafetyBenchSkillSafetyBench measures whether LLM agents (CLI agents) behave unsafely when adversarial influence is placed… | Safety |
| C | SLDBenchSLDBench measures whether AI agents can discover accurate scaling law formulas from experimental LLM training… | Agentic |
| C | SmartHome-BenchSmartHome-Bench evaluates LLM agents on multi-turn smart home control tasks in a simulated home environment,… | Agentic |
| C | SMDD-BenchSMDD-Bench measures how well LLM agents solve multi-turn, long-horizon small molecule drug design tasks such… | Biomedicine |
| C | SmellBenchSmellBench measures how well LLM agents repair architectural code smells in a Python codebase (scikit-learn),… | Software Engineering |
| C | SNN Fairness BenchmarkA benchmark that evaluates fairness of spiking neural networks under biased training data, spurious features,… | Alignment |
| C | Social CaptionSocial Caption evaluates multimodal LLMs' social understanding of human interactions across social inference,… | Multimodal Understanding |
| C | Social IQaSocial IQa measures commonsense reasoning about social situations through 38,000 multiple-choice questions… | Knowledge |
| C | SocialMemBenchSocialMemBench measures how well AI memory systems recall, attribute and track facts across multi-party social… | Agentic |
| C | SocialPersonaSocialPersona measures whether multimodal LLMs can infer users' revealed preferences from longitudinal… | Agentic |
| C | SoohakSoohak measures LLMs' research-level mathematical problem solving on 439 mathematician-authored problems,… | Math |
| C | SoundnessBenchSoundnessBench measures whether LLMs can judge the methodological soundness of machine-learning research… | Science |
| C | SpatialActSpatialAct measures whether vision-language model agents can perform action-conditioned spatial reasoning in… | Spatial Reasoning |
| C | SpatialBabelSpatialBabel evaluates vision-language models on reconstructing 3D scenes from geometric primitives as code… | Spatial Reasoning |
| C | SpatialBench-LongSpatialBench-Long measures whether AI agents can recover biological claims from raw or near-raw spatial… | Biology |
| C | SpreadsheetBench V1SpreadsheetBench V1 measures LLM agents' ability to manipulate complex real-world spreadsheets using 912… | Spreadsheets |
| C | SpurAudioSpurAudio measures how much few-shot audio classification methods rely on spurious correlations between… | Audio |
| C | SQuIDSQuID measures quantitative spatial reasoning of vision-language models on 2,000 satellite image… | Geospatial |
| C | SREGymSREGym measures how well AI agents diagnose and mitigate injected failures in live cloud-native production… | Agentic |
| C | STAGE-EvalSTAGE-Eval measures how accurately models extract information from long unstructured reports into structured… | Structured Output |
| C | STaRKSTaRK measures how well LLM-driven retrieval systems retrieve correct items from semi-structured knowledge… | Retrieval |
| C | StatABenchStatABench measures LLMs' statistical analysis capabilities through closed-form questions across 18… | Data |
| C | STEBSTEB evaluates Chinese–English speech-to-speech translation systems on translation fidelity, speaker… | Translation |
| C | STELLASTELLA measures how well embedding models retrieve aerospace technical documents (from NASA NTRS), separately… | Aerospace |
| C | StereoSetStereoSet measures stereotypical bias in pretrained language models. | Generalization |
| C | StreamPro-BenchStreamPro-Bench evaluates streaming video models on perception understanding, temporal reasoning, and… | Video |
| C | StreamProfileBenchStreamProfileBench measures how well LLMs infer and continuously update fine-grained user profiles from… | Behavioral |
| C | StressEvalStressEval is a failure-driven dynamic benchmarking framework that synthesizes challenging, controllable test… | Reasoning |
| C | StrongREJECTStrongREJECT measures jailbreak effectiveness by scoring how much useful forbidden information a model gives… | Safety |
| C | StructBreakStructBreak measures how structural cognitive overload attacks trigger safety failures (toxic generation) in… | Safety |
| C | StylisticBiasStylisticBias measures attribute-level social bias in multimodal LLMs by testing how single visual attribute… | Behavioral |
| C | SummScreenFDSummScreenFD measures abstractive summarization of long TV series transcripts into human-written plot recaps. | Summarization |
| C | SVAMPSVAMP is a 1,000-problem challenge set of elementary math word problems that tests whether models solve them… | Math |
| C | SVHallucSVHalluc measures whether audio-visual LLMs hallucinate by failing to ground speech content in visual… | Multimodal |
| C | SWE-MutationSWE-Mutation measures whether LLM-generated test suites for software engineering tasks can detect… | Software Engineering |
| C | SWE-PRBenchSWE-PRBench measures whether LLMs can identify the same issues human reviewers flagged on 350 real merged pull… | Coding |
| C | SynBio-ReasonSynBio-Reason measures how well models generate pysbol3 Python code to repair and design genetic circuits in… | Biology |
| C | TAEFTAEF measures LLMs' semantic understanding and reasoning over large aggregated collections of social media… | Question Answering |
| C | TangramPuzzleTangramPuzzle measures multimodal LLMs' compositional spatial reasoning through tangram-based outline… | Spatial Reasoning |
| C | TeachObsTeachObs measures how well vision-capable LLMs code observable teaching practices in classroom video scenes… | Education |
| C | TeamBenchTeamBench measures how well multi-agent teams coordinate across Planner, Executor, and Verifier roles under… | Agentic |
| C | TECCITECCI measures how well text-guided image editing models handle challenging edits, judged on instruction… | Vision |
| C | TeleResilienceBenchTeleResilienceBench measures LLMs' reasoning resilience in telecommunications: their ability to continue and… | Reasoning |
| C | TempCompassTempCompass measures the temporal perception ability of Video LLMs across temporal aspects such as speed and… | Multimodal |
| C | TempGlitchTempGlitch measures how well vision-language models detect temporal glitches in gameplay videos, using… | Game |
| C | TERMS-BenchTERMS-Bench evaluates LLM agents in simulator-based bilateral price negotiation, scoring surplus extraction,… | Agentic |
| C | Text2Opt-BenchText2Opt-Bench measures how well models translate natural-language problem descriptions into correct,… | Reasoning |
| C | ThaiSafetyBenchThaiSafetyBench measures LLM safety against 1,954 malicious Thai-language prompts, including Thai culturally… | Safety |
| C | The Age of Curiosity Meets the Age of AIKIDBench measures how safely and age-appropriately LLMs respond to child queries (ages 7-11) across ten… | Safety |
| C | The Agent CompanyTheAgentCompany measures how well LLM agents autonomously complete realistic professional tasks in a simulated… | Agentic Business |
| C | The Complexity Ceiling BenchmarkCCB measures how LLM sequential reasoning accuracy decays as the number of required reasoning steps grows… | Reasoning |
| C | The Compliance GapThe Compliance Gap paper releases BS-Bench, which measures whether AI models actually follow process… | Alignment |
| C | The Compliance TrapMeasures whether frontier AI models maintain metacognitive stability (refusing unanswerable questions,… | Safety |
| C | The Harder Text Embedding Benchmark (HTEB)Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static,… | Language |
| C | The Metanym GameThe Metanym Game is a competitive word game benchmark in which LLMs create analogies and rate each other's… | Reasoning |
| C | ThreatCoreThreatCore measures how well models detect fine-grained threats in text, distinguishing explicit threats,… | Safety |
| C | TIDE-BenchTIDE-Bench evaluates tool-integrated reasoning methods in language models across math reasoning,… | Tool Use |
| C | Time PuzzlesTime Puzzles measures LLMs' iterative temporal reasoning with tools such as web search, via algorithmically… | Tool Use |
| C | Time to REFLECTREFLECT is a meta-evaluation benchmark measuring how reliably LLM judges detect fine-grained process- and… | Agentic |
| C | Timely-EvalTimely-Eval measures how well LLM agents adapt their test-time strategies to wall-clock time budgets across… | Agentic |
| C | TimeSage-MTTimeSage-MT measures whether LLM agents can conduct reliable multi-turn time series analysis, from basic… | Agentic |
| C | ToolAlpacaToolAlpaca evaluates language models' generalized tool-use ability, scoring procedure and response accuracy… | Agentic |
| C | ToolPrivacyBenchToolPrivacyBench measures whether tool-using LLM agents complete tasks while routing private information only… | Tool Use |
| C | ToolQAToolQA measures how well large language models answer questions that require using external tools over text,… | Agentic |
| C | TraceEvalTraceEval measures whether LLMs can recover a program's runtime call structure (call graph) from source code… | Reasoning |
| C | Translation en→Set1 COMET22Measures machine translation quality from English into a set of target languages (Set1), scored with the… | Translation |
| C | Translation en→Set1 spBleuMeasures machine translation quality from English into a set of target languages using spBLEU (SentencePiece… | Translation |
| C | Translation Set1→en COMET22Measures how well models translate text from a set of source languages into English, scored with the COMET-22… | Translation |
| C | Translation Set1→en spBleuMeasures machine translation quality into English from a set of source languages using spBLEU… | Translation |
| C | TriggerBenchTriggerBench measures LLMs' prospective memory: spontaneously recalling and acting on latent constraints in… | Reasoning |
| C | TRIP-BenchTRIP-Bench measures how well LLM agents handle long-horizon, multi-turn travel-planning dialogues with many… | Agentic |
| C | ts-benchts-bench measures how well AI coding agent and model combinations solve TypeScript tasks (Exercism exercises… | Coding |
| C | TS-SkillTS-Skill measures how well LLMs and time-series language models answer time-series questions requiring… | Reasoning |
| C | TSCognitionTSCognition measures LLMs' multimodal time series reasoning across five cognitive tasks (Decoding, Grounding,… | Reasoning |
| C | TSM-BenchTSM-Bench evaluates machine-generated text detectors on task-specific LLM-generated text from real-world… | Writing |
| C | TukaBenchTukaBench measures LLM safety against jailbreak prompts in seven African languages, including culturally… | Safety |
| C | TVWorld: Foundations for Remote-Control TV AgentsTVWorld measures how well vision-language agents navigate and ground UI elements on real-world TV interfaces… | Embodied |
| C | TW-LegalBench: Measuring Taiwanese Legal UnderstandingTW-LegalBench measures LLMs' understanding of Taiwanese law through official exam multiple-choice questions,… | Legal |
| C | TxBench-PPTxBench-PP measures whether AI agents can recover accurate small-molecule preclinical pharmacology conclusions… | Biomedicine |
| C | UA-Legal-BenchUA-Legal-Bench measures how well large language models perform Ukrainian legal reasoning tasks, using court… | Legal |
| C | UCSF-PDGM-VQAUCSF-PDGM-VQA measures how well vision-language models answer clinically relevant questions about brain tumor… | Biomedicine Healthcare |
| C | Ukrainian Legal Text BenchmarkIt measures tokenizer fertility and zero-shot performance of foundation models on three tasks over validated… | Legal |
| C | UltraEval-AudioUltraEval-Audio is a unified evaluation framework with public leaderboards that scores audio foundation models… | Audio |
| C | UniREditBenchUniREditBench measures how well image editing models handle reasoning-based edits across real-world and… | Multimodal |
| C | Urban-ImageNetUrban-ImageNet evaluates how well vision and vision-language models perceive urban spaces from social media… | Geospatial |
| C | USAMO25USAMO25 scores models on the six proof-based problems of the 2025 USA Mathematical Olympiad. | Math |
| C | UXBenchUXBench measures how reliable and actionable LLM-generated UX critiques of runnable web interfaces are, scored… | GUI |
| C | V4FinBenchV4FinBench measures how well tabular models, foundation models and LLMs predict corporate bankruptcy across… | Finance |
| C | Vaani-Benchmark-V1.0Vaani-Benchmark-V1.0 measures Hindi automatic speech recognition accuracy by multi-reference word error rate… | Speech |
| C | VBVR-BenchVBVR-Bench evaluates video generation models' reasoning capabilities across abstraction, knowledge,… | Multimodal |
| C | VCG-BenchVCG-Bench measures vision-language models' ability to generate and edit structured diagrams as mxGraph XML… | Structured Output |
| C | VELI5VELI5 measures how well factuality correction methods fix systematically injected factual errors in long-form… | Factuality |
| C | VERA-MHVERA-MH evaluates the safety of chatbots responding to users with suicidal ideation risk in mental health… | Safety Healthcare |
| C | VERHalluVERHallu measures event relation hallucination (causal, temporal, subevent) in video large language models via… | Video |
| C | VeriContestVeriContest measures how well models generate verifiable Rust/Verus code, including formal specifications,… | Software Engineering |
| C | Verified MisguidanceMeasures structural citation failures in search-augmented LLMs, scoring each citation on intent-purpose… | Factuality |
| C | VeriTripVeriTrip measures how well multimodal travel planning agents retrieve and reason over noisy, unstructured… | Agentic |
| C | Verus-SpecBenchVerus-SpecBench measures whether LLM agents can translate informal Codeforces programming problems into… | Software Engineering |
| C | Vibe-EvalVibe-Eval measures multimodal chat models on 269 expert-authored visual understanding prompts, including a… | Multimodal |
| C | VIBenchVIBench measures vertical integration bias, whether LLMs favor their provider's own ecosystem, in direct and… | Coding |
| C | VibeSearchBenchVibeSearchBench measures how well LLM search agents handle long-horizon, multi-turn proactive search with… | Search |
| C | VideoFDBVideoFDB measures how well full-duplex audio-visual conversational agents perceive and generate nonverbal cues… | Multimodal |
| C | VideoNetVideoNet measures vision-language models' ability to recognize domain-specific actions in videos, covering… | Video |
| C | ViDoRe V3 PipelineViDoRe V3 Pipeline evaluates full retrieval-augmented generation pipelines on visually rich multi-domain… | Retrieval |
| C | VietMed-MCQVietMed-MCQ measures LLM knowledge and diagnostic reasoning in Vietnamese Traditional Medicine through 3,190… | Healthcare |
| C | VisAnomBenchVisAnomBench measures how well vision-language models localize anomalies in time-series data and explain them… | Data |
| C | Vision-DeepResearch BenchmarkVDR-Bench measures how well multimodal LLM deep-research systems use visual and textual search to answer 2,000… | Multimodal Understanding |
| C | Vision2CodeVision2Code measures whether vision-language models can generate executable code that reconstructs source… | Coding |
| C | VisIT-Bench Single ImageVisIT-Bench measures how well vision-language models follow real-world instructions about images, scored… | Multimodal |
| C | VisualNeedleVisualNeedle measures whether multimodal LLMs can perform active, fine-grained visual search in… | Vision |
| C | VisualWebArenaVisualWebArena measures how well multimodal web agents complete realistic, visually grounded tasks on websites… | Agentic |
| C | VNU-BenchVNU-Bench measures how well multimodal large language models understand news videos across multiple sources,… | Video |
| C | VoiceAgentBenchVoiceAgentBench evaluates speech language models and ASR-LLM pipelines on spoken agentic tasks: tool… | Speech |
| C | VotIE: Information Extraction from Meeting MinutesVotIE measures how well models extract structured voting events from free-form Portuguese municipal meeting… | Document Understanding |
| C | VPCTVPCT scores models on visual perception and comprehension tasks, ranking multimodal models on a leaderboard. | Multimodal |
| C | V-STaRV-STaR measures how well Video-LLMs perform sequential spatio-temporal reasoning in videos, evaluating what… | Multimodal |
| C | VT-BenchVT-Bench evaluates models on visual-tabular multi-modal discriminative prediction and generative reasoning… | Multimodal |
| C | VULCA-BenchVULCA-Bench evaluates vision-language models on multicultural art critique, scoring cultural understanding… | Multimodal Understanding |
| C | VulnLLM-RVulnLLM-R measures how accurately LLMs detect function-level vulnerabilities in white-box C, Java and Python… | Cybersecurity |
| C | WatchActWatchAct measures whether vision-language planners and robot policies can perform manipulation tasks grounded… | Robotics |
| C | WeatherBenchWeatherBench measures how well data-driven models forecast global medium-range weather, scored by Z500 and… | Weather |
| C | WeatherBench 2WeatherBench 2 evaluates and compares data-driven and traditional numerical global weather forecasting models… | Weather |
| C | WebCoderBenchWebCoderBench measures how well LLMs generate web applications from 1,572 real user requirements, scored with… | Coding |
| C | WebGameBenchWebGameBench measures whether coding agents can turn a structured game specification into a working… | Software Engineering |
| C | WebGymWebGym is a large open-source environment of nearly 300,000 rubric-evaluated tasks on real websites for… | Agentic |
| C | WebIGBenchWebIGBench evaluates multimodal LLMs on generating code for complex interactive webpages from visual designs,… | Coding |
| C | WebMainBenchWebMainBench measures how accurately extractors and model-backed pipelines pull main content from web pages,… | Multimodal |
| C | WebTrap ParkWebTrap Park measures the security of web agents by observing their actions on live web pages across 1,226… | Cybersecurity |
| C | When (and How) to Trust the ExpertA benchmark comparing query-time expert-guided reinforcement learning methods on continuous-control tasks with… | Robotics |
| C | When Cases Get RareOGCaReBench evaluates LLMs on free-form, retrieval-focused answering of rare, off-guideline clinical questions… | Biomedicine Healthcare |
| C | When Context FlipsSafety benchmark scores provide incomplete evidence of deployment readiness: aligned language models often… | Safety |
| C | When LLMs Imagine PeopleAn auditing benchmark (Persona Brainstorm Audit) that scores LLMs for social bias across 16 identity… | Safety |
| C | When Lower Privileges SufficeToolPrivBench, introduced in the paper When Lower Privileges Suffice, measures whether LLM agents select or… | Safety |
| C | When Multiple Scripts MatterMultiClin measures automatic speech recognition accuracy on non-English clinical doctor–patient dialogues,… | Speech Healthcare |
| C | When Reviews DisagreeRevCI, introduced in the When Reviews Disagree paper, measures how well models identify contradiction evidence… | Language-understanding |
| C | When Safe Skills CollideMeasures compositional security risk in LLM agent skill ecosystems: whether individually safe skills combine… | Agentic |
| C | When Search Agents Should AskDiscoBench measures whether LLM search agents can detect ambiguity in vague or incorrect queries, ask… | Agentic |
| C | When the Manual LiesMeasures how vulnerable LLM agents are to Tool Description Poisoning attacks hidden in MCP tool metadata,… | Cybersecurity |
| C | WHOOPS Explanation of ViolationWHOOPS Explanation of Violation measures whether vision-language models can identify and explain why a… | Multimodal |
| C | WideSearchWideSearch measures how reliably LLM-powered search agents collect large-scale atomic information across many… | Search |
| C | Wiki Live ChallengeWiki Live Challenge measures how well deep research agents generate Wikipedia Good Article-quality reports,… | Agentic |
| C | WildAgtEvalWildAGTEval evaluates LLM agents' function-calling capabilities under realistic API complexity, including… | Agentic |
| C | WildGraphBenchWildGraphBench measures how well GraphRAG and RAG systems retrieve and aggregate evidence from long,… | Retrieval |
| C | WMW (World Models in Words)Vision-language models (VLMs) are increasingly used to answer questions about physical scenes, yet most… | Reasoning |
| C | WorldArena Track 2WorldArena evaluates embodied world models on perceptual video quality and functional utility in embodied… | Multimodal |
| C | WorldLinesWorldLines measures how well embodied household-assistant agents use long-term memory of routines, world… | Embodied |
| C | WorldMemArenaWorldMemArena measures how well multimodal agents write, maintain, retrieve and use memory across 400… | Agentic |
| C | WorldScoreWorldScore measures world generation models (3D, 4D scene and video generation) on controllability, quality… | Multimodal |
| C | Wow-wo-valWow-wo-val is an Embodied Turing Test benchmark that evaluates video foundation models as world models for… | Embodied |
| C | XCR-BenchXCR-Bench measures LLMs' cross-cultural reasoning by testing identification, prediction and adaptation of… | Reasoning |
| C | XDomainBenchXDomainBench measures LLMs' interactive interdisciplinary scientific reasoning across 20 domains, diagnosing… | Reasoning |
| C | Xent GamesXent Games are cross-entropy-based single- and multi-player games over LLM probability measures, assembled… | Game |
| C | XL-SafetyBenchXL-SafetyBench measures LLM safety across 10 country-language pairs, scoring jailbreak robustness against… | Safety |
| C | XSTestXSTest measures exaggerated safety behaviour in language models, i.e. whether they wrongly refuse clearly safe… | Safety |
| C | Agentic Commerce World (ACWorld) BenchmarkACWorld Benchmark evaluates Buyer and Merchant agents conducting commerce tasks (product discovery,… | |
| C | BFI-AdaptBFI-Adapt scores whether event-induced Big Five personality shifts in personality-conditioned LLM agents align… | |
| C | Canary ToolsCanary Tools measures how LLM agents reason about tool selection by planting diagnostic decoy tools in an MCP… | |
| C | CircuitReason-1kCircuitReason-1k measures whether multimodal models can solve 1,000 textbook electrical circuit problems from… | |
| C | FormalRx-TestFormalRx-Test is a fine-grained diagnostic benchmark scoring how well models judge, categorize, localize and… | |
| C | IABench-CAIABench-CA measures how deployment rules for allocating consequences causally shape collective safety behavior… | |
| C | LLMForge evaluation benchmarkLLMForge evaluates foundation models on generating parametric CAD designs of mechanical parts from… | |
| C | LongVQUBenchLongVQUBench measures how well vision-language models understand video quality and distortions in… | |
| C | MatPhaseBenchMatPhaseBench evaluates vision-language models on open-ended understanding and description of materials phase… | |
| C | MentalHospitalMentalHospital evaluates LLMs on complete simulated psychiatric clinical encounters (interviewing,… | |
| C | MIOHMIOH measures object hallucination in multimodal LLMs across multi-image scenarios, covering existence,… | |
| C | P-BenchP-Bench measures whether LLM agents can perform statistically valid hypothesis testing — selecting a method,… | |
| C | PolicyShiftBenchPolicyShiftBench measures whether vision-language models and image guardrails correctly judge if an image… | |
| C | RAVEN-EvalRAVEN-Eval evaluates AI video generation models on 150 text-to-video and 100 image-to-video tasks using… | |
| C | SCHEDBenchSCHEDBench measures whether LLMs produce constraint-feasible, optimal schedules for combinatorial scheduling… | |
| C | Sci-VBenchSci-VBench measures how well video generation models produce scientifically accurate, knowledge- and… | |
| C | SciMIFSciMIF measures how well multimodal large language models follow complex, constraint-laden instructions on… | |
| C | Spider 2.0-AIFuncSpider 2.0-AIFunc measures whether language models can generate AI-native SQL queries that use Snowflake… | |
| C | TeXFix-BenchTeXFix-Bench measures how well LLMs repair broken LaTeX, Typst, and Markdown document sources so they compile… | |
| C | VisEditBenchVisEditBench measures how well vision-language models edit visualization code from multimodal feedback,… | |
| C | Bias in the Ear of the ListenerBiasInEar measures speech bias and robustness of multilingual audio language models on 11,200 spoken… | Audio |
| C | Enhancing Agent Safety JudgmentIt measures how well LLMs judge the safety of tool-using agent trajectories rewritten into deceptive,… | Safety |
| C | ChartREG++ChartREG++ measures how well multimodal models ground referring expressions to chart elements across multiple… | Document Understanding |
| C | MedHal-LocMedHal-Loc measures whether medical hallucination detectors faithfully localize the erroneous span in clinical… | Healthcare |
| C | One Instruction Does Not Fit AllMeasures how well multilingual embedding models align personas and instructions across 12 Indian languages via… | Multilingual |
| C | RSPCRSPC measures how well models classify psychiatric disorders, relational stress triggers, and relationship… | Biomedicine Healthcare |
| D | BrowseComp-VLBrowseComp-VL evaluates multimodal agents on deep web-research tasks that combine visual perception with… | Agentic |
| D | AIME 2023AIME 2023 measures models' mathematical problem-solving on 15 competition problems from the AIME 2023 I exam,… | Math |
| D | OSWorld 2.1OSWorld 2.1 measures how well AI agents complete long-horizon computer-use tasks across desktop apps, mocked… | Agentic |
| D | OSWorld 2.1 OfflineMeasures computer-use agents on real desktop OS and app tasks run offline, scored by whether each task ends in… | Agentic |
| D | TAU-benchTAU-bench measures how reliably language agents use domain-specific API tools and follow policy guidelines in… | Agentic |
| D | Tax Agent BenchTax Agent Bench measures how well AI agents answer research-grade US corporate tax questions through… | Agentic |
| D | BrowseComp ExtendedBrowseComp Extended measures agents' ability to persistently navigate the web to locate hard-to-find,… | Agentic |
| D | Terminal-Bench 2.1 ExtendedTerminal-Bench 2.1 Extended measures how well models complete tasks in command-line terminal environments,… | Agentic |
| D | AA LiveCodeBenchAA LiveCodeBench is Artificial Analysis's independently run LiveCodeBench evaluation scoring models' code… | Coding |
| D | HumanEvalHumanEval measures the functional correctness of model-generated Python code by running completions against… | Coding |
| D | LiveCodeBench v6LiveCodeBench v6 scores models on 1055 code generation problems released between May 2023 and April 2025. | Coding |
| D | LiveCodeBench v5LiveCodeBench v5 evaluates LLMs on contest-style competitive programming problems collected August 2024 to… | Coding |
| D | LiveCodeBench ProLiveCodeBench Pro measures LLM performance on continuously updated competitive programming problems from… | Coding |
| D | SWE-Bench Pro V2 FullSWE-Bench Pro V2 Full measures AI agents' resolve rate on long-horizon software engineering tasks drawn from… | Coding |
| D | SWE-Bench Pro V2 HARDSWE-Bench Pro V2 HARD is the hard split of Scale's refreshed SWE-Bench Pro public benchmark, measuring the… | Coding |
| D | SWE-bench Verified ExtendedSWE-bench Verified Extended measures whether models can edit real Python repositories to resolve GitHub… | Coding |
| D | ARC-AGI-3 Provider AdapterARC-AGI-3 Provider Adapter scores models on ARC-AGI-3 interactive reasoning game environments using a harness… | Reasoning |
| D | HLE-DiamondHLE-Diamond measures model accuracy on 1,000 refined Humanity's Last Exam questions testing expert knowledge… | Knowledge |
| D | AA-HLEAA-HLE is Artificial Analysis's run of Humanity's Last Exam, scoring models on 2,158 text-only questions… | Knowledge |
| D | Chinese-SimpleQAChinese SimpleQA measures how factually accurate language models are when answering short Chinese-language… | Knowledge |
| D | HLE w/o toolsHLE w/o tools scores models on Humanity's Last Exam questions answered without tool access, measuring raw… | Knowledge |
| D | SimpleQA VerifiedSimpleQA Verified measures LLM short-form factuality and parametric knowledge on 1,000 human-crafted questions… | Knowledge |
| D | HLE ExtendedHLE Extended tests models on frontier-level, closed-ended academic questions across many disciplines that… | Knowledge |
| D | AA Global-MMLU-LiteAA Global-MMLU-Lite measures models' multilingual MMLU-style knowledge and reasoning across STEM, humanities… | Multilingual |
| D | AA AIME 2025AA AIME 2025 is Artificial Analysis's independent evaluation of models on the 30 integer-answer AIME I and II… | Math |
| D | FrontierMath v2FrontierMath v2 is Epoch AI's corrected benchmark of private, advanced research-level mathematics problems… | Math |
| D | HLE-MultiChoiceHLE-MultiChoice measures model accuracy on 513 text-only multiple-choice questions from Humanity's Last Exam,… | Decision Models |
| D | FHIR-AgentBenchFHIR-AgentBench measures how well LLM agents answer real-world clinical questions by retrieving and reasoning… | Healthcare |
| D | SWE-bench DockerSWE-bench Docker is a Dockerized evaluation tool for SWE-bench that scores whether model-generated patches… | Coding |
| D | SWE-bench JavaScriptSWE-bench JavaScript evaluates software-engineering agents on realistic issue resolution, repository… | Coding |
| B | SWE Atlas: Codebase QnA, Test Writing, Refactoring | Coding Computer Use+2 |
| D | Adversarial Customer ServiceMeasures whether a model acting as a bank support agent can distinguish genuine customers from identity… | Agentic |
| D | ExtractBenchExtractBench measures how accurately models and agents perform schema-guided extraction from enterprise… | Multimodal |
| D | FACTS Benchmark SuiteFACTS Benchmark Suite measures the factual accuracy of LLMs across grounding in documents, image-based… | Knowledge |
| D | Artificial Analysis Decision IndexThe Artificial Analysis Decision Index scores decision models (and LLMs) on structured single-call decisions… | Decision Models |
| D | Beyond AIMEBeyond AIME measures difficult mathematical reasoning requiring deeper reasoning chains and harder problem… | Math |
| C | Tuesday Work Index leaderboard | Enterprise Workflows Long-Horizon |
| D | HANDBOOK.mdHANDBOOK.md measures long-context agentic instruction following, where agents must apply company handbook… | Agentic |
| D | Hermes IndexHermes Index averages model scores across four agentic benchmarks (Hermes Bench, Terminal-Bench 4.0,… | Agentic |
| D | Vals Science IndexVals Science Index averages three agentic science benchmarks (BioMysteryBench, MysteryMechanism,… | Agentic |
| D | FACTS Search V2FACTS Search V2 measures how well models use a shared web search tool (Brave Search API) to retrieve and… | Agentic |
| D | STEMBenchSTEMBench is a private benchmark from Ulam that scores models on STEM tasks such as engineering and astronomy. | Reasoning |
| D | Human Behavior BenchHuman Behavior Bench measures how accurately AI models predict response distributions of specific U.S. adult… | Reasoning |
| D | Game ArenaGame Arena ranks AI models by their play in strategic games such as chess, Go, poker and Werewolf, using a… | Reasoning |
| D | ExtractBench ProExtractBench Pro measures schema-guided document extraction on 100 synthetic long-tail enterprise documents… | Multimodal |
| D | MGSM BengaliMGSM Bengali measures mathematical reasoning on grade-school math word problems presented in Bengali. | Math |
| D | ClaimBenchClaimBench measures whether a model can tell if a 10-K financial document excerpt supports, contradicts, or… | Decision Models |
| D | JudgeBenchJudgeBench measures how well LLM-based judges and reward models identify the objectively correct response in… | Decision Models |
| D | When2CallWhen2Call measures LLM tool-calling decision-making: whether to call a tool, ask a follow-up question, answer… | Decision Models |
| D | WildGuardTestWildGuardTest measures how well LLM safety moderation models detect harmful prompts, harmful responses, and… | Decision Models |
| B | Vals IndexVals Index is a GDP-weighted composite benchmark scoring AI models on real-world finance, coding, legal, and… | Cross-domain |
| C | Andon FMAndon FM tests whether AI agents can autonomously run live radio stations around the clock, covering music… | Computer Use Long-Horizon |
| C | ShelfLife (measured 2026-08-13): | Coding Enterprise Workflows+1 |
| C | ShelfLife E-Sim (measured 2026-08-26): | Coding Enterprise Workflows+1 |
| D | ALUEALUE benchmarks LLMs on aviation and aerospace-specific language tasks such as hazard classification,… | Aerospace |
| D | TEAChTEACh evaluates embodied agents on completing household tasks in simulation using dialogue history, language… | Robotics |
| D | XTREMEXTREME measures cross-lingual generalization of multilingual representations across 40 languages and 9 tasks… | Multilingual |
| D | APPSAPPS measures a model's ability to generate Python code from natural language problem specifications, checked… | Coding |
| D | FLUEFLUE (Financial Language Understanding Evaluation) is a suite of benchmarks scoring language models on five… | Finance |
| D | FormalizeLarge Language Models (LLMs) struggle to solve complex combinatorial problems through direct reasoning, so… | Coding |
| D | GIMGIM measures how well language models integrate multiple cognitive operations (constraint satisfaction, state… | Reasoning |
| B | HARPHARP measures how perturbations or attacks on individual agents in multi-agent LLM systems amplify into… | Safety |
| B | MUSEMUSE measures how well LLMs generate manufacturable, functional, and assemblable multi-part CAD assemblies… | Alignment |
| D | SURESURE is a unified evaluation framework that scores speech understanding systems, from conventional pipelines… | Speech |
| D | TRIAGETRIAGE measures LLMs' prospective metacognitive control: selecting, ordering, and allocating a token budget… | Agentic |
| D | EQ-Bench 4EQ-Bench 4 measures active emotional and social intelligence of models in 16-turn multi-turn roleplay chats… | Agentic |
| D | AA-AnalystAgentAA-AnalystAgent measures how reliably AI agents answer quantitative analyst questions from real-world… | Agentic |
| D | CTI-REALMCTI-REALM measures AI agents' ability to interpret cyber threat intelligence reports and develop working… | Agentic |
| D | AI4AI-BenchAI4AI-Bench measures whether LLM agents can redesign the training algorithm inside frozen AI research… | Agentic |
| D | ApprenticeBenchApprenticeBench measures whether a computer-use agent can learn a real accounts-payable job end to end,… | Agentic |
| D | DiG-benchDiG-bench measures AI agents' ability to discover hidden rules and unknown win conditions through interaction… | Agentic |
| D | Bug Hunt BenchBug Hunt Bench measures how many of 105 planted real bugs in two production codebases frontier coding models… | Coding |
| D | DSBench-HardDSBench-Hard is DeepSeek's internal hard benchmark measuring complex coding-agent capabilities. | Coding |
| D | APEX-SWE (Terminus-2)APEX-SWE (Terminus-2) measures models on software integration and production-failure diagnosis tasks run with… | Coding |
| D | APEX-SWE (Inspect)APEX-SWE (Inspect) measures models on software integration and production-failure diagnosis tasks run with the… | Coding |
| D | CVDPCVDP evaluates LLMs and agents on RTL hardware design, verification, and comprehension problems in Verilog. | Coding |
| D | FACTS Grounding V1FACTS Grounding V1 measures whether model long-form responses are factually grounded in a supplied context… | Reasoning |
| D | Flame-VLM-CodeFlame-VLM-Code scores vision-language models on generating correct code, reported as a percentage, with GLM 5V… | Multimodal |
| D | A-OKVQAA-OKVQA measures visual question answering on about 25K questions that require commonsense and world knowledge… | Multimodal |
| S | AA Openness IndexA composite index scoring how open AI models are, based on weights availability, licensing, and transparency… | Knowledge |
| S | Artificial Analysis Intelligence IndexA composite index from Artificial Analysis that aggregates ten evaluations into a single 0-100 score measuring… | Knowledge |
| D | DrugDiscoveryBenchDrugDiscoveryBench measures AI agents on 82 real drug-discovery and life-sciences research tasks run in a… | Knowledge |
| C | FORTRESSFORTRESS measures the robustness of LLM safeguards against national-security and public-safety risks using 500… | Knowledge |
| D | AdvancedIFAdvancedIF measures LLMs' ability to follow complex, multi-turn, and system-level instructions, graded with… | Instruction Following |
| D | BRUMO 2025BRUMO 2025 measures LLMs' ability to solve 30 olympiad-style competition math problems with verifiable final… | Math |
| D | AIME26AIME26 scores models on the 30 answer-checked problems of the 2026 American Invitational Mathematics… | Math |
| D | APEX-1APEX-1 measures whether frontier models can perform economically valuable single-turn professional text tasks… | Cross-domain |
| D | AA-PersonalityBenchAA-PersonalityBench measures language models' default personality across 30 traits and their communication… | Cross-domain |
| D | LiveClawClaw-Eval-Live (LiveClaw) scores AI agents on completing realistic multi-step enterprise workflows across… | Agentic |
| D | When2Call Multi-TurnWhen2Call Multi-Turn measures whether an assistant selects the correct response policy as a conversation… | Agentic |
| D | RelyToolBenchRelyToolBench measures tool hallucination in LLM agents, scoring pass rate, tool hallucination rate and… | Agentic |
| D | ExplorationBench AlienLogicTests whether a system can discover altered logic rules through exploration and use them to prove or correctly… | Agentic |
| D | MCP-Atlas claim coverageMCP-Atlas claim coverage measures the mean coverage of required claims in model answers when discovering and… | Agentic |
| D | LAB all-passLAB all-pass measures the share of long-horizon legal agent tasks where every expert-curated rubric criterion… | Agentic |
| D | LAB criterion-passLAB criterion-pass measures the mean fraction of expert-written rubric criteria that AI agents pass across… | Agentic |
| D | Toolathlon Verified avg. turnsMeasures the average number of assistant turns models take per trajectory on Toolathlon Verified, a set of 108… | Agentic |
| D | Design Arena Agentic Web DevDesign Arena Agentic Web Dev ranks models by blinded head-to-head human preference on multi-file web… | Agentic |
| D | Harvey LAB-AA v1.1Harvey LAB-AA v1.1 measures whether AI legal agents complete realistic legal tasks satisfying every rubric… | Agentic |
| D | Harvey LAB-AA v1.1 Criterion PassMeasures the share of legal-deliverable rubric criteria a model passes on Harvey LAB-AA v1.1 tasks across 24… | Agentic |
| D | AA Tau3 BankingAA Tau3 Banking measures conversational banking agents that retrieve policy from a large document corpus,… | Agentic |
| D | AA Cyber IndexAA Cyber Index scores AI models on security-relevant tasks such as vulnerability analysis, exploit reasoning… | Agentic |
| D | CWE-Bench-AACWE-Bench-AA is Artificial Analysis' independent run of Collinear AI's CWE-bench, which measures whether AI… | Agentic |
| D | DeepsecBench-AADeepsecBench-AA measures how well AI agents find security vulnerabilities in a given codebase, scored against… | Agentic |
| D | CyberGym-E2E-AACyberGym-E2E-AA measures AI agents' end-to-end cyber defense capabilities on memory-safety tasks, from… | Agentic |
| D | b3 Agent Securityb3 (backbone breaker benchmark) measures how well backbone LLMs inside AI agents resist crowdsourced… | Agentic |
| D | Cyber Misuse RefusalCyber Misuse Refusal measures how often a model refuses malicious cybersecurity-related requests. | Agentic |
| D | Toolathlon-VerifiedToolathlon-Verified measures language agents' general tool use on long-horizon tasks across 600+ tools in… | Agentic |
| D | Harvey Legal AgentHarvey Legal Agent scores AI agents on legal-practice tasks, grading their deliverables against per-task… | Agentic |
| D | Kimi Hallu Bench HardAn internal Moonshot AI stress test of factual accuracy that measures models' claim-level hallucination rate… | Agentic |
| D | τ²-bench resultsτ²-bench measures how well conversational AI agents resolve customer-service tasks in airline, retail and… | Agentic |
| D | τ²-bench Airlineτ²-bench Airline measures whether a model can act as an airline customer-support agent, following a policy… | Agentic |
| B | OpenHands IndexOpenHands Index is a continually updated leaderboard evaluating language models in the OpenHands agent on… | Agentic |
| D | SWE Refactor BenchSWE Refactor Bench measures whether coding agents can complete whole-repository stack migrations, checking… | Agentic |
| D | MM-ClawBenchMM-ClawBench scores multimodal agents on everyday daily tasks. | Agentic |
| D | QwenWebBenchQwenWebBench is an internal Qwen benchmark scoring front-end code generation (web design, apps, games, SVG,… | Agentic |
| D | CoWorkBenchCoWorkBench measures how well models complete long-running office tasks across fields such as computer… | Agentic |
| D | ReviewBenchReviewBench measures how well AI code review agents find true, relevant issues in real GitHub pull requests,… | Agentic |
| D | Ramp Accounting BenchRamp Accounting Bench measures frontier models on 137 long-horizon accounting tasks across simulated… | Agentic |
| D | Visual Tool BenchVisualToolBench (VTB) is a leaderboard benchmark that scores AI models, out of 100, on visual tasks that… | Agentic |
| D | SciUniverse Level 1SciUniverse Level 1 measures whether AI models can carry out real-world scientific research across biology,… | Agentic |
| D | Parallel Search CapabilityA leaderboard by Parallel Web Systems ranking how well models use Parallel's search and extract tools to… | Agentic |
| D | Vals Router IndexVals Router Index compares model routers against single models on long, tool-using agentic tasks in finance,… | Agentic |
| D | Vals Legal IndexVals Legal Index scores models on legal research, legal drafting/review (HLAB-Vals-1.1) and tax agent tasks,… | Agentic |
| D | Vals Inference Provider IndexCompares the same open-weight models served by different inference providers against the creator's own API on… | Agentic |
| D | Vals Finance IndexThe Vals Finance Index scores models on agentic finance work by combining Finance Agent v2.1, EMB v1.1 and Tax… | Agentic |
| D | xdev-bench-hardxdev-bench-hard measures a model agent's ability to discover and exploit real, recently disclosed software… | Agentic |
| D | The Last Ones completionMeasures whether AI models can complete a 32-step cyber range simulation (The Last Ones) end to end, reported… | Agentic |
| D | SEC-Bench ProSEC-bench Pro measures how well frontier models and coding agents find real vulnerabilities by reproducing… | Agentic |
| D | CyScenarioBench solvedCyScenarioBench solved counts the number of cybersecurity scenarios a model completes successfully in at least… | Agentic |
| D | Atomic network attacksAtomic network attacks scores models on simulated network attack tasks as part of a cybersecurity capability… | Agentic |
| D | SCONE simulated revenueMeasures the simulated value (revenue) captured by models exploiting smart-contract vulnerabilities in the… | Agentic |
| D | Anthropic OSS-Fuzz crashAnthropic's internal OSS-Fuzz-derived evaluation measuring the fraction of roughly 830 entry points where the… | Agentic |
| D | Anthropic OSS-Fuzz write primitiveAnthropic's internal OSS-Fuzz-derived evaluation measuring the share of entry points where a model achieves a… | Agentic |
| D | CVE-Bench zero-dayCVE-Bench zero-day mode measures whether AI models can discover and then exploit unknown vulnerabilities in… | Agentic |
| D | RuneScape-BenchRuneScape-Bench measures how well AI coding agents play RuneScape gameplay tasks by writing TypeScript against… | Agentic |
| D | Agent Poker BenchAgent Poker Bench measures how well AI models, acting as agents, play poker against each other, scored by… | Agentic |
| D | Public Benefits Bench v1.1Measures whether general-purpose models can correctly answer SNAP food-assistance benefits questions across… | Agentic |
| D | MysteryMechanismMysteryMechanism measures whether models can rediscover hidden mathematical mechanisms through mathematical… | Agentic |
| D | GDPval ExtendedGDPval Extended evaluates models on producing real-world, economically valuable knowledge-work deliverables… | Agentic |
| B | Harvey LABHarvey's Legal Agent Benchmark (LAB) measures how well AI agents complete long-horizon legal tasks from… | Agentic |
| D | PostTrainBench v1.1PostTrainBench measures how well CLI AI agents can post-train small base language models within ten hours on… | Coding |
| D | VulcanBench v3VulcanBench v3 measures model accuracy, cost, tokens, and time on real software engineering tasks graded by… | Coding |
| D | OpenHarmony BenchOpenHarmony Bench measures whether LLM-based coding agents can implement app-level changes in buildable… | Coding |
| D | VulcanBench CII v1VulcanBench CII v1 evaluates coding agents on post-cutoff open-source software engineering tasks using hidden… | Coding |
| D | FrontierCode 1.1 ExtendedFrontierCode 1.1 Extended measures models on real-world software engineering tasks, grading mergeability and… | Coding |
| D | FrontierCode 2.0FrontierCode 2.0 measures whether coding agents produce mergeable code changes on real open-source repository… | Coding |
| D | ScarfBenchScarfBench measures whether coding agents can perform behavior-preserving migrations of enterprise Java… | Coding |
| D | SWE MultimodalA multimodal variant of SWE-bench that adds visual context such as screenshots and design mockups to software… | Coding |
| D | Kimi Code Bench v2Kimi Code Bench v2 is Moonshot AI's coding-agent benchmark used to score models on coding and agent… | Coding |
| D | Kimi Code Bench 3.0 Long-horizonKimi Code Bench 3.0 Long-horizon is Moonshot AI's internal benchmark measuring coding agents' multi-step… | Coding |
| D | PostTrainBench v1.2PostTrainBench v1.2 measures how well AI agents can post-train small base language models on one H100 GPU in… | Coding |
| D | QwenReactBenchQwenReactBench is Qwen's internal benchmark measuring React application generation, reported as a BT/Elo… | Coding |
| D | DSBench-FullStackDSBench-FullStack is DeepSeek's internal agent benchmark that scores models on complete full-stack software… | Coding |
| D | Next.js EvalsNext.js Evals measures how well AI coding agents build and migrate Next.js apps, scored by withheld assertions… | Coding |
| D | Spider 2.0-LiteSpider 2.0-Lite measures how well language models translate natural-language questions into SQL over… | Coding |
| D | AA Coding IndexThe Artificial Analysis Coding Index scores models on coding as a weighted average of coding benchmarks such… | Coding |
| D | SVG-BenchSVG-Bench (svgbench.ai) is a blind arena benchmark where people vote between pairs of model-generated SVG… | Coding |
| D | BridgeBenchBridgeBench rates frontier AI models on vibe-coding and software-building axes such as reasoning, front end,… | Coding |
| D | IOI v1IOI v1 measures how well AI agents solve International Olympiad in Informatics programming problems in C++,… | Coding |
| D | Vibe Code Bench 1-100Measures whether models can extend a working web application through a sequence of up to ten dependent… | Coding |
| D | SWE Atlas - Codebase QnA ExtendedMeasures coding agents on codebase question answering, test writing, and refactoring in software repositories,… | Coding |
| D | Web Clone BenchWeb Clone Bench measures how well coding agents can build functional, visually faithful replicas of nine… | Coding |
| D | SciCode Main StandardMeasures language models' ability to solve 80 core expert-level scientific research coding problems from… | Coding |
| D | SciCode Main With BackgroundMeasures models' ability to write code solving 80 core scientific research problems when scientist-curated… | Coding |
| D | SciCode Subproblem StandardMeasures models' ability to solve decomposed scientific research coding subproblems curated by scientists,… | Coding |
| D | SciCode Subproblem With BackgroundMeasures models' ability to write code solving 338 decomposed scientific research subproblems when relevant… | Coding |
| D | LongMemEvalLongMemEval measures chat assistants' long-term memory, testing information extraction, multi-session… | Reasoning |
| D | GraphWalks BFS 256K–1MGraphWalks BFS 256K-1M measures whether a model can perform breadth-first search over graphs encoded as edge… | Reasoning |
| D | Conceptual ReasoningThe Conceptual Reasoning Index aggregates three benchmarks (LMCA, ACCoRD, DTBench) measuring how well models… | Reasoning |
| D | Pencil Puzzle BenchPencil Puzzle Bench measures large language models' multi-step reasoning by having them solve pencil puzzles… | Reasoning |
| D | MRCR v2 64K-128KMRCR v2 64K-128K measures a model's ability to retrieve information from long contexts in the 64K-128K token… | Reasoning |
| D | MRCR v2 128K-256KMRCR v2 128K-256K measures long-context retrieval, scoring models on multi-round coreference resolution over… | Reasoning |
| D | MRCR v2 256K-512KMRCR v2 256K-512K measures a model's ability to retrieve multiple relevant items (eight needles) placed within… | Reasoning |
| D | MRCR v2 512K-1MMRCR v2 512K-1M measures a model's ability to retrieve information across very long contexts of 512K to 1M… | Reasoning |
| D | AI-NeedleNEEDLE is a suite of search and retrieval benchmarks from Keenable.ai that evaluates search engines on query… | Reasoning |
| D | MLCR-AAMLCR-AA measures how well models reason over and synthesize long, fragmented medical case files (25k–64k… | Reasoning |
| D | BullshitBench v2BullshitBench v2 measures whether AI models detect and reject nonsensical prompts instead of confidently… | Reasoning |
| D | PropensityBenchPropensityBench measures LLMs' propensity to choose misaligned, high-risk tools under operational pressure… | Reasoning |
| D | NoLiMaNoLiMa measures long-context retrieval in LLMs using needle-in-a-haystack tests where questions and needles… | Reasoning |
| D | NoLiMa-HardNoLiMa-Hard measures long-context retrieval accuracy of (mainly reasoning) models on the 10 most challenging… | Reasoning |
| D | Long Context Reasoning (AA-LCR) ExtendedMeasures models' ability to answer hard questions requiring synthesis and reasoning across multiple long… | Reasoning |
| D | Long Context Reasoning (AA-LCR v1.1)AA-LCR measures models' ability to answer hard questions requiring synthesis and reasoning across multiple… | Reasoning |
| D | FACTS Grounding v2FACTS Grounding v2 measures whether a model's long-form responses to user requests about a supplied context… | Reasoning |
| D | OfficeQA Pro v2OfficeQA Pro v2 measures model and agent performance on grounded, multi-document retrieval and numerical… | Multimodal |
| D | GDP.pdf w/ PythonGDP.pdf w/ Python measures professional document understanding, where the model receives each original PDF in… | Multimodal |
| D | MMMUMMMU measures multimodal models on 11.5K college-level exam questions requiring subject knowledge and… | Multimodal |
| D | olmOCRolmOCR-bench measures how accurately OCR systems convert PDF documents to markdown, preserving text, tables,… | Multimodal |
| D | VoxPopuli WERVoxPopuli WER measures speech-recognition accuracy as word error rate on European Parliament speech clips from… | Multimodal |
| D | Design Arena WebsiteDesign Arena Website is a crowdsourced, vote-based Elo leaderboard ranking AI models on generating websites… | Multimodal |
| D | MathVision w/ PythonMathVision w/ Python measures visual mathematical reasoning where the model may use Python as a tool. | Multimodal |
| D | WorldVQA ForceAnswerWorldVQA ForceAnswer measures a model's accuracy on visual question answering grounded in real-world knowledge… | Multimodal |
| D | We-MathWe-Math measures how large multimodal models solve visual math problems, decomposing composite problems into… | Multimodal |
| D | RefCOCORefCOCO measures referring expression comprehension: whether a model can localize the image object described… | Multimodal |
| D | ImageMiningImageMining evaluates multimodal agents on combining high-density visual understanding with multi-step… | Multimodal |
| D | MMSearchMMSearch evaluates how well large multimodal models perform as multimodal AI search engines, via requery,… | Multimodal |
| D | Facts-VLMFacts-VLM is a public leaderboard benchmark scoring vision-language models on grounded multimodal factuality… | Multimodal |
| D | CharXiv w/o toolsCharXiv w/o tools scores models on visual reasoning over scientific charts without tool use, as tracked on a… | Multimodal |
| D | OmniDocBench 1.6OmniDocBench 1.6 measures how accurately models parse diverse documents (text, formulas, tables, reading… | Multimodal |
| B | AudioMCAudioMultiChallenge measures whether spoken dialogue systems and audio-language models can follow… | Multimodal |
| D | Humanity's Sixth SenseHSS measures intuitive visual reasoning in multimodal models: inferring implicit temporal, spatial, social,… | Multimodal |
| D | ViDoRe V3 ImageViDoRe V3 Image measures retrieval quality (nDCG@10) of embedding models over visual document images given… | Multimodal |
| D | ViDoRe V3 MarkdownViDoRe V3 Markdown measures retrieval quality (nDCG@10) over specialized industry document corpora in English… | Multimodal |
| B | Vals Multimodal IndexA GDP-weighted composite index scoring models on image-inclusive economic tasks across finance, coding, and… | Multimodal |
| D | VoxelBench TextA human-preference leaderboard where language models build voxel structures from text prompts and voters… | Multimodal |
| D | VoxelBench ImageVoxelBench is a human-preference Elo leaderboard that rates how well models build 3D Minecraft-like voxel… | Multimodal |
| D | CharXiv ExtendedMeasures multimodal models' understanding of real arXiv research-paper charts through descriptive and… | Multimodal |
| D | MedXpertQA MM ExtendedMeasures advanced medical reasoning via multimodal multiple-choice questions over clinical images and patient… | Multimodal |
| D | MMMU-Pro ExtendedMMMU-Pro Extended measures joint visual and textual reasoning on multimodal questions, with text-only-solvable… | Multimodal |
| D | GaRAGeGaRAGe measures whether LLMs ground long-form RAG answers in the relevant supplied passages, attribute sources… | Knowledge |
| D | Protein DesignEvaluates protein foundation models on protein design tasks such as generating sequences and structures,… | Knowledge |
| D | Morphology-to-molecule matchingMeasures how well models match multi-channel cell imaging data from a human liver in-vitro assay to the… | Knowledge |
| D | Protein Design library rankingMeasures how well models rank a library of protein designs to prioritize them for experimental (wet-lab)… | Knowledge |
| D | FrontierScienceFrontierScience measures expert-level scientific reasoning of frontier language models on olympiad-level and… | Knowledge |
| D | AA-Omniscience AccuracyAA-Omniscience Accuracy measures the percentage of knowledge questions, spanning science, software… | Knowledge |
| D | Kimi Hallu BenchKimi Hallu Bench is Moonshot AI's internal benchmark that stress-tests factual accuracy, scoring models by… | Knowledge |
| D | FrontierScience ResearchFrontierScience Research evaluates models on open-ended PhD-level scientific research subtasks in physics,… | Knowledge |
| D | MultiLoKoMultiLoKo measures LLMs' multilingual local knowledge across 31 languages, with locally sourced questions plus… | Knowledge |
| D | FACTS ParametricFACTS Parametric measures language models' world knowledge and factuality by having them answer closed-book… | Knowledge |
| D | Immigration BenchImmigration Bench measures how well AI models answer realistic, attorney-written U.S. immigration law… | Knowledge |
| D | SUPERChemSUPERChem measures expert-level, multi-step multimodal chemical reasoning of LLMs on 500 expert-curated… | Knowledge |
| D | VWT2k-liteVWT2k-lite, now cataloged as a Qwen provider comparison on WMT24++, measures machine translation quality… | Multilingual |
| D | NanoBEIR MultilingualNanoBEIR Multilingual measures text retrieval quality of embedding/retriever models across multiple languages… | Multilingual |
| D | MKQA-11MKQA-11 measures cross-lingual open-domain QA retrieval quality (Recall@20) of retrieval models across 11… | Multilingual |
| D | WMT26WMT26 General MT task evaluates machine translation systems' translation quality across many language pairs,… | Multilingual |
| D | Brevity BenchmarkBrevity Benchmark scores conversational responses on both answer quality and conciseness relative to… | Instruction Following |
| D | RoleBenchRoleBench measures large language models' character-level role-playing abilities, such as imitating a given… | Instruction Following |
| D | RoleMRCRoleMRC measures how well LLMs follow diverse, nested and prioritized instructions while maintaining a defined… | Instruction Following |
| D | IHEvalIHEval measures whether language models follow the instruction hierarchy (system messages, user messages,… | Instruction Following |
| D | InverseIFEvalInverseIFEval measures whether language models can follow counter-intuitive instructions that conflict with… | Instruction Following |
| D | SOB Value AccSOB Value Acc measures structured-output value accuracy: whether a model's extracted JSON values exactly match… | Instruction Following |
| D | MASKMASK measures whether large language models lie under pressure, disentangling honesty from accuracy using a… | Instruction Following |
| D | ACCR Daybreak BlueMeasures the share of approved advanced-cyber requests that GPT-5.6 Sol completes rather than refuses under… | Instruction Following |
| D | ArXivMath Jun. 2026ArXivMath Jun. 2026 measures models on final-answer research mathematics problems drawn from June 2026 arXiv… | Math |
| D | ArXivMath Aug. 2026ArXivMath measures models on research-level mathematics problems, reported as a percentage score. | Math |
| D | CMathCMATH measures large language models' ability to solve 1.7k Chinese elementary school math word problems,… | Math |
| D | AIME25AIME25 measures mathematical reasoning using the 30 problems from the 2025 American Invitational Mathematics… | Math |
| D | HMMT Feb 2023HMMT Feb 2023 scores models on 30 competition math problems from the February 2023 Harvard-MIT Mathematics… | Math |
| D | HMMT Feb 2024HMMT Feb 2024 scores models on competition-level math problems from the February 2024 Harvard-MIT Mathematics… | Math |
| D | HMMT Feb 2025HMMT Feb 2025 scores models on short-answer problems from the February 2025 Harvard–MIT Mathematics… | Math |
| D | IPhO 2025IPhO 2025 evaluates models on the three official theory problems of the 2025 International Physics Olympiad,… | Math |
| D | HMMT Nov 2025Measures advanced mathematical reasoning on 30 competition problems from the HMMT November 2025 high school… | Math |
| D | HMMT Feb 2026HMMT Feb 2026 scores models on problems from the February 2026 Harvard-MIT Mathematics Tournament, measuring… | Math |
| D | MMAnswerBenchMMAnswerBench is a math benchmark on which language models are scored by answer accuracy, with results shown… | Math |
| D | Cloudflare Decision Index 0.2.1Cloudflare's Decision Index is an evaluation suite scoring decision models on typed classification and… | Decision Models |
| D | FinancialPhraseBankFinancialPhraseBank measures whether models can classify the sentiment of English financial news sentences as… | Decision Models |
| D | TruthfulQA binaryTruthfulQA binary measures whether language models pick the truthful answer over an imitative falsehood when… | Decision Models |
| D | Decision Index 0.2.1Decision Index 0.2.1 is a leaderboard that scores models on decision tasks such as classification and routing… | Decision Models |
| D | CLadderCLadder measures whether language models can perform formal causal inference (associational, interventional,… | Decision Models |
| D | Decision Index Full ScoreDecision Index 0.4 publishes full scores for 209 evaluated systems. The full score weights private tests of… | Decision Models |
| D | Decision Index KnowledgeDecision Index 0.4 publishes chance-corrected knowledge scores for 209 evaluated systems. | Decision Models |
| D | Decision Index LanguageDecision Index 0.4 publishes chance-corrected language scores for 209 evaluated systems. | Decision Models |
| D | Decision Index RetrievalDecision Index 0.4 publishes chance-corrected retrieval scores for 209 evaluated systems. | Decision Models |
| D | KMMLUKMMLU measures Korean-language expert-level knowledge with 35,030 multiple-choice questions from original… | Korean |
| D | KMMLU-HardKMMLU-Hard is a 4,104-question subset of the Korean expert-level multiple-choice KMMLU benchmark containing… | Korean |
| D | KMMLU-ReduxKMMLU-Redux measures LLMs' Korean expert-level and industrial knowledge using cleaned questions from the… | Korean |
| D | KoBALTKoBALT measures advanced Korean linguistic competence of LLMs via 700 expert-curated multiple-choice questions… | Korean |
| D | HRM8KHRM8K measures mathematical reasoning in Korean with 8,011 problems drawn from Korean exams and competitions… | Korean |
| D | Mental Health BenchMental Health Bench scores how AI models respond in 1,215 synthetic multi-turn mental-health conversations,… | Cross-domain |
| D | AA-SciCodeAA-SciCode is Artificial Analysis's run of SciCode, scoring models' pass@1 on 288 scientific coding subtasks… | Coding |
| D | Korean CSATMeasures LLM performance on ten years of Korean CSAT Korean-language section exams, scored with official… | Korean |
| D | Cohere Embed 5 launch: FinanceBench text retrievalMeasures embedding models' retrieval quality (RCP-nDCG@10) on FinanceBench financial document retrieval as… | Cross-domain |
| D | Cohere Embed 5 launch: FinQA text retrievalFinQA text retrieval measures how well embedding models retrieve relevant financial document passages for… | Cross-domain |
| D | Cohere Embed 5 launch: Chinese retrieval compositeA Cohere-reported composite measuring embedding models' document retrieval quality on Chinese-language queries… | Cross-domain |
| D | A Human-Centric Pipeline for Aligning Large Language Models with Chinese Medical EthicsMedES is a scenario-centric benchmark that scores LLMs on Chinese medical ethics through objective and… | Alignment |
| D | ActivityNet-QAActivityNet-QA measures how well video question answering models answer human-written questions about long web… | Knowledge |
| D | Adaptive Hierarchical Evaluation of LLMs and SAST tools for CWE Prediction in PythonALPHA is a function-level Python benchmark scoring LLMs and SAST tools on CWE-specific vulnerability… | Cybersecurity |
| D | ADR: An Agentic Detection System for Enterprise Agentic AI SecurityWe present the Agentic AI Detection and Response (ADR) system, the first large-scale, production-proven… | Cybersecurity |
| D | AgentCARDAgentCARD is a role-aware benchmark suite that evaluates the cost and accuracy of LLM agent teams across role… | Agentic |
| D | AgentKernelArenaAgentKernelArena measures AI coding agents on GPU kernel optimization across 196 HIP, Triton and… | Agentic |
| D | AgentOdysseyAgentOdyssey evaluates test-time continual learning agents in procedurally generated open-ended long-horizon… | Agentic |
| D | AgentQuestAgentQuest is a modular benchmark framework that measures LLM agent progress on multi-step reasoning tasks… | Agentic |
| A | AIIQ Composite IQAIIQ Composite IQ estimates an IQ-like score for AI models by mapping public benchmark results into calibrated… | Knowledge |
| D | AIRS-BenchAIRS-Bench measures the autonomous machine learning research abilities of LLM agents across 20 tasks sourced… | Agentic |
| D | AMC_2022_23AMC_2022_23 scores models on American Mathematics Competition problems from the 2022-23 year, testing… | Math |
| D | AmharicStoryQAAmharicStoryQA measures LLMs' long-sequence narrative understanding in Amharic through question answering over… | Multilingual |
| D | An Empirical Study of Proactive Coding Assistants in Real-World Software DevelopmentProCodeBench, introduced in this study, measures how well models predict latent developer intent from real IDE… | Coding |
| D | An explainable hypothesis-driven approach to Drug-Induced Liver Injury with HADESThe DILER Benchmark evaluates models on predicting drug-induced liver injury and generating mechanistic… | Biomedicine |
| D | Android in the WildAndroid in the Wild measures how well device-control agents follow natural language instructions by operating… | Mobile |
| D | APEX-v1-extendedAPEX-v1-extended measures whether frontier models can perform economically valuable expert tasks in four jobs:… | Agentic Business |
| D | AppWorldAppWorld measures how well autonomous agents complete day-to-day digital tasks by writing interactive code… | Agentic |
| D | ArchBench ADR GenerationArchBench ADR Generation measures how well LLMs generate Architecture Decision Records from given decision… | Coding |
| D | ArchBench Dynamic Service GenerationArchBench's Dynamic Service Generation task scores how well LLMs and coding agents generate services… | Coding |
| D | ArchBench Microservice GenerationArchBench's Microservice Generation task measures how well LLMs and coding agents generate microservice… | Coding |
| D | Are Agents Ready to Teach? A Multi-Stage Benchmark for Real-World Teaching WorkflowsEduAgentBench measures how well tutor agents handle the full scope of teaching work: pedagogical judgment,… | Education |
| D | Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR AssistancePhysAssistBench measures how well LLMs assist physicians in interactive, multi-turn doctor-patient-EHR… | Healthcare |
| A | Arena AI WebDev ArenaWebDev Arena is a crowdsourced human-preference leaderboard ranking AI models on building front-end web… | Coding |
| B | Arena AI DocumentDocument Arena is a crowdsourced pairwise human-preference leaderboard ranking AI models on document analysis… | Document-ai |
| A | Arena AI Image EditA crowdsourced human-preference leaderboard ranking AI models on single- and multi-image editing tasks. | Multimodal |
| A | Arena AI Text-to-ImageA crowdsourced pairwise human-preference leaderboard ranking text-to-image generation models by Arena score… | Multimodal |
| B | Arena AI Text-to-VideoA crowdsourced pairwise human-preference leaderboard ranking text-to-video generation models by votes on… | Multimodal |
| C | Arena AI Video EditCrowdsourced pairwise human-preference leaderboard ranking AI video editing models by Arena score from user… | Multimodal |
| D | ART (Action-based Reasoning Task)Reliable clinical decision support requires medical AI agents capable of safe, multi-step reasoning over… | Healthcare |
| A | Artificial Analysis Coding Agent IndexAn Artificial Analysis aggregate index scoring coding agents on end-to-end software engineering tasks,… | Coding |
| A | Artificial Analysis Image Editing ArenaRanks image models on editing an image from a text instruction using Elo scores derived from pairwise human… | Multimodal |
| B | Artificial Analysis Image-to-Video ArenaRanks image-to-video generation models (with audio) by Elo from blind pairwise human preference votes on… | Multimodal |
| B | Artificial Analysis Instrumental Music ArenaRanks instrumental music-generation models by Elo from blind pairwise human-panel preference votes on tracks… | Audio |
| C | Artificial Analysis Vocals Music ArenaRanks music-generation models that produce vocals using Elo ratings from blind pairwise human-panel preference… | Audio |
| A | Artificial Analysis Speech Arena — Provider VoicesRanks text-to-speech models by Elo from blind human preference comparisons using each provider's own native… | Audio |
| A | Artificial Analysis Text-to-Image ArenaRanks image generation models on producing an image from a text prompt, using Elo scores from pairwise human… | Multimodal |
| B | Artificial Analysis Text-to-Video ArenaRanks text-to-video models (with and without audio) by Elo from blind pairwise human preference votes on… | Multimodal |
| C | Artificial Analysis Video Editing ArenaRanks video models on editing a video from a text instruction (with audio kept) by Elo from blind pairwise… | Multimodal |
| D | AssistantBenchAssistantBench measures whether web agents can solve 214 realistic, time-consuming tasks by navigating the… | Agentic |
| D | ASTA BenchAstaBench measures AI agents' ability to assist scientific research across literature understanding, code and… | Agentic |
| C | AutoLabAutoLab measures AI agents' auto-research capability on hours-long iterative performance-engineering tasks… | Agentic |
| B | BALROG Official LLM LeaderboardBALROG measures the agentic reasoning, planning and exploration abilities of LLMs and VLMs across a set of… | Game |
| C | BALROG Official VLM LeaderboardBALROG's VLM leaderboard measures the agentic, long-horizon reasoning of vision-language models across game… | Game |
| D | BEHAVIOR 2025 ChallengeBEHAVIOR 2025 Challenge measures robot policies on 50 long-horizon household mobile manipulation tasks in… | Embodied |
| D | BeliefTrackBeliefTrack measures whether language models correctly update, preserve, or isolate their belief state over… | Reasoning |
| A | BenchBenchBenchBench is a meta-benchmark leaderboard that evaluates LLM benchmarks against their peers via standardized… | Meta |
| C | Beyond BinaryBBBench (Beyond-Binary Bench) evaluates GUI critic models on fine-grained ranking of candidate GUI agent… | GUI |
| D | Beyond ContentBeyond Content (ToxiAlert-Bench) measures toxic speech detection in audio, including toxicity arising from… | Speech |
| D | Beyond Fixed Benchmarks and Worst-Case AttacksDynamic Boundary Evaluation (DBE) adaptively locates each LLM's ability boundary on a calibrated item bank… | Safety |
| D | BioASQBioASQ is a challenge that evaluates systems on biomedical semantic indexing, question answering,… | Healthcare |
| D | BRIGHTBRIGHT measures how well text retrieval models find relevant documents for real-world queries that require… | Retrieval |
| D | BugsInPyBugsInPy is a benchmark of real, reproducible bugs in Python projects for controlled testing, debugging and… | Coding |
| D | BugsJSBugsJS is a benchmark of real JavaScript bugs from 10 Node.js projects, with buggy and fixed revisions and… | Coding |
| D | CAIA BenchmarkCAIA measures how well LLM agents answer high-stakes crypto and financial-market analysis tasks, scoring… | Agentic |
| A | CAIS Risk IndexA composite risk index on the CAIS AI Dashboard that scores frontier models on safety risks such as prompt… | Safety |
| B | CAIS Text Capabilities IndexA composite index on the CAIS AI Dashboard that averages frontier model scores on text benchmarks such as… | Knowledge |
| B | CAIS Vision Capabilities IndexA composite index on the CAIS AI Dashboard that averages model scores on vision benchmarks covering embodied… | Vision |
| D | Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost ReasoningChemCost measures whether LLM agents can estimate chemical procurement costs for reactions by grounding… | Chemistry |
| D | Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability?AgenticInterpBench measures whether LM agents can explain the function of components in already-localized… | Reasoning |
| D | Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency OperationsDORA measures how well LLM agents carry out end-to-end disaster response workflows, such as perception,… | Agentic |
| D | Cangjie-benchCangjie-bench measures how well LLMs perform code generation, translation, and program repair in the… | Coding |
| D | ChainWorldChainWorld measures how well computer use agents complete long-horizon desktop workloads built by chaining… | Agentic |
| B | LMArena Vision ArenaLMArena Vision Arena ranks multimodal AI models on reasoning over visual inputs using crowdsourced pairwise… | Multimodal |
| D | ChehreChehre measures how well vision-language models recognize dynamic facial expressions in videos, both the… | Video |
| D | CodeContestsCodeContests is a competitive programming dataset with train, validation and test splits used to evaluate… | Coding |
| D | ContextBenchContextBench measures how well coding agents retrieve and use code context (recall, precision, efficiency)… | Coding |
| D | CrackMeBenchCrackMeBench measures how well language-model agents can reverse engineer symbol-poor binaries in a sandbox to… | Cybersecurity |
| D | CrowS-PairsCrowS-Pairs measures the degree to which masked language models exhibit U.S. stereotypical social biases… | Generalization |
| D | DALPHINDALPHIN measures how well vision-language pathology AI copilots answer diagnostic visual questions on… | Biomedicine Healthcare |
| D | DAWNDAWN measures how well model agents fit quantitative statistical models to data, covering distribution fitting… | Data |
| C | Decomposing and Measuring Evaluation AwarenessMeasures evaluation awareness in frontier language models, i.e., whether models recognize they are being… | Safety |
| D | DeepSurvey-BenchDeepSurvey-Bench evaluates the academic value of automatically generated scientific surveys across… | Summarization |
| D | DigiWorldDigiWorld evaluates computer use agents on 15 realistic sandboxed mobile applications across over 3.2 million… | Agentic |
| D | Disentangling Language Roles in Multilingual LLM Task ExecutionMTM-Bench measures how LLMs execute tasks when instruction, source content, and required response languages… | Multilingual |
| D | Do Agents Know What They Can't Do? Evaluating Feasibility Awareness in Tool-Using AgentsMeasures whether tool-using agents can recognize infeasible tasks (with critical tools masked) and stop… | Agentic |
| D | Do Enterprise Systems Need Learned World Models? The Importance of Context to Infer DynamicsCascadeBench evaluates how well models predict field-level state changes (business-rule cascades) caused by… | Agentic Business |
| D | Do LLMs Build World Models From Text? A Multilingual Diagnostic of Spatial ReasoningMentalMap measures whether LLMs build spatial world models from pure-text descriptions of household scenes… | Spatial Reasoning |
| D | Do Vision-Language-Models show human-like logical problem-solving capability in point and click puzzle games?VLATIM evaluates whether vision-language models can solve physics puzzles in The Incredible Machine 2, from… | Game |
| D | DocILEDocILE measures key information localization and extraction and line item recognition from semi-structured… | Document Understanding Business |
| D | Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action ModelsAct2Answer measures how much commonsense and world knowledge Vision-Language-Action models retain by requiring… | Embodied |
| D | DS-1000DS-1000 measures code generation on a thousand realistic data science problems drawn from StackOverflow across… | Coding |
| D | DUE BenchmarkDUE Benchmark measures how well models understand layout-rich documents across document QA, key information… | Document Understanding |
| D | Efficient Multimodal Clinical Question Answering for Pulmonary Embolism Risk AssessmentEvaluates efficient multimodal LLMs on eight diagnostic and prognostic pulmonary embolism clinical question… | Healthcare |
| D | EgoBabyVLMEgoBabyVLM measures how well vision-language models trained on naturalistic egocentric video (e.g., infant… | Multimodal Understanding |
| D | Embodied Multi-Agent Coordination by Aligning World Models Through DialogueMeasures whether LLM-based embodied agents with partial observability align their world models through… | Embodied |
| D | Emotional intelligence in large language models is fragmented across perceptionFACET evaluates the emotional intelligence of LLMs with 480 expert-crafted items covering perception,… | Human-preference |
| D | EngiAIEngiAI is a benchmark suite that evaluates LLM agents on engineering design workflows, covering tool-use… | Agentic |
| D | ENPMR-BenchENPMR-Bench measures whether memory-augmented emotional support agents can infer users' latent emotional needs… | Agentic |
| D | EnterpriseRAG-BenchEnterpriseRAG-Bench measures how well RAG systems retrieve company-internal documents and generate correct,… | Agentic Business |
| D | EvalCrafterEvalCrafter benchmarks text-to-video generation models on 700 prompts across visual quality, content quality,… | Video |
| D | Evaluating and Achieving Controllable Code Completion in Code LLMC3-Bench measures LLMs' ability to follow user instructions during code completion, beyond functional… | Coding |
| D | Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMsRS-Neg evaluates how well multimodal large language models understand negation in remote sensing imagery… | Vision |
| D | Evaluating Cross-lingual Knowledge Consistency in Code-Mixed vis-a-vis Indian Languages using IndicKLARIndiKLAR measures cross-lingual knowledge recall consistency of LLMs across English, 18 Indian languages, and… | Multilingual |
| D | Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI ReportsMeasures how well open-source LLMs classify sentences from unstructured cyber threat intelligence reports into… | Cybersecurity |
| D | Evaluating the Architectural Reasoning Capabilities of LLM Provers via the Obfuscated Natural Number GameMeasures whether LLM provers can synthesize Lean 4 formal proofs using only local axioms and definitions in an… | Math |
| D | Ex Ante Evaluation of AI-Induced Idea Diversity CollapseMeasures, ex ante from model-only generations versus matched unaided human baselines, how much creative LLM… | Creative |
| D | Executable Benchmarking Suite for Tool-Using AgentsAn executable benchmarking suite that evaluates closed-loop tool-using agents across web (WebArena Verified,… | Agentic |
| D | FAB-BenchFAB-Bench evaluates retrieval-augmented generation systems on semiconductor manufacturing question answering… | Retrieval |
| D | FactoryBenchFactoryBench measures how well time-series models and LLMs understand industrial robot machines from… | Robotics |
| D | FEVERFEVER measures claim verification against Wikipedia, requiring systems to label claims as Supported, Refuted… | Knowledge |
| D | FEVEROUSFEVEROUS measures a model's ability to retrieve evidence from Wikipedia sentences and table cells and verify… | Retrieval |
| D | FieldWorkArenaFieldWorkArena measures how well multimodal agentic AI performs real-world field work tasks, such as detecting… | Domain Business |
| A | Focusable Monocular Depth EstimationFDE-Bench measures target-centric monocular relative depth estimation, scoring how well models prioritize… | Vision |
| D | FoundationalASSISTFoundationalASSIST evaluates whether LLMs can predict K-12 math student performance and exact responses… | Education |
| D | From RAG to Agentic RAG for Faithful Islamic Question AnsweringIslamicFaithQA is a bilingual Arabic/English generative benchmark of 3,810 Islamic questions measuring answer… | Question Answering |
| D | Habitat ChallengeHabitat Challenge is an annual embodied AI challenge that evaluates uploaded navigation agents in novel,… | Embodied |
| B | Harvey Legal Agent BenchmarkOpen-source long-horizon legal agent benchmark where agents work from client-matter instructions,… | Legal |
| D | Health Memory ArenaHealth Memory Arena is an open, live leaderboard for healthcare AI agents that scores models on benchmarks… | Agentic Healthcare |
| D | HELM InstructHELM Instruct scores instruction-following language models with absolute ratings from human and LM evaluators… | Knowledge |
| D | HELM Long ContextHELM Long Context is a leaderboard that evaluates LLMs' long context capabilities on curated tasks like… | Generalization |
| D | HiddenMathHiddenMath measures model performance on an internal holdout set of competition math problems. | Math |
| D | HourVideoHourVideo measures multimodal models' understanding of hour-long egocentric videos through multiple-choice… | Multimodal |
| B | IDP LeaderboardIDP Leaderboard evaluates AI models on document understanding tasks such as OCR, table extraction, key… | Multimodal |
| A | JARVIS-Leaderboard AIJARVIS-Leaderboard AI scores AI models on materials science tasks such as property prediction, force fields… | Materials |
| D | LAB-BenchLAB-Bench measures language models and AI systems on practical biology research tasks such as literature… | Science |
| D | Language-TableLanguage-Table is a multi-task continuous control benchmark measuring language-conditioned robotic tabletop… | Robotics |
| D | LeanDojo BenchmarkLeanDojo Benchmark measures how well language models can prove formal theorems in Lean, using 98,734 theorems… | Math |
| D | LegalBench-RAGLegalBench-RAG evaluates retrieval systems on complex legal contract understanding questions, measuring… | Legal |
| D | LLM Agents Already Know When to Call Tools -- Even Without ReasoningWhen2Tool measures whether tool-augmented LLM agents call tools only when necessary, across 18 environments… | Tool Use |
| B | LMArena Search ArenaSearch Arena ranks LLMs with integrated web search using crowdsourced pairwise human-preference votes on… | Agentic |
| B | LMArena WebDev ArenaLMArena WebDev Arena (Code Arena) ranks models on web development tasks using human preference votes in… | Coding |
| B | LMArena Text ArenaLMArena Text Arena ranks text chat models by crowdsourced anonymous pairwise human-preference votes aggregated… | Generalization |
| D | localmaxxingLocalMaxxing is a community leaderboard that measures local LLM inference speed (tokens/sec,… | Inference |
| D | LocalScoreLocalScore measures how well local hardware runs LLMs, scoring prompt processing speed, generation speed, and… | Inference |
| D | ManiSkill2ManiSkill2 is a unified benchmark of 20 simulated robotic manipulation task families (rigid/soft-body,… | Robotics |
| D | MASSIVEMASSIVE measures multilingual natural language understanding of virtual assistant utterances, scoring intent… | Multilingual |
| D | MatSci MLMatSci ML measures how well machine learning models predict properties of solid-state materials (energies,… | Materials |
| D | MedVidBenchMedVidBench measures how well vision-language models understand medical and surgical videos across video,… | Healthcare |
| D | MineDojoMineDojo is a Minecraft-based simulation benchmark of 3,142 open-ended tasks measuring how well embodied… | Agentic |
| D | MiniF2FMiniF2F measures automated theorem proving systems on formalized olympiad and high-school/undergraduate math… | Math |
| D | MLPerf ClientMLPerf Client measures ML inference performance (e.g., LLM chatbots, image classification) on client devices… | Inference |
| D | MLPerf Inference v6.0MLPerf Inference v6.0 measures AI system inference performance (throughput, latency, energy efficiency) across… | Inference |
| D | Mobile-Agent-EMobile-Eval-E, introduced with the Mobile-Agent-E framework, measures how well mobile agents complete complex,… | Mobile |
| D | NYU LLM CTFNYU CTF Bench evaluates LLM agents on solving real-world capture-the-flag cybersecurity challenges. | Cybersecurity |
| D | OASB Skills Security BenchmarkOASB is a labeled corpus and methodology for evaluating AI agent skill security scanners on detecting… | Cybersecurity |
| D | OmniAI OCR LeaderboardCompares OCR and data extraction performance (JSON accuracy, text similarity, latency, cost) of traditional… | Multimodal |
| B | Open FinLLM LeaderboardOpen FinLLM Leaderboard evaluates LLMs across about 50 financial tasks in seven financial categories, such as… | Finance |
| D | OpenAI EvalsOpenAI Evals is a framework and open registry of evals that score language models on varied tasks, such as… | Knowledge |
| D | PatentBenchPatentBench measures AI systems on patent prosecution tasks such as USPTO Office Action docketing, deadline… | Legal |
| D | PAWS-XPAWS-X measures cross-lingual paraphrase identification using human-translated adversarial PAWS pairs in six… | Multilingual |
| D | PDEBenchPDEBench measures how well machine learning models act as surrogates for time-dependent PDE simulations (fluid… | Science |
| B | PEFT-ArenaPEFT-Arena measures how parameter-efficient finetuning methods trade off target-task gains (math and medical… | Machine Learning |
| D | Poor Paul's BenchmarkPoor Paul's Benchmark measures local LLM inference performance on consumer hardware, ranking model and GPU… | Inference |
| D | ProjDevBenchProjDevBench measures how well AI coding agents build complete, executable software repositories from… | Coding |
| D | PubTables-1MPubTables-1M evaluates models on table detection, table structure recognition and functional analysis in… | Document Understanding |
| C | PulseMindThe PulseMind Benchmark evaluates multi-modal medical models on multi-turn diagnostic consultations, scoring… | Healthcare |
| D | Quantitative Video World Model Evaluation for Geometric-ConsistencyPDI-Bench scores generative video models on geometric coherence (scale-depth alignment, 3D motion consistency,… | Video |
| D | RAGBenchRAGBench measures how well methods evaluate retrieval-augmented generation outputs, using 100k labeled RAG… | Retrieval |
| D | Realm WarrenRealm Warren (Realm legal-reasoning) evaluates models on long-horizon legal work products in litigation,… | Legal |
| D | ResearchGymResearchGym measures how well AI agents perform end-to-end AI research, proposing hypotheses, running… | Agentic |
| D | RoboCasa365RoboCasa365 measures generalist robot policies' task success on kitchen manipulation tasks, including atomic… | Embodied |
| D | RoboDojoRoboDojo evaluates generalist robot manipulation policies on 42 simulation and 18 real-world tasks covering… | Robotics |
| D | RSIBench-DataRSIBench-Data measures whether frontier researcher agents can improve a fixed Qwen3.5 target model by… | Agentic |
| D | Room-Across-RoomRoom-Across-Room (RxR) measures how well agents follow multilingual (English, Hindi, Telugu) natural-language… | Robotics |
| D | SafeArenaSafeArena measures how often autonomous LLM web agents complete or refuse deliberately harmful versus safe… | Safety |
| D | SEATauBenchSEATauBench measures how well conversational tool-use agents perform customer-service tasks (airline, retail,… | Agentic |
| D | SEC-benchSEC-bench evaluates LLM agents on authentic security engineering tasks, namely generating proof-of-concept… | Cybersecurity |
| D | SheetCopilot BenchmarkMeasures how well LLM-based agents complete natural-language spreadsheet control tasks (formatting, charts,… | Spreadsheets |
| B | Speech ArenaSpeech Arena ranks text-to-speech models by Elo ratings from crowd-sourced blind pairwise listening tests. | Audio |
| D | Speech-MASSIVESpeech-MASSIVE is a multilingual spoken language understanding dataset and benchmark that scores models on… | Speech |
| D | SUNRGBDSUN RGB-D is a benchmark suite of 10,000 densely annotated RGB-D indoor images for evaluating scene… | Spatial Reasoning |
| D | SurgVLA-BenchSurgVLA-Bench evaluates vision-language-action models on simulated laparoscopic surgical robotics tasks, from… | Robotics |
| D | SWE-ArenaSWE-Arena is an arena-style platform that compares AI models' coding capabilities in real time with direct… | Coding |
| D | Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective ResponsesCEDAR measures whether multilingual LLMs understand culturally grounded affective responses, scoring emotion… | Multilingual |
| D | TravelPlannerTravelPlanner measures whether language agents can use search tools to gather information and produce… | Agentic |
| D | TUA-BenchTUA-Bench measures how well terminal-use agents complete 120 real-world, execution-scored tasks spanning… | Computer Use |
| D | TurnBenchTurnBench measures how accurately and quickly spoken-dialogue systems detect end-of-turn and interruption… | Audio Speech |
| D | VAKRAVAKRA measures LLM agents' ability to complete multi-step tool-calling tasks over executable APIs and document… | Agentic Business |
| D | WANDRWANDR measures how well research agents perform wide-and-deep web research, discovering many qualifying… | Agentic Business |
| D | WebChoreArenaWebChoreArena measures how well web browsing agents complete 532 tedious, labor-intensive web chores requiring… | Agentic |
| D | WebShopWebShop measures how well language agents can navigate a simulated e-commerce website to search, customize,… | Agentic |
| D | WikiTableQuestionsWikiTableQuestions measures answering complex, multi-step questions over semi-structured Wikipedia tables,… | Knowledge |
| D | WorkArena-L3WorkArena-L3 measures how well web agents complete harder, compositional knowledge-work workflows on the… | Agentic Business |
| D | XNLIXNLI evaluates cross-lingual natural language inference, testing whether models can classify entailment in 15… | Multilingual |
| D | XTREME-RXTREME-R evaluates multilingual models on ten natural language understanding tasks, including… | Multilingual |
| D | Design Arena — 3D DesignDesign Arena's 3D Design leaderboard ranks models by blind crowd votes on generated 3D design outputs. | Multimodal |
| D | Design Arena — ASCII ArtDesign Arena ASCII Art measures which models produce the best ASCII art via blind human pairwise voting… | Multimodal |
| D | Design Arena — Android AppsDesign Arena's Android/Mobile Arena measures how well coding agents build native Android (Kotlin/Jetpack… | Multimodal |
| D | Design Arena — Data VisualizationDesign Arena's Data Visualization category scores models by Elo rating on generating data visualizations. | Multimodal |
| D | Design Arena — Frontend Web AppsDesign Arena's Web App Arena ranks AI models on building functional React web applications (now including… | Multimodal |
| D | Design Arena — Full-Stack Web AppsDesign Arena's full-stack web app evaluation measures whether AI models can build complete, production-ready… | Multimodal |
| D | Design Arena — Godot Game DevelopmentDesign Arena's Godot Game Dev leaderboard ranks models by Elo from head-to-head human preference battles on… | Multimodal |
| D | Design Arena — Graphic DesignDesign Arena — Graphic Design measures model quality at generating graphic design via blind human pairwise… | Multimodal |
| D | Design Arena — HTML Game DevelopmentDesign Arena's Game Dev leaderboard ranks models by Elo rating from head-to-head comparisons of the HTML games… | Multimodal |
| D | Design Arena — Image EditingDesign Arena's Image Editing arena ranks image-editing models by Elo from comparative evaluations, also… | Multimodal |
| D | Design Arena — Image GenerationDesign Arena — Image Generation is an arena leaderboard that ranks models by Elo on image generation tasks. | Multimodal |
| D | Design Arena — Image to VideoDesign Arena's Image to Video leaderboard ranks AI models on generating videos from images using blind crowd… | Multimodal |
| D | Design Arena — Image to Web AppDesign Arena's Image-to-Website leaderboard ranks AI models by user-preference Elo on building websites/web… | Multimodal |
| D | Design Arena — Image to WebsiteDesign Arena's Image-to-Website leaderboard ranks AI models by user-preference Elo on generating websites from… | Multimodal |
| D | Design Arena — Logo DesignDesign Arena — Logo Design is a crowdsourced head-to-head voting benchmark that ranks AI models on logo design… | Multimodal |
| D | Design Arena — Multi-Image to VideoDesign Arena's Multi-Image to Video leaderboard ranks video generation models by Elo on generating videos from… | Multimodal |
| D | Design Arena — Multimodal to VideoDesign Arena's video leaderboard ranks video generation models by Elo ratings from head-to-head human… | Multimodal |
| D | Design Arena — Proprietary Harness SlidesDesign Arena's Slides leaderboard ranks AI presentation-generation models and tools by Elo from blind,… | Multimodal |
| D | Design Arena — Python-PPTX SlidesDesign Arena's Python-PPTX Slides leaderboard ranks models by Elo on automated presentation creation using… | Multimodal |
| D | Design Arena — React Native AppsDesign Arena's Mobile Arena (React Native track) ranks coding agents by blind pairwise user votes on mobile… | Multimodal |
| D | Design Arena — SVG DesignDesign Arena SVG is an Elo-rated, preference-based leaderboard scoring how well models generate SVG designs. | Multimodal |
| D | Design Arena — Text to SpeechA crowdsourced arena that ranks text-to-speech models by Elo from human votes on generated speech. | Multimodal |
| D | Design Arena — UI ComponentsDesign Arena UI Components is a crowdsourced arena where models generate UI components from design prompts and… | Multimodal |
| D | Design Arena — Video EditingDesign Arena's Video Editing leaderboard ranks AI video editing models by blind head-to-head human preference… | Multimodal |
| D | Design Arena — Video GenerationDesign Arena's video generation leaderboard ranks text-to-video models by score on how well they turn written… | Multimodal |
| D | Design Arena — Video to WebsiteDesign Arena's Video-to-Website benchmark measures how well an AI model can watch a reference video of a… | Multimodal |
| D | Design Arena — Web Design OverallDesign Arena is a crowd-voted arena where users compare AI models' outputs on creative tasks such as websites,… | Multimodal |
| D | Design Arena — Website BuildersDesign Arena's Website leaderboard ranks AI models by Elo rating from head-to-head comparisons of the websites… | Multimodal |
| D | Codegolf v2.2Codegolf v2.2 is a code-generation benchmark scored by pass@1, reported in the Gemma 3n model card under STEM… | Coding |
| D | MMBench-LiveMMBench-Live is a continuously evolving multimodal benchmark that evaluates vision-language models with… | |
| D | 3,034-sample human-curated spatial understanding benchmarkMeasures whether vision-language models understand 3D spatial layout via depth-ordered occlusion,… | Spatial Reasoning |
| D | A Closer Look at Failure Modes in Temporal Understanding of Large Audio-Language ModelsA benchmark of 1,657 questions across three foundational tasks measuring temporal reasoning in large… | Audio |
| D | A Visually Impaired Assistance Benchmark for VLM-as-a-Judge EvaluationVIABLE measures how reliably vision-language models act as judges of AI assistance responses for blind and… | Multimodal Understanding |
| D | Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision SupportMeasures how well LLMs actively gather evidence through multi-turn inquiry with a simulated standardized… | Biomedicine Healthcare |
| D | Adversarial Empathy BenchmarkAEB evaluates the robustness of empathetic language models in adversarial emotional dialogues where users… | Behavioral |
| D | AgentCollabBenchAgentCollabBench is a diagnostic benchmark of 900 human-validated multi-agent tasks that measures process… | Agentic |
| D | AgentEscapeBenchAgentEscapeBench measures whether LLM agents can infer, execute, and revise novel tool-use procedures in… | Agentic |
| D | AI Spreadsheet BenchmarkMeasures how spreadsheet AI assistants complete 53 realistic spreadsheet tasks, scoring success… | Spreadsheets |
| A | ALL Bench MultimodalALL Bench Multimodal is a composite leaderboard scoring models on aggregated vision-language, image… | Multimodal |
| D | ALME BenchmarkALME measures how often audio-language models follow conflicting text instead of the audio signal (Text… | Audio |
| D | Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examinationVS-Bench (VisualSwap) measures whether vision-language models genuinely re-examine images during reasoning by… | Multimodal Understanding |
| D | Aristotelian Virtue Profiling of LLMs through Ethical DilemmasVirtueMap profiles LLMs on five Aristotelian virtues by having them rank five responses to each of seven… | Alignment |
| D | Bench4HLSBench4HLS evaluates LLM-generated high-level synthesis (HLS) hardware designs on compilation success,… | Coding |
| D | Benchmarking Large Language Models for Grapheme-to-Phoneme ConversionMeasures how accurately large language models convert Japanese text into kana readings (grapheme-to-phoneme… | Speech |
| D | Benchmarking LLM Agents on Meta-Analysis Articles from Nature PortfolioMetaSyn evaluates LLM agents and RAG pipelines on reproducing meta-analyses from Nature Portfolio journals… | Biomedicine |
| D | Benchmarking Local Language Models for Social Robots using Edge DevicesBenchmarks 25 open-source language models on edge hardware for social-educational robots across inference… | Robotics |
| D | Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity ClassificationMeasures how accurately and efficiently small language models and small reasoning models classify severity… | Classification |
| D | Benchmarking von ASR-Modellen im deutschen medizinischen KontextEvaluates 29 ASR models on a curated dataset of simulated German doctor-patient anamnesis conversations,… | Speech |
| D | Beyond AccuracyBeyond Accuracy evaluates how many distinct valid solution strategies LLMs can produce on 80 AMC 10/12 and… | Reasoning |
| D | Beyond Known FactsA contamination-free synthetic evaluation dataset of predicted future temporal facts with text descriptions,… | Retrieval |
| C | BioToolBioTool measures how accurately LLMs make biomedical tool calls to NCBI, Ensembl and UniProt APIs, scored by… | Biomedicine |
| B | Block-R1Block-R1 is a benchmark for multi-domain reinforcement learning post-training of block-based diffusion large… | Reasoning |
| D | BountyBenchBountyBench evaluates AI agents on detecting, exploiting, and patching real-world bug bounty vulnerabilities… | Cybersecurity |
| D | BPL OCR BenchBPL OCR Bench ranks OCR models on document images using pairwise VLM-as-judge comparisons scored with… | Multimodal |
| D | Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe ExtractionVG-GUI-Bench measures whether MLLM-based GUI agents can follow video tutorials to complete corresponding… | GUI |
| D | CALVINCALVIN is an open-source simulated benchmark measuring how well agents learn long-horizon,… | Embodied |
| D | Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial MalayMalayPrag evaluates LLMs' ability to connect discourse particles in colloquial Malay with their pragmatic… | Multilingual |
| D | Can LLMs Compress (and Decompress)? Evaluating Code Understanding and Execution via InvertibilityRoundTripCodeEval measures whether LLMs can consistently reason about code execution forward and backward by… | Coding |
| D | Can OCR-VLMs Read Devanagari? A Stress-Test Benchmark and Post-Correction StudyMeasures how well ten OCR systems, from classical engines to OCR-VLMs and frontier multimodal LLMs, read… | Ocr |
| D | Can Segmentation Models Understand the World? Towards Proactive Affordance Reasoning via Visual Chain-of-ThoughtAn intent-to-part benchmark that evaluates whether segmentation models can segment the affordance-bearing… | Embodied |
| D | Can You Trust What You See? Human and AI Detection of Synthetic Legal EvidenceMeasures how well humans and multimodal LLMs distinguish authentic legal evidence photographs from… | Legal |
| D | CarCrashNetCarCrashNet benchmarks machine-learning neural solvers on predicting structural vehicle crash simulation… | Aerospace |
| D | CARLA Autonomous Driving LeaderboardEvaluates the driving proficiency of autonomous vehicle agents navigating predefined routes through realistic… | Embodied |
| D | CausaLabCausaLab measures whether LLM agents can perform interactive causal discovery in a synthetic laboratory, both… | Agentic |
| D | CFAgentBenchCFAgentBench measures autonomous construction-finance agents operating across a mock software stack (ERP,… | Agentic |
| D | ChaosBench-LogicChaosBench-Logic measures LLM logical and symbolic reasoning about chaotic dynamical systems via TRUE/FALSE… | Reasoning |
| D | CharadesSTACharades-STA measures how well models temporally localize activities in untrimmed videos given natural… | Multimodal |
| D | ChemSafetyBenchChemSafetyBench measures the accuracy and safety of LLM responses in chemistry, covering chemical property… | Safety |
| D | ChildEvalChildEval measures how well LLMs infer and follow child-centered preferences of children aged 3-6 in… | Education |
| D | ClaimRAG-LAWClaimRAG-LAW evaluates legal retrieval-augmented generation systems in French and English at the claim level,… | Legal |
| D | CLIMATE-FEVERCLIMATE-FEVER measures how well models retrieve evidence for and verify real-world climate change claims,… | Climate |
| D | ClimateBenchClimateBench measures how well emulators predict key climate variables (temperature, diurnal temperature… | Climate |
| D | Code-QA-BenchCode-QA-Bench measures repository-level code understanding by agents, separating genuine code comprehension… | Coding |
| D | CodeXGLUECodeXGLUE measures code intelligence across 10 programming language tasks such as clone detection, defect… | Coding |
| D | Collider-BenchCollider-Bench measures whether LLM agents can reproduce Large Hadron Collider particle physics analyses from… | Science |
| D | CounselBenchCounselBench evaluates large language models on open-ended mental health question answering using expert… | Healthcare |
| D | CoVoST2CoVoST 2 measures speech-to-text translation quality across many languages, covering translation from 21… | Speech |
| D | CSCBenchCSCBench measures LLM reasoning in commodity supply chains through 2.3K+ single-choice questions spanning… | Business |
| D | Decoupled Web Penetration Testing BenchmarkMeasures LLM-based web penetration testing agents' exploitation and reconnaissance capabilities separately… | Cybersecurity |
| D | DedeuceBenchDedeuceBench scores LLM agents on budgeted active-learning episodes, measuring success within a query budget,… | Agentic |
| D | Deep Research Agent BenchmarkMeasures how well frontier deep research agents produce decision-grade management consulting deliverables… | Agentic Business |
| D | DeepResearch Bench MultilingualMultilingual prompt translations of the 100 DeepResearch Bench research tasks across eight languages, for… | Agentic |
| D | DFIR-MetricDFIR-Metric evaluates large language models on digital forensics and incident response through expert-reviewed… | Cybersecurity |
| D | Diagnosing and Mitigating Sycophancy and Skepticism in LLM Causal JudgmentCAUSALT3 is a 454-instance expert-curated benchmark that scores LLM causal reasoning across Pearl's three… | Reasoning |
| D | Diagnosing LLM Arbitration Behavior over Pre-evidence Epistemic States in RAG-based Fact-CheckingPAVE measures how LLM verifiers in RAG-based fact-checking arbitrate between their pre-evidence parametric… | Factuality |
| B | DiagnosticIQDiagnosticIQ measures whether LLMs can recommend correct maintenance actions for industrial assets from… | Reasoning |
| D | Do Open-Vocabulary Detectors Transfer to Aerial Imagery? A Comparative EvaluationMeasures zero-shot transfer of open-vocabulary object detection models to aerial imagery on the LAE-80C… | Geospatial |
| D | Do Text Edits Generalize to Visual Generation? Benchmarking Cross-Modal Knowledge Editing in UMMsUniKE measures whether knowledge edits applied to unified multimodal models' text outputs transfer to their… | Multimodal |
| D | DomainCodeBenchDomainCodeBench measures code generation models on 20 domain-specific tasks in healthcare, finance, molecular… | Coding |
| D | DRIP-RDRIP-R measures how LLM agents make decisions and reason in customer-service return scenarios governed by… | Domain Business |
| C | DrivingGenDrivingGen evaluates generative video world models for autonomous driving on visual realism, trajectory… | Video |
| B | Edge LLM Leaderboard: Raspberry Pi 5Benchmarks throughput, memory/model size, and MMLU accuracy of quantized LLMs running locally on a Raspberry… | Inference |
| D | EditPropBenchEditPropBench measures whether LLM editors propagate factual edits through dependent claims in synthetic… | Factuality |
| D | EditRisk-BenchEditRisk-Bench evaluates how maliciously injected knowledge edits (misinformation, bias, safety violations)… | Safety |
| D | EDU-CIRCUIT-HWEDU-CIRCUIT-HW evaluates multimodal LLMs on recognizing and auto-grading authentic university-level STEM… | Education |
| D | EHRStructEHRStruct measures how well large language models perform 11 clinical tasks over structured electronic health… | Healthcare |
| D | ElecBenchElecBench measures how well large language models handle electric power domain knowledge and four business… | Energy |
| D | EmbSpatialBenchEmbSpatialBench scores vision-language models on embodied spatial understanding, reported under Spatial… | Embodied |
| D | EngiBenchEngiBench measures how well large language models solve real-world engineering problems across three levels:… | Science |
| D | Equation-suffix prediction benchmarkMeasures whether a model's forecast of the hidden remainder of a displayed equation in physics and mathematics… | Math |
| D | ESI-BENCHESI-Bench measures embodied spatial intelligence of multimodal LLM agents that must perceive, navigate and… | Embodied |
| D | Evaluating Long-Horizon Memory for Multi-Party Collaborative DialoguesEverMemBench measures long-horizon conversational memory of LLMs and memory systems over multi-party,… | Long Context |
| D | Evaluating Reward Model Generalization via Pairwise Maximum Discrepancy CompetitionsPMDC evaluates reward model generalization by selecting open-domain prompt-response pairs where two reward… | Alignment |
| D | Evaluating the Pre-Consultation Ability of LLMs using Diagnostic GuidelinesEPAG measures how well LLMs conduct medical pre-consultation, scoring collected History of Present Illness… | Healthcare |
| D | EvasionBenchEvasionBench measures how well models classify the directness of corporate management answers to analyst… | Safety |
| D | Exploring Autonomous Agentic Data Engineering for Model SpecializationEvaluates LLMs as autonomous data engineers that plan, generate and iteratively optimize training data to… | Agentic |
| D | extended needle-in-a-haystack benchmarkAn extended needle-in-a-haystack benchmark measuring how long-context LLMs perform literal extraction, logical… | Long Context |
| D | Extending Beacon to HindiMeasures sycophancy of language models in Hindi versus English using a forced-choice diagnostic with English,… | Alignment |
| D | FActScoreFActScore measures the factual precision of long-form text generated by language models by computing the… | Behavioral |
| D | FATE-HFATE-H measures LLM provers' ability to formally prove, in Lean, 100 hard abstract and commutative algebra… | Math |
| D | FATE-XFATE-X measures LLM provers' ability to formally prove, in Lean, 100 extra-hard abstract and commutative… | Math |
| D | FigQAFigQA is a LAB-Bench subtask of multiple-choice questions testing whether AI systems can reason about… | Multimodal |
| D | Fine-Grained Benchmark Generation for Comprehensive Evaluation of Foundation ModelsA framework that automatically generates textbook-grounded benchmarks with fine-grained metadata, producing… | Domain |
| D | FinToolBenchFinToolBench measures how well LLM agents use 760 real executable financial tools to answer 295 tool-required… | Finance |
| D | FiQA 2018FiQA 2018 is an open challenge that evaluates systems on aspect-based financial sentiment analysis and… | Finance |
| D | FIRE-BenchFIRE-Bench measures whether LLM agents can autonomously rediscover established findings from recent machine… | Agentic |
| D | first bias evaluation dataset for multimodal speech recognitionMeasures bias in multimodal (audio-visual) speech recognition by pairing different faces with the same audio… | Multimodal |
| D | ForeSciForeSci measures whether LLM agents can make forward-looking AI research judgements (which bottleneck,… | Reasoning |
| D | FPEval / FPBenchFPEval/FPBench measures LLM code generation in functional programming languages (Haskell, OCaml, Scala),… | Coding |
| D | From Task Solving to Robust Real-World Adaptation in LLM AgentsMeasures robustness of agentic LLMs in a long-horizon grid-based game under partial observability, dynamic… | Agentic |
| D | FunctionalMATHFunctionalMATH (MATH()) is a functional variant of the MATH benchmark that measures the reasoning gap between… | Math |
| D | GCSE Double-Marked BenchmarkMeasures how closely LLMs agree with human examiners when marking 32,534 double-marked real GCSE student… | Education |
| B | Generating synthetic computed tomography for radiotherapySynthRAD2025 is a challenge benchmark scoring methods that generate synthetic CT from MRI or CBCT for… | Biomedicine Healthcare |
| D | GeoBuildBenchGeoBuildBench measures whether LLMs and multimodal agents can turn natural-language plane geometry problems… | Reasoning |
| D | GeoQAGeoQA measures multimodal numerical reasoning by having models answer geometric questions involving diagrams,… | Knowledge |
| D | GISAGISA measures how well search agents and LLMs answer 373 human-crafted, real-world information-seeking queries… | Agentic |
| B | GR-BenGR-Ben measures how well process reward models and LLMs detect process-level reasoning errors across science… | Reasoning |
| C | Greenland Sovereignty GameMeasures LLM geopolitical behavior by having frontier models play six roles in game-theoretic simulations of… | Game |
| D | GUI-OdysseyGUI-Odyssey measures how well multimodal agents navigate mobile GUIs across multiple apps to complete long,… | GUI |
| C | GuideWebGuideWeb measures how well models automatically generate in-app guides on real-world web UIs by selecting… | GUI |
| D | HANSHANS is an evaluation set that tests whether natural language inference models rely on syntactic heuristics… | Knowledge |
| D | HAO / TBFactHAO/TBFact evaluates LLM-generated patient summaries for Molecular Tumor Boards, using a model-as-a-judge… | Healthcare |
| D | Hidden State Poisoning Attacks against Mamba-based Language ModelsRoBench-25 measures how well language models, especially Mamba-based SSMs, retain information retrieval… | Cybersecurity |
| D | HoudiniVexBenchHoudiniVexBench measures how well models generate, complete and explain VEX shader code for SideFX Houdini,… | Coding |
| D | How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?Measures how well tool-augmented LLM agents solve 243 expert-curated real-world energy market analytics tasks… | Energy |
| D | How Does Personalized Memory Shape LLM Behavior? Benchmarking Rational Preference Utilization in Personalized AssistantsRPEval measures whether LLM-powered personalized assistants rationally use (ignore, support, or let dominate)… | Reasoning |
| D | How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical StudyImmersedPrivacy measures how well vision-language models used as embodied assistants recognize and respect… | Safety |
| D | How Implicit Bias Accumulates and Propagates in LLM Long-term MemoryThe DIB Benchmark measures how implicit bias in LLMs with long-term memory accumulates over time and… | Alignment |
| D | IDEA-Bench ArenaIDEA-Bench Arena ranks image generation models on professional-level design and image processing tasks using… | Multimodal |
| D | (implicit — evaluation of prompting-based defenses against domain-camouflaged injection)Measures how well prompting-based defenses reduce the success of domain-camouflaged prompt injection attacks… | Safety |
| C | InteractiveBench OverallInteractiveBench Overall is a composite score of LLM reasoning through budgeted multi-turn interaction across… | Agentic |
| D | Is He Extroverted? Identifying Missing Relevant Personas for Faithful User SimulationPICQ-drama measures whether LLMs can identify relevant but missing persona dimensions needed to simulate a… | Behavioral |
| D | ISL FinePDFs OCRRanks OCR models on ISL FinePDFs document images using pairwise VLM-as-judge comparisons scored with… | Multimodal |
| D | KamonBenchKamonBench measures how well vision-language models recover compositional factors (container, modifier, motif)… | Vision |
| D | KnotBenchKnotBench measures whether vision-language models can reason about and operate on knot diagrams across 14… | Spatial Reasoning |
| D | L-EvalL-Eval measures long-context language models on 20 sub-tasks over long documents (3k–200k tokens), including… | Generalization |
| A | La LeaderboardLa Leaderboard evaluates generative LLMs on 66 datasets in Basque, Catalan, Galician and Spanish varieties of… | Language |
| D | Large Language Models Lack Temporal Awareness of Medical KnowledgeTempoMed-Bench measures whether LLMs have temporal awareness of medical knowledge, recalling which guideline… | Biomedicine Healthcare |
| B | LatamBoardLatamBoard is a leaderboard and evaluation hub scoring AI models on Latin American and Iberian language tasks… | Multilingual |
| D | Legal RAG BenchLegal RAG Bench measures end-to-end performance of legal retrieval-augmented generation systems on expert… | Legal |
| D | LegalBench-BR v2LegalBench-BR v2 measures how well models classify Brazilian court decisions in Portuguese into five legal… | Legal |
| D | LegalRikai: Open BenchmarkLegalRikai: Open Benchmark evaluates LLMs on four long-form Japanese corporate legal tasks: amendment… | Legal |
| D | LexNeo-BenchLexNeo-Bench measures whether LLMs can classify Luxembourgish tokens as native or as French, German, or… | Multilingual |
| D | LLaVA-BenchLLaVA-Bench evaluates large multimodal models on open-ended visual chat, i.e. answering free-form questions… | Knowledge |
| D | LLMs Infer Cultural Context but Fail to Apply It When RespondingCAPRI evaluates whether LLMs infer a user's cultural background from conversational cues and adapt responses… | Multilingual |
| D | LongDocURLLongDocURL measures how well large vision-language models understand, numerically reason over, and locate… | Document Understanding |
| D | LongMemEval-V2LongMemEval-V2 measures whether memory systems can turn long histories of multimodal web-agent trajectories… | Agentic |
| D | LOVEU-TGVELOVEU-TGVE is a CVPR 2023 competition benchmark scoring text-guided video editing models on 76 videos with… | Multimodal |
| D | MACE-THORMACE-THOR measures how well LLM-based planners enable heterogeneous multi-robot teams to complete long-horizon… | Embodied |
| D | ManiSoftManiSoft measures how well vision-language-action policy models perform manipulation tasks with simulated soft… | Robotics |
| D | MAUDMAUD measures models' reading comprehension of merger agreements, scoring answers to expert-annotated legal… | Legal |
| D | MAWPSMAWPS is a repository of math word problems used to evaluate models on solving arithmetic word problems by… | Math |
| D | MDGYMMDGYM measures whether AI agents can autonomously set up, run and interpret molecular dynamics simulations in… | Agentic |
| D | Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law CourtsTF-RefusalBench measures over-alignment (refusals and disclaimers) of LLMs on multilingual criminal-law… | Legal |
| D | Measuring Safety Alignment Effects in Autonomous Security AgentsA trace-based benchmark of 30 local vulnerability-analysis tasks measuring how safety-aligned versus… | Safety |
| D | MedBrowseCompMedBrowseComp measures how well agentic systems retrieve and synthesize multi-hop medical facts from live,… | Healthcare |
| D | MedCase-StructuredMedCase-Structured measures LLM diagnostic accuracy on clinician-authored diagnostic cases presented as… | Healthcare |
| D | MEGA XStoryClozeMEGA XStoryCloze, part of the MEGA multilingual suite, measures whether generative language models can pick… | Knowledge |
| C | Memes-as-RepliesMeasures whether models can select contextually humorous Japanese manga panel replies to social media posts,… | Multimodal Understanding |
| D | MemLeakMemLeak measures whether facts deleted from multimodal agent memory remain recoverable through retained… | Safety |
| C | Memory-Induced Tool-Drift in LLM AgentsMEMDRIFT measures how personality-driven biases stored in an LLM agent's long-term memory inappropriately… | Agentic |
| D | Memory Retention Is Not Enough to Master Memory Tasks in Reinforcement LearningMeasures RL agents' ability to rewrite (not just retain) memory under partial observability, via Endless… | Agentic |
| D | MemoryQuestMemoryQuest measures how well dialogue assistants retrieve user-specific facts from long multi-session… | Retrieval |
| D | MentisOculiMentisOculi measures whether multimodal models can form, maintain, and manipulate visual representations… | Multimodal Understanding |
| D | Meta-WorldMeta-World measures how well multi-task and meta-reinforcement learning algorithms learn and generalize across… | Robotics |
| C | Metacognitive ProbeThe Metacognitive Probe measures LLM confidence behaviour across five calibration dimensions: confidence… | Behavioral |
| D | MHRC-BenchMHRC-Bench measures how well language models complete repository-level hardware description language code… | Coding |
| D | MIMeBenchMIMeBench is an open-ended benchmark that evaluates LLMs on semantic abstraction and contrastive… | Reasoning |
| D | Mind2WebMind2Web measures how well generalist web agents follow language instructions to complete complex tasks on… | Agentic |
| D | MM-Mind2WebMM-Mind2Web measures how well multimodal web agents plan and execute multi-step actions (click, type, select)… | Agentic |
| D | MMVetMM-Vet evaluates large multimodal models on complicated multimodal tasks that require integrating six core… | Multimodal |
| D | MobileMiniWob++_SRMobileMiniWob++_SR measures the success rate of agents completing 92 MiniWob++ web interaction tasks adapted… | Agentic |
| D | MoleculeNetMoleculeNet benchmarks machine learning methods on predicting molecular properties across curated public… | Science |
| D | Moroccan Darija ASR LeaderboardA leaderboard ranking automatic speech recognition models on Moroccan Darija transcription accuracy. | Speech |
| D | MortarBenchMortarBench measures how accurately LLM agents perform mortgage loan origination tasks such as evaluating… | Finance Business |
| D | MTG-Causal-RLMTG-Causal-RL is a Gymnasium benchmark built on Magic: The Gathering that evaluates RL agents on win rate,… | Game |
| D | Multi-Domain Benchmark for Detecting AI-Generated Text-Rich ImagesMeasures how well AI-generated image detectors identify text-rich images (posters, infographics, receipts,… | Document Understanding |
| D | Multi-domain Multi-modal Document Classification Benchmark with a Multi-level TaxonomyMMM-Bench evaluates how well large multimodal models classify real-world business documents into a five-level… | Document-ai Business |
| D | Multi-genre Semantic Segmentation Benchmark for Conversational ArabicMeasures how well models perform semantic (discourse) segmentation of conversational dialectal Arabic across… | Language |
| D | Multi-Source Cybersecurity LogsEvaluates language models on classifying multi-source (system, network, browser) log chunks as normal or… | Cybersecurity |
| D | multi-target cross-lingual element-awareMEA measures how well LLMs summarize a source document into multiple target languages, covering 24 target… | Summarization |
| D | Multi-View Decompilation BenchmarkMeasures how well LLMs classify decompiled pseudo-C (from Ghidra and RetDec) of compiled binaries as benign or… | Cybersecurity |
| D | Multicultural SpyfallMulticultural Spyfall measures LLMs' multilingual and multicultural capabilities by having them play the… | Game |
| D | MultiHop-RAGMultiHop-RAG measures retrieval-augmented generation on multi-hop queries whose evidence spans 2 to 4… | Generalization |
| D | Multilingual MGSMMGSM measures language models' chain-of-thought math reasoning on 250 grade-school math problems manually… | Math |
| D | MultiPL-EMultiPL-E measures multi-language code generation by translating HumanEval and MBPP unit-test-driven… | Coding |
| D | MultiTurnPSBMultiTurnPSB measures how patient-facing medical chatbots degrade into unsafe responses under four-turn… | Safety Healthcare |
| D | MUTATEMUTATE is an interactive benchmark that evaluates agentic divergent thinking in LLM agents at path level… | Agentic |
| D | NanoGPT-BenchNanoGPT-Bench measures how much historical human progress on the NanoGPT Speedrun autonomous coding agents can… | Agentic |
| D | Needle In A HaystackNeedle In A Haystack pressure-tests LLM long-context retrieval by placing facts (needles) at varying depths in… | Generalization |
| D | Nexus Function CallingNexus Function Calling Benchmark measures how accurately models generate single, parallel and nested function… | Agentic |
| D | NL4COPNL4COP measures how well LLM-based methods translate natural-language combinatorial optimization problem… | Math |
| D | NoReGeo: Non-Reasoning Geometry BenchmarkNoReGeo measures whether LLMs and multimodal models natively understand geometric and spatial relationships,… | Spatial Reasoning |
| D | NRT-BenchNRT-Bench measures the robustness of LLM operator agents running a simulated nuclear power plant control room… | Safety |
| D | NsankuNsanku measures zero-shot machine translation performance of LLMs between English and 43 Ghanaian languages… | Translation |
| D | OCR Bench UFOOCR Bench UFO ranks OCR models on UFO document images using VLM-as-judge pairwise comparisons scored with… | Multimodal |
| D | OfficeBenchOfficeBench measures LLM agents' ability to complete realistic office workflows that span multiple… | Agentic Business |
| C | OMLAB Open Agent Math LeaderboardA leaderboard that compares prompting and agent algorithms (IO, CoT, SC-CoT, PoT, ReAct, ToT) across LLMs on… | Agentic |
| D | OmniInteractOmniInteract measures how well real-time omnimodal models handle streaming audio-visual interaction, scoring… | Multimodal |
| D | On the Cultural Anachronism and Temporal Reasoning in Vision Language ModelsTAB-VLM measures vision-language models' temporal reasoning and susceptibility to cultural anachronism on 600… | Translation |
| A | Open Arabic LLM LeaderboardA leaderboard that evaluates and compares Arabic large language models on Arabic-native benchmarks covering… | Language |
| B | Open ASR LeaderboardOpen ASR Leaderboard benchmarks speech recognition systems on word error rate and inverse real-time factor… | Audio |
| A | Open Chinese LLM LeaderboardA leaderboard from BAAI that evaluates LLMs on Chinese-language versions of key benchmarks covering reasoning,… | Language |
| B | Open Italian LLM LeaderboardA leaderboard that scores open LLMs on Italian-language benchmarks translated into Italian, such as MMLU, ARC… | Language |
| A | Open Japanese LLM LeaderboardA leaderboard evaluating Japanese LLMs across 16 tasks from more than 20 datasets, including NLI, QA,… | Language |
| A | Open Ko-LLM LeaderboardThe Open Ko-LLM Leaderboard evaluates Korean large language models on Korean-language benchmarks covering… | Language |
| A | Open LLM Leaderboard v2Open LLM Leaderboard v2 compares open large language models in a reproducible way, scoring them on IFEval,… | General Knowledge |
| D | Open Multilingual LLM Evaluation LeaderboardA leaderboard ranking LLMs across 29 mostly non-English languages on machine-translated ARC, HellaSwag, MMLU… | Multilingual |
| A | Open Portuguese LLM LeaderboardA public leaderboard that evaluates and ranks large language models on Portuguese-language tasks. | Language |
| B | Open Universal Arabic ASR LeaderboardA continuous leaderboard benchmarking open-source Arabic automatic speech recognition models across… | Speech |
| B | OpenVLM Video LeaderboardOpenVLM Video Leaderboard ranks vision-language models on general video understanding by aggregating scores… | Multimodal Understanding Video |
| D | OpenADMET ExpansionRxA blind community challenge scoring models on predicting nine ADMET properties of late-stage drug-discovery… | Science |
| D | OpenAI-MRCR: 2 needle 1MOpenAI-MRCR 2 needle 1M measures a model's ability to distinguish and retrieve one of two similar needles… | Long Context |
| D | OpenClaw Arena Config LeaderboardMeasures how different SOUL.md personal-agent configurations affect GPT-4.1's score on real-world personal… | Agentic |
| D | OpenScore String Quartets OMREvaluates optical music recognition systems on converting string quartet score page images (scanned and… | Multimodal |
| C | OracleProtoOracleProto measures LLMs' native forecasting ability on resolved real-world events reconstructed as… | Reasoning |
| D | OSUniverseOSUniverse measures how well multimodal GUI-navigation AI agents complete complex desktop tasks, from… | Agentic |
| D | PaperAudit-BenchPaperAudit-Bench measures how well language models detect subtle, within-section and cross-section errors in… | Long Context |
| D | PaperFit-BenchPaperFit-Bench measures how well systems transform compilable LaTeX papers into visually polished,… | Document-ai |
| D | PAREDAPAREDA measures automatic speech recognition accuracy (word error rate) on multi-accent, spontaneous speech… | Speech |
| D | PAS (Police Action Scenarios)The use of Large Language Models (LLMs) in police operations is growing, yet an evaluation framework tailored… | Legal |
| D | PCoAPCoA measures how well language models generate medical aspect-based summaries with phrase-level attribution… | Biomedicine |
| D | PerceptionTestPerception Test measures perception and reasoning skills of pre-trained multimodal models over real-world… | Multimodal |
| D | PerMemBenchPerMemBench evaluates personalized memory systems for LLM agents, testing whether they retain user-relevant… | Agentic |
| D | Personalized Turn-Level User Conversation Satisfaction BenchmarkPersTurnBench measures how well generation models satisfy individual users at specific conversation turns,… | Human-preference |
| D | PetroBenchPetroBench measures LLM knowledge in petroleum engineering (production, reservoir, drilling) with 1,200… | Domain |
| D | Phoenix-benchPhoenix-bench measures whether AI agents can resolve real hardware engineering bugs in Verilog repositories,… | Coding |
| D | Physics Question Scene GraphPQSG evaluates the physical plausibility of text-to-video model outputs using a graph of VLM-generated yes/no… | Reasoning |
| B | PII Masking BenchmarkPIIMB measures zero-shot PII masking: a model's ability to detect and mask personally identifiable information… | Safety |
| C | PoisonForgePoisonForge measures how vulnerable instruction-tuned open-weight LLMs are to task-level targeted data… | Safety |
| D | PolyMath-enPolyMath measures mathematical reasoning of LLMs across 18 languages and 4 difficulty levels; PolyMath-en is… | Math |
| D | Power Systems Agent BenchmarkMeasures how well tool-using AI agents solve executable electric power engineering tasks (power flow,… | Energy |
| D | PR ArenaPR Arena is a public leaderboard that ranks AI coding agents by how many of the GitHub pull requests they… | Coding |
| C | Prediction Bottlenecks Don't Discover Causal StructureA reusable falsification benchmark that tests claims that causal-discovery methods (e.g., prediction… | Science |
| D | ProjectionBenchProjectionBench measures LLMs' ability to generate scientific hypotheses as paper details are progressively… | Reasoning |
| D | ProtStructQAProtStructQA measures whether language models can answer natural-language protein structural questions by… | Biomedicine |
| D | Pseudo-Deliberation in Language ModelsMeasures alignment between LLMs' stated values and their reasoning and generated dialogue actions (the… | Alignment |
| D | PTCBENCHPTCBENCH measures how consistent LLM personalities (assessed via the NEO Five-Factor Inventory) remain across… | Behavioral |
| D | PulseBench-TabPulseBench-Tab measures how accurately table extraction systems convert document-image tables across 9… | Document Understanding |
| D | Push Your AgentPushBench measures quantitative goal persistence: whether long-horizon LLM agents keep working with tool calls… | Agentic |
| D | QasperQasper measures how well models answer information-seeking questions grounded in the full text of NLP research… | Long Context |
| D | Quantifying the Gap between Understanding and Generation within Unified Multimodal ModelsGapEval measures the gap between understanding and generation in unified multimodal models via bidirectional… | Multimodal Understanding |
| D | Query Symbolically or Retrieve Semantically? A Dataset and Method for Semi-Structured Question AnsweringSpecsQA measures question answering over semi-structured product specification documents from a commercial… | Question Answering |
| D | QuixBugsQuixBugs measures automated program repair on 40 classic algorithm programs in Python and Java, each… | Coding |
| D | RealICURealICU measures how well LLM agents assess patient status, acute problems, recommended actions and red-flag… | Healthcare |
| D | Redundant or Necessary? A Benchmark for Detecting Redundant Steps in Agent TrajectoriesRedundancyBench measures how well methods can detect redundant versus necessary steps in annotated LLM agent… | Efficiency |
| B | RefCOCO-avgRefCOCO-avg measures object grounding accuracy in images, averaged across the RefCOCO, RefCOCO+, and RefCOCOg… | Spatial Reasoning |
| D | RefSpatialBenchRefSpatialBench measures spatial reference understanding and grounding in vision-language models. | Spatial Reasoning |
| D | ReproRepoReproRepo evaluates whether LLM agents can identify real-world reproducibility blockers in ML paper-repository… | Uncategorized |
| D | RigorBenchRigorBench measures engineering process discipline (planning, verification, recovery, abstention, atomic… | Software Engineering |
| D | RLBenchRLBench measures vision-guided robot manipulation learning across 100 hand-designed simulated tasks, from… | Robotics |
| D | RMGAPRMGAP measures how well reward models generalize by correctly ranking responses to match diverse user… | Alignment |
| D | Roboflow Vision Evals - OCRMeasures how accurately vision models read text from images, combining OCRBench, a license-plate recognition… | Vision |
| D | Roboflow Vision Evals - SegmentationMeasures open-vocabulary instance segmentation: models output pixel masks from text queries, scored by… | Vision |
| D | Robust Asynchronous Planning via Auto-FormalizationMeasures how well LLMs plan asynchronous tasks with non-uniform durations, concurrency and execution-time… | Reasoning |
| C | Robusto-2Robusto-2 benchmarks humans and VLMs on visual question answering over dashcam footage from Lima and New York… | Vision |
| D | RubricsTreeRubricsTree evaluates open-ended responses of LLM personal health agents using a hierarchical taxonomy of over… | Healthcare |
| D | RW-PostRW-Post measures how well large vision-language models fact-check real-world social-media text-image posts,… | Factuality |
| D | SAFE Audio Challenge Task 1SAFE Audio Challenge Task 1 evaluates how well detection models distinguish raw synthetic speech generated by… | Audio |
| D | SAFE Audio Challenge Task 2SAFE Challenge Task 2 benchmarks synthetic speech detection models on processed audio (e.g., compression,… | Audio |
| D | SAFE Audio Challenge Task 3SAFE Audio Challenge Task 3 measures how well detection models identify synthetic speech in laundered audio… | Audio |
| D | SALTSALT measures LLM uncertainty estimation (correctness, calibration, and confidence ranking) in long-form… | Truthfulness |
| D | Scammer4UScammer4U measures how often autonomous web agents leak users' personally identifiable information to… | Safety |
| D | SCDBenchSCDBench measures how well LLMs decompile smart contract bytecode back into Solidity source, scoring format… | Software Engineering |
| D | SciBenchSciBench measures large language models' ability to solve college-level scientific problems in mathematics,… | Science |
| D | SCRuBEvalSCRuBEval measures how well language models reason about social concepts (norms, culture, institutions) with… | Reasoning |
| D | SEAGymSEAGym is an evaluation environment that measures how self-evolving LLM agents' harness updates affect… | Agentic |
| D | SEC-QASEC-QA evaluates how well models answer multi-document, long-context financial questions over SEC filings that… | Finance |
| D | self-evolving instrument for subjective behavior evaluationA self-evolving evaluation instrument that scores LLMs on subjective, human-facing behaviors such as emotional… | Behavioral |
| D | SlakeVQASLAKE (SlakeVQA) evaluates medical visual question answering systems on bilingual English-Chinese questions… | Healthcare |
| D | SLVMEvalSLVMEval meta-evaluates text-to-video evaluation systems on long videos (up to ~3 hours), testing whether they… | Multimodal |
| D | SocialBias-BenchSocialBias-Bench measures social bias in LLM-generated code across 343 real-world coding tasks spanning seven… | Coding |
| D | SpaceRejectSpaceReject measures whether embodied vision-language agents correctly refuse deliberately unanswerable… | Embodied |
| C | SpaRRTaSpaRRTa measures the ability of visual foundation models to recognize relative spatial positions of objects in… | Spatial Reasoning |
| D | SpecXSpecX measures how well specialized spectral models and multimodal language models perform molecular… | Chemistry |
| B | SphinxSphinx is a checklist-based benchmark that evaluates LLM pull request review quality by coverage of actionable… | Software Engineering |
| D | SQuALITYSQuALITY measures how well models write overview and question-focused summaries of long public-domain short… | Long Context |
| D | ST-VQAST-VQA measures visual question answering that requires reading and understanding scene text in images. | Knowledge |
| D | StableToolBenchStableToolBench measures LLMs' ability to use external tools/APIs, scoring solvable pass and win rates using a… | Agentic |
| B | SuiChat-CNSuiChat-CN measures how well language models assess users' suicide risk levels from contextual multi-party… | Healthcare |
| D | SUPERBSUPERB measures how well a shared (self-supervised) speech model performs across a wide range of speech… | Speech |
| D | SurgGoalSurgGoal is a multicentric meta-evaluation benchmark assessing whether vision-language models produce valid… | Healthcare |
| B | SWE AtlasSWE Atlas benchmarks coding agents on software engineering tasks beyond issue resolution. | Software Engineering |
| D | SWE-ChainSWE-Chain measures how well coding agents implement chained release-level upgrades to real Python packages,… | Software Engineering |
| D | SWE-PolyBenchSWE-PolyBench measures how well AI coding agents resolve repository-level software engineering tasks (bug… | Coding |
| D | Synthesis and Evaluation of Long-term History-aware Medical DialogueEvaluates healthcare agents' memory and reasoning over synthetic long-term patient medical dialogue histories… | Healthcare |
| D | T2QBenchT2QBench measures whether LLM agents acquire a grounded understanding of their environment, using 1,967 QA… | Agentic |
| A | Tau2 AverageTau2 measures how well conversational AI agents solve customer-support-style tasks in dual-control… | Agentic |
| D | The Abstraction Gap in Vision-Language Causal ReasoningCAGE measures whether vision-language models perform faithful causal reasoning over images versus merely… | Reasoning |
| D | The Curse of HelpfulnessDistractionIF measures LLM robustness to benign instruction-like distractor text embedded in reference… | Instruction Following |
| D | The Register GapThe Register Gap evaluates language models on the Meaning Intelligence Framework, scoring nine dimensions of… | Behavioral |
| B | The Wrong Kind of RightVETO, introduced in The Wrong Kind of Right, measures misfired alignment: how often LLMs reject… | Alignment |
| D | Theory-Scale Auto-Formalization of Logics for Computer ScienceLCS-Bench measures theory-scale auto-formalization and theorem-proving in Lean, translating hundreds of… | Math |
| D | TOFU LLaMA 10%TOFU measures how well unlearning methods make LLMs forget fictitious author profile QA data (here the LLaMA,… | Safety |
| D | TOFU LLaMA 1%TOFU measures how well unlearning methods make LLMs forget fictitious author profiles (here the 1% forget-set… | Safety |
| D | TOFU LLaMA 5%TOFU measures how well unlearning methods make LLMs forget fictitious author profile QA data (here LLaMA… | Safety |
| D | TOFU Phi 10%TOFU measures how well unlearning methods make LLMs forget a forget set of synthetic fictitious author QA… | Safety |
| D | TOFU Phi 1%TOFU measures how well unlearning methods make LLMs forget fictitious author profiles (here the Phi model with… | Safety |
| D | TOFU Phi 5%TOFU measures how well unlearning methods make LLMs forget a subset (here 5%) of fictitious author profile QA… | Safety |
| D | Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-CheckingFactArena benchmarks LLMs across the full fact-checking pipeline—claim extraction, tool-augmented evidence… | Factuality |
| D | Towards Direct Evaluation of Harness Optimizers via Priority RankingDirectly evaluates harness optimizer agents by having them rank harness components (prompt, tool, workflow,… | Agentic |
| D | Towards Evaluation of Implicit Software World Models in Coding LLMsMeasures whether coding LLMs can predict runtime behaviour of real software (test outcome, exception class,… | Software Engineering |
| D | Towards Understanding and Measuring COGNITIVE ATROPHY in LLM BehaviourMeasures cognitive atrophy in LLM mental-health support conversations: whether model responses help users keep… | Safety |
| B | TraitFactoryTraitFactory measures how well methods steer language models on multiple behavioral attributes (behavioral… | Behavioral |
| D | Truthful AI AdvisorsMeasures whether LLM advisors stay truthful when their objective is misaligned with the user's, scoring… | Alignment |
| D | UAV-VLN-FOVUAV-VLN-FOV measures whether a UAV agent can ground a visible target from vision-language instructions and… | Uncategorized |
| D | Uncovering Entity Identity Confusion in Multimodal Knowledge EditingEC-Bench is a diagnostic benchmark that measures Entity Identity Confusion in multimodal knowledge-edited… | Multimodal Understanding |
| D | Uncovering Political Bias in Large Language Models using Parliamentary Voting RecordsMeasures political bias in LLMs by comparing model-generated voting predictions on parliamentary motions with… | Alignment |
| D | Understanding Mental States to Guide Social Influence in Multi-Person Group DialogueSocialMindChange measures whether language models can use Theory of Mind to generate dialogue in multi-person,… | Behavioral |
| D | Unintended Long-Term State Poisoning BenchULSPB measures how routine multi-turn user interactions poison a personalized LLM agent's persistent long-term… | Safety |
| D | VAIRVAIR measures whether large reasoning models can correctly evaluate math solutions that reach valid answers… | Reasoning |
| D | VATEXVATEX is a multilingual (English-Chinese) video-and-language benchmark evaluating multilingual video… | Multimodal |
| D | Vera-BenchVera-Bench measures the safety of tool-using LLM agents against attacks, using 1,600 executable safety cases… | Safety |
| D | Veterinary Clinical Summarization EvalEvaluates commercial veterinary-focused LLM tools on summarizing veterinary oncology records, scored by a… | Healthcare |
| D | VIDAVIDA evaluates whether multimodal models use visual evidence to resolve visually dependent ambiguous… | Translation |
| D | VIMA-BenchVIMA-Bench is a simulation benchmark of procedurally generated tabletop robot manipulation tasks specified by… | Robotics |
| D | Vision Language Models for Optimization-Driven Intent Processing in Autonomous NetworksIntentOpt measures whether vision-language models can turn annotated network diagrams into correct… | Coding |
| D | VisIT-Bench Multiple ImagesVisIT-Bench measures how well vision-language models follow real-world instructions over images, scored… | Multimodal |
| D | VisualLeakBenchVisualLeakBench measures whether vision-language agents propagate sensitive or unsafe text visible in images… | Safety |
| D | VLABenchVLABench evaluates vision-language-action models and VLM workflows on 100 categories of language-conditioned… | Multimodal |
| D | VLegal-BenchVLegal-Bench measures how well large language models understand, reason about, and apply Vietnamese law across… | Legal |
| D | VLMsAreBlindVLMsAreBlind (BlindTest) measures whether vision-language models can perform simple low-level visual tasks… | Multimodal |
| D | Voice BenchmarkVoice Benchmark ranks voice AI platforms by measured real-world end-to-end response latency from recurring… | Audio |
| D | VSD FashionVSD Fashion scores models on zero-shot in-catalog visual similarity retrieval of fashion images, reporting ROC… | Vision |
| D | WBCMor VQAWBCMor VQA measures how well vision-language models answer bilingual English–Urdu morphology-aware questions… | Healthcare |
| D | WebTailBenchWebTailBench scores web agents on realistic, long-tail browser tasks such as compositional searches, bookings… | Agentic |
| D | What Does the AI Doctor Value? Auditing Pluralism in the Clinical Ethics of Language ModelsAudits the clinical ethical value priorities of language models using a benchmark of clinician-verified… | Healthcare |
| D | What Makes LVLMs Hallucinate Less? Unveiling the Architectural Factors Behind Hallucination RobustnessCoSimUE measures hallucination robustness of large vision-language models across co-occurrence, similarity,… | Factuality |
| D | When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech RecognitionMeasures whether audio-language models can use clinical context (diagnosis labels, speech ratings, clinical… | Audio Healthcare |
| C | When English Rewrites Local KnowledgeMeasures whether LLMs answering culturally grounded Bengali questions in Bangla versus English substitute… | General Knowledge |
| C | When Symptoms Are Not EnoughMeasures how accurately LLMs screen for psychiatric disorders (anxiety, depression, PTSD, any disorder) from… | Healthcare |
| D | Who Generated This 3D Asset? Learning Source Attribution for Generative 3D ModelsA benchmark for passive source attribution of generated 3D assets, testing whether and which of 22 3D… | Vision |
| D | Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM AgentsPrincipalBench measures whether multi-party LLM agents stay loyal to the principal they represent against a… | Agentic |
| D | WMT23WMT23 General Machine Translation shared task evaluates machine translation systems across multiple language… | Translation |
| D | Workspace-Bench 1.0Workspace-Bench measures how well AI agents complete workspace tasks that require retrieving, reasoning over… | Agentic Business |
| D | World-Ego Modeling for Long-Horizon Evolution in Hybrid Embodied TasksHTEWorld, introduced in this paper, measures long-horizon world modeling (future video prediction) for robots… | Embodied |
| B | WorldArena Track 1WorldArena Track 1 scores embodied world models on video perception quality and embodied task functionality,… | Multimodal |
| D | XLGoBenchXLGoBench uses synthetic algorithmic tasks posed in different languages to measure cross-lingual gaps in large… | Reasoning |
| C | Constraint Saturation EvaluationConstraint Saturation Evaluation measures how well large language models follow many simultaneous explicit… | |
| D | ELBenchELBench evaluates education-facing large language models across general capability, safety and… | |
| D | EvoPolicyGymEvoPolicyGym measures how well harness-model agents iteratively edit and improve executable policies in… | |
| D | JavaVulBenchJavaVulBench measures how well models detect security vulnerabilities in Java code at method and line level,… | |
| D | MathAdvMathAdv measures advanced undergraduate- and graduate-level mathematical reasoning across 13 domains through… | |
| D | MedStreamBenchMedStreamBench measures whether vision-language models give correct and well-timed answers and proactive… | |
| D | RustMizanRustMizan measures how well LLM agents detect, classify (CWE), and localize vulnerabilities in compilable Rust… | |
| D | SocietyBenchSocietyBench measures whether language models can forecast how real social events unfold over time, using… | |
| D | SPEARBenchSPEARBench evaluates the conversational naturalness of streaming speech-to-speech language models across… | |
| D | VGA-BenchV2VGA-BenchV2 evaluates video generation models on aesthetic value and generation quality using human-aligned… | |
| D | YOMI-BenchYOMI-Bench measures how well LLMs read kanji and understand Japanese phonology, through four tasks on reading… | |
| D | Are Video Models Zero-Shot Learners and Reasoners in Education? EduVideoBenchEduVideoBench evaluates whether video generation models produce educationally valid videos across knowledge,… | Education |
| D | Can Large Language Models Reliably Code Qualitative Humanitarian Data? A Benchmark Study Against Human Expert AdjudicationMeasures how reliably LLMs perform deductive coding of qualitative humanitarian interview transcripts compared… | Coding |
| D | FinBoardBenchFinBoardBench evaluates LLMs' dynamic wealth management and strategic financial decision-making through… | Finance |
| D | Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions from Japan's National Assessment of Academic AbilityMeasures multimodal LLMs on officially released Japanese middle-school exam items in Science, Mathematics, and… | Education |
| D | InvestPhilBenchInvestPhilBench measures whether large language models can accurately reconstruct and apply the procedural… | Finance |
| D | KGCEKGCE measures how well multimodal LLM agents execute educational tasks across Windows, Android, and… | Education |
| D | Legal Hallucination Detection BenchmarkAttorneys, judges, and pro se filers increasingly use AI to draft legal documents, yet these tools frequently… | Legal |
| D | Towards Error-Free EHRsEHR-ReasonCon measures how well models verify consistency between clinical notes and structured tables in… | Healthcare |
| S | HLEAn expert-authored benchmark designed to probe frontier knowledge and reasoning. | Knowledge |
| S | HLE (Mercor)Answering difficult academic questions across many subjects. This table shows Mercor-run configurations for… | Knowledge |
| A | HLE w/ tools | Agentic |
| A | WebArena-VerifiedWebArena-Verified is an audited release of the WebArena browser-agent benchmark. It rechecks task… | Agentic |
| A | Codeforces ELO | Coding |
| A | Hy-BrowseComp-Pro2 | Search |
| A | SimpleQA Net Score | Knowledge |
| A | LifeSciBench | Science Long-Horizon |
| A | BioMysteryBench | Agentic |
| A | EQ-Bench 3 | Instruction Following |
| A | LOCA-Bench 256KTests long-context agents as tool environments grow to a 256K-token environment description. | Agentic |
| A | GDPval rubricsA display-only provider-table GDPval rubric score for economically valuable work tasks. | Agentic |
| A | GMMLUMMLU-style knowledge evaluation across 42 high- and low-resource languages. | Multilingual |
| A | HELM LegalBenchHELM LegalBench: Measures legal reasoning, contract review, statute interpretation, or legal-domain QA. | Legal |
| A | SQASQA: Measures structured-data reasoning over tables, spreadsheets, charts, databases, or data analysis tasks. | Knowledge |
| A | Hy-CompanyBench V2Tencent internal evaluation listed in the agentic-coding section of the HY4 preview benchmark appendix. | Coding |
| A | Hy-FinAgentBench | Finance |
| A | Hy-LifeSearch | Search |
| A | timm ImageNet Robustness | Vision |
| A | 100Q-Hard Net Score | Knowledge |
| A | AraGen v3Arabic generative-task benchmark for chat and instruct models, evaluated with the 3C3H rubric over culturally… | Language |
| A | LABBench2 Source Reliability | Biology |
| A | Multilingual Factual Questions Net Score | Multilingual |
| A | OpenUGI | Alignment |
| A | SWE-ProSoftware performance optimization is a notoriously complex and manual task. Despite the growing use of Large… | Software Engineering |
| B | Excel Modeling Benchmark | |
| B | SWE-Marathon | Coding |
| B | OpenThoughts | Coding |
| B | OpenReward | Finance |
| B | MazeBenchTests whether a coding agent can explore a 256-room 3D puzzle world and collect its 100 hidden gems, using… | Agentic |
| B | ParseBenchScores document parsers and vision-language models on 2,078 human-verified enterprise pages across tables,… | Multimodal |
| B | MultiChallengeTests instruction retention, memory of user information, versioned editing, and coherence over multi-turn… | Instruction Following |
| B | Web Search Index | |
| B | Autoresearch | ML Science |
| B | benchmarks.bio | Science |
| B | CREW | Coding Enterprise Workflows |
| B | FrontierSWE v2 | Coding Long-Horizon+1 |
| B | CUA data | Computer Use Tool Use |
| B | Gandalf the Grader | Multi-Domain |
| B | Human Creativity Benchmark | Creative Coding |
| B | AIMSWe argue that safety classifiers should model user intent as an explicit signal between the prompt and the… | Safety |
| B | EnterpriseOps-GymServiceNow benchmark of stateful planning and tool use across simulated enterprise systems, as run by model… | Agentic |
| B | ToolSandboxApple benchmark of stateful, conversational tool use, where tool calls change state and a simulated user takes… | Agentic |
| B | RedlineBenchNegotiating and revising contracts across multiple turns. This table shows Mercor-run configurations for… | Agentic |
| B | FinanceArenaAn AfterQuery benchmark of open-ended financial analysis that requires models to read financial data, make… | Knowledge |
| B | Multi-IFMeta benchmark of instruction following over multi-turn conversations in eight languages. | Instruction Following |
| B | AgentIFInstruction-following benchmark built from real agentic applications, with long instructions that carry many… | Instruction Following |
| B | Beyond Literal MappingLarge Language Models (LLMs) have significantly advanced Machine Translation (MT), applying them to… | Translation |
| B | BLXBench | Coding |
| B | A Positive Case for FaithfulnessLLM self-explanations are often presented as a promising tool for AI oversight, yet their faithfulness to the… | Alignment |
| B | AI-Secure LLM Trustworthy LeaderboardTrustworthiness benchmark for LLMs covering toxicity, stereotypes, adversarial robustness, out-of-distribution… | Safety |
| B | Chart-FR1Multimodal large language models (MLLMs) have shown considerable potential in chart understanding and… | Reasoning |
| B | Clembench Multimodal v1.6.5Clembench Multimodal evaluates chat-optimized multimodal models as conversational agents through visual… | Agentic |
| B | FinSTaRTime series (TS) reasoning models (TSRMs) have shown promising capabilities in general domains, yet they… | Finance |
| B | GeoDialSeveral educational domains rely heavily on diagrams and visual cues, yet most existing tutoring datasets are… | Education |
| B | HindiGen v1Hindi generative-task benchmark for chat and instruct models, evaluated with the 3C3H rubric across Hindi QA,… | Language |
| B | InfiniteBM Liar's DiceHead-to-head LLM game-arena ladder for Liar's Dice, using InfiniteBM's per-game Bradley-Terry Elo ratings… | Game |
| B | kluster.ai LLM Hallucination Detection Leaderboard | Behavioral |
| B | LanguageBenchLanguageBench evaluates AI models across many languages and tasks, including translation, classification,… | Multilingual |
| B | Med-CoReasonerWhile reasoning-enhanced large language models perform strongly on English medical tasks, a persistent… | Biomedicine |
| B | Mixing Expert KnowledgeLarge language models (LLMs) have demonstrated exceptional performance in reasoning tasks such as mathematics… | Game |
| B | PlanAheadDespite recent advances, LLM-based web agents still struggle with limited exploration, omission of critical… | Agentic |
| B | PROBE Family ClassificationPROBE drug-target protein family classification benchmark, reporting MCC across random and sequence-identity… | Science |
| B | PROBE Semantic SimilarityPROBE semantic similarity benchmark for protein representation methods, measuring correlation with Gene… | Science |
| B | Reeve Foundation Multilingual CorpusBuilding on our previous work, this paper develops practical, low-barrier methods for freelance translators… | Translation |
| B | RoboAlign-R1 | Robotics |
| B | SearchGymSearch agents have emerged as a pivotal paradigm for solving open-ended, knowledge-intensive reasoning tasks.… | Search |
| B | TabEmbedFoundation models have established unified representations for natural language processing, yet this paradigm… | Tabular |
| B | Triaging Threats to Specialized Guardrails | Safety |
| C | Turing Terminal-Bench Data | Coding Enterprise Workflows+1 |
| C | BrowseComp+Evaluates research agents against a fixed document corpus, separately from live-web BrowseComp. | Agentic |
| C | OSWorld-Verified | Computer Use Enterprise Workflows |
| C | OSWorld Windows tasks on Daytona Windows sandboxes | Coding Computer Use |
| C | SWE-bench MultimodalJavaScript issue resolution where bug reports depend on screenshots, evaluated by Vals AI with no agent… | Coding |
| C | Technical Report | Enterprise Workflows |
| C | Agent-environment papers | Coding Computer Use+2 |
| C | MMLU Professional MedicineMMLU Professional Medicine: Evaluates clinical, biomedical, medical-exam, coding, or healthcare-document… | Healthcare |
| C | Firehorse open harness library | Coding Long-Horizon |
| C | inspect-openreward | Coding Long-Horizon |
| C | Public Benefits Bench | |
| C | KernelQuest | Coding Enterprise Workflows+1 |
| C | PLSQLBench | Coding Enterprise Workflows+1 |
| C | Turing Frontier Agentic STEM | Coding Enterprise Workflows+1 |
| C | ConjectureBench: 14,865 open math problems | Long-Horizon |
| C | Dynamic Semantic Tags technical report with Intuit | Long-Horizon |
| C | The Valley of Code ReasoningThe Valley of Code Reasoning (arXiv 2510.06101) | Coding Computer Use+2 |
| C | Cats Confuse Reasoning LLMCats Confuse Reasoning LLM (arXiv 2503.01781) | Coding Computer Use+2 |
| C | Self-rationalization improves LLM as a fine-grained judgeSelf-rationalization improves LLM as a fine-grained judge (arXiv 2410.05495) | Coding Computer Use+2 |
| C | AI bosses studies, parts 1 and 2 | Computer Use Long-Horizon |
| C | env0 SFT study on SkillsBench/ClawsBench | Coding Computer Use+1 |
| C | GenesisBench | Coding Computer Use+1 |
| C | FinQA / SnorkelFinance environment and benchmark | Coding Computer Use+2 |
| C | RIFT / Hitchhiker's Guide to Rubric QualityRIFT / Hitchhiker's Guide to Rubric Quality (arXiv 2604.01375) | Coding Computer Use+2 |
| C | Shrinking the Generation-Verification Gap with Weak Verifiers | Coding Computer Use+2 |
| C | Snorkel: Rapid Training Data Creation with Weak Supervision | Coding Computer Use+2 |
| C | Human Baselines for Benchmarks | Coding Enterprise Workflows+1 |
| C | INTELLECT-3 Technical ReportINTELLECT-3 Technical Report (https://arxiv.org/abs/2512.16144) | Coding Enterprise Workflows+2 |
| C | INTELLECT-2INTELLECT-2 (https://www.primeintellect.ai/blog/intellect-2) | Coding Enterprise Workflows+2 |
| C | Systematic Reward Hacking studySystematic Reward Hacking study (https://primeintellect.ai/blog/reward-hacking) | Coding Enterprise Workflows+2 |
| C | True Agents Model the World / ECHOTrue Agents Model the World / ECHO (https://primeintellect.ai/blog/true-agents-model-the-world) | Coding Enterprise Workflows+2 |
| C | Scaling Agentic RL: 365k+ SWE/terminal/search tasksScaling Agentic RL: 365k+ SWE/terminal/search tasks (https://primeintellect.ai/blog/scaling-agentic-rl) | Coding Enterprise Workflows+2 |
| C | No Last Mile: A Theory of the Human Data MarketNo Last Mile: A Theory of the Human Data Market (arXiv 2603.00932) | Enterprise Workflows |
| C | PII Transformation for Enterprise Datasets | Enterprise Workflows |
| C | AI Models Now Introduce Safety Risks in the Physical World | Enterprise Workflows |
| C | Data-Centric Interpretability for LLM-based Multi-Agent RLData-Centric Interpretability for LLM-based Multi-Agent RL (arXiv 2602.05183) | Long-Horizon |
| C | StarSkirmish Bench | Long-Horizon |
| C | 1830 railroad game to Finance-Agent transfer study | Long-Horizon |
| C | Cross-Benchmark Transfer from RL on Agentic Coding Tasks - arXiv | Enterprise Workflows Long-Horizon |
| C | Post-Training on Office Work Improves Software Engineering - arXiv | Enterprise Workflows Long-Horizon |
| C | Cross-Benchmark Generalization in Long-Horizon Agents - arXiv | Enterprise Workflows Long-Horizon |
| C | Robo-benchTests agents controlling a simulated robot arm and humanoid through motor commands, sensor readings, and… | Agentic |
| C | RULERSynthetic long-context benchmark of retrieval, multi-hop tracing, aggregation and question answering, scored… | Reasoning |
| C | Simulated-to-Real Transfer for Web Agents (2026-07-27): | Computer Use Enterprise Workflows |
| C | BackSearch point-in-time search for leak-free forecasting evals | Coding Long-Horizon |
| C | Galactica, Llama 2, Llama 3 | Coding Long-Horizon |
| C | OpenThoughts datasets | Long-Horizon |
| C | GEPA | Long-Horizon |
| C | Bespoke-Nimble-9B model | Long-Horizon |
| C | Bespoke Curator | Long-Horizon |
| C | Evalchemy | Long-Horizon |
| C | Bespoke-MiniCheck | Long-Horizon |
| C | Finance Agent Benchmark - arXiv:2508.00828; Finance Agent v2 | |
| C | Vals Index v2.1 and Vals Multimodal Index | |
| C | AI cheating audit | |
| C | Project Vend | Computer Use Long-Horizon |
| C | Cheating in Drone-Bench | Computer Use Long-Horizon |
| C | Multi-token Residual PredictionMulti-token Residual Prediction (arXiv 2605.18817; co-authored by Modal Research, 2026-05) | Coding |
| C | LLM Engineer's Almanac inference benchmarks | Coding |
| C | Quail open-source AI-SQL inference engine with CMU Full Stack Data Lab | Coding |
| C | FlashAttention-4 inference contributions | Coding |
| C | Trillion-token | Coding |
| C | ApexAgents SkyRL 397B RL training recipe | Coding Enterprise Workflows+1 |
| C | The Benchmark CeilingThe Benchmark Ceiling (arXiv 2607.01254) | Enterprise Workflows |
| C | COS-PLAYCOS-PLAY (arXiv 2604.20987) | Long-Horizon |
| C | Jev verifier cost study | Long-Horizon |
| C | Trinity: autoformalization system | Coding |
| C | Trinity/Gauss Lean verification environments infrastructure | Coding |
| C | MiniMuSiQue: multi-hop QA eval dataset | Coding |
| C | LiveWeb samples (with Browserbase) - 2026-09-22/28 | Computer Use Enterprise Workflows |
| C | Production-debugging RL environment (hud-ops-diagnostics) - 2026-01-20 | Computer Use Enterprise Workflows |
| C | Dojo-Bench-Mini public task set on Hugging Face | Computer Use |
| C | Pango crowdsourced computer-use trajectory datasets on Hugging Face | Computer Use |
| C | GLADOS-1 | Computer Use |
| C | PopuLoRAPopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Play (arXiv:2605.16727, 2026-05-16) | Coding Long-Horizon |
| C | ~1k JavaScript coding tasks in Harbor format | Coding Long-Horizon |
| C | SRE Bench (2026-08) and CyberBench v1.1 | |
| C | Vals Legal AI Report (VLAIR), Feb 2025 | |
| C | Gauntlet 4K RLVR dataset, access by request | Coding Computer Use+1 |
| C | OpenEnv Visual Memory and Spreadsheet environments on Hugging Face | Coding Computer Use+3 |
| C | Messier | Long-Horizon |
| C | Predicting Task Difficulty Without Rollouts | Long-Horizon |
| C | FINDFinancial decision-making in multilingual settings demands accurate numerical reasoning grounded in diverse… | Finance |
| C | MIRROR | Cybersecurity |
| C | House Rules | Long-Horizon |
| C | GAPJigsaw puzzle solving has been an increasingly popular task in the computer vision research community. Recent… | Vision |
| C | User-simulation realism: Veris vs τ²-bench | Enterprise Workflows |
| C | Printing Machines (2026-01-28) | Computer Use Enterprise Workflows |
| C | No peer-reviewed paper authored by Fleet found. Harbor | Computer Use Enterprise Workflows |
| C | Legal Research Bench | |
| C | Editorial and infrastructure posts on chakra.dev/blog | Computer Use |
| C | PROPELPROPEL: Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier (arXiv:2606.18284,… | Coding Long-Horizon |
| C | Robustly Evaluating Cyber Capabilities | Coding Long-Horizon+1 |
| C | Team research listed on the research page | |
| C | HuzzleWorld computer-use models | Coding Computer Use+3 |
| C | Searching Over Sandbox States for Coding Agents | Coding Computer Use |
| C | No Runloop-authored papers/benchmarks found. Platform supports running third-party academic coding benchmarks | Coding |
| C | SheetBench-50 (released 2025-10-01) | Enterprise Workflows Long-Horizon+1 |
| C | FinchA benchmark of real finance and accounting workflows built around spreadsheets: creating, editing and… | Agentic |
| C | Frontier FinanceAn investment-workflow benchmark covering complete tasks from screening to profitability analysis. | Agentic |
| C | CLUEWSC | Reasoning |
| C | Graphwalks BFS 128KLong-context graph traversal benchmark using breadth-first search tasks. | Reasoning |
| C | RAGTruthCorpus of retrieval-augmented responses annotated word by word for unsupported or contradicted content. | Knowledge |
| C | IFStructLiquid AI benchmark of following structured-output instructions, such as producing output that matches a… | Instruction Following |
| C | StructEval-TMeasures whether a model produces valid structured output in text formats such as JSON, YAML and CSV. | Instruction Following |
| C | Human EvaluationHuman evaluation programs for model quality, usefulness, safety, and subjective output characteristics. | Alignment Multi-domain |
| C | A Matter of TASTE | Agentic |
| C | ArchBench Traceability Link RecoveryArchitecture Traceability Link Recovery: Recover traceability links between software architecture… | Coding |
| C | ArtifactLinkerScientific artifacts such as models and datasets are foundations for research. With the rapid growth of… | Machine Learning |
| C | AutoDFTDensity functional theory (DFT) serves as the basis for computational discovery in materials science and… | Materials |
| C | Beyond PerplexityTraining-free AI text detection methods primarily rely on model log-probabilities, achieving strong… | Classification |
| C | CaptchaMind | Computer Use |
| C | Chain-of-ProcedureRecent advances in vision-language models (VLMs) have achieved impressive results on standard image-text… | Multimodal Understanding |
| C | DramaDirector: Geometry-Guided Short Drama GenerationShort dramas, with their rapid shot rhythms, dialogue-driven focus shifts, and demanding cinematographic… | Video |
| C | HELM MedQAHELM MedQA: Evaluates clinical, biomedical, medical-exam, coding, or healthcare-document reasoning. | Healthcare |
| C | MLPerf EndpointsMLCommons benchmark for API-hosted GenAI endpoints, measuring serving latency, throughput, concurrency, and… | Inference |
| C | Physics-R1We audit the multimodal-physics evaluation pipeline end-to-end and document three undetected construction… | Reasoning |
| C | Agent-as-a-Router: Agentic Model Routing for Coding Tasks | Coding |
| C | Compositional Skill Routing for LLM AgentsLLM agents increasingly rely on external skills -- reusable tool specifications -- but real-world tasks often… | Agentic |
| C | ConvReConvRe evaluates language models' understanding of converse relations through Re2Text and Text2Re tasks in… | Knowledge |
| C | DualEvalCurrent LLM evaluation relies on two complementary but often disconnected signals: static benchmarks with… | Meta |
| C | DVMapCurrent Large Language Models (LLMs) typically rely on coarse-grained national labels for pluralistic value… | Alignment |
| C | EmplifAIThis paper introduces EmplifAI, a Japanese empathetic dialogue dataset designed to support patients coping… | Healthcare |
| C | Endless TerminalsEnvironments are the bottleneck for self-improving agents. Current terminal benchmarks were built for… | Agentic |
| C | From Talking to SingingWith rapid advances in audio-visual generative models, reliable forgery detection becomes increasingly… | Multimodal Understanding |
| C | GaMMAIn this paper, we propose GaMMA, a state-of-the-art (SoTA) large multimodal model (LMM) designed to achieve… | Audio |
| C | Generate README EvalBenchmark for generating structured README files from entire GitHub repositories, evaluating long-context… | Summarization |
| C | Governance DecayModern LLM agents increasingly rely on context compaction, summarization, or eviction to keep long-running… | Safety |
| C | Grounding Agent Memory in Contextual IntentDeploying large language models in long-horizon, goal-oriented interactions remains challenging because… | Agentic |
| C | GroundShotGenerating visually consistent multi-shot videos remains an open challenge. As videos span more shots,… | Video |
| C | HelicaseLLM-based multi-agent systems have been widely adopted for knowledge retrieval and report generation,… | Agentic Business |
| C | HyperEyesExisting multimodal search agents process target entities sequentially, issuing one tool call per entity and… | Agentic |
| C | InfiniteBM CoupHead-to-head LLM game-arena ladder for Coup, using InfiniteBM's per-game Bradley-Terry Elo ratings across… | Game |
| C | InfiniteBM WerewolfHead-to-head LLM game-arena ladder for Werewolf, using InfiniteBM's per-game Bradley-Terry Elo ratings across… | Game |
| C | InteractiveBench LogicInteractive proof domain on Situation Puzzle logic instances. | Knowledge |
| C | InterGPSInterpretable Geometry Problem Solver (Inter-GPS) with Geometry3K dataset of 3,002 geometry problems with… | Spatial Reasoning |
| C | LID BenchmarkLanguage identification benchmark comparing LID models across FLORES+, MADAR, Gherbal-Multi, ATLASIA-LID,… | Language |
| C | LiveStarProDespite the remarkable progress of Video Large Language Models (Video-LLMs), current online architectures… | Uncategorized |
| C | MAAT: Multi-phase Adapter-Aware Targeted UnlearningMachine unlearning evaluation is structurally skewed: Why-type questions, which probe causal and relational… | Safety |
| C | Membership Inference on LLMs in the WildMembership Inference Attacks (MIAs) act as a crucial auditing tool for the opaque training data of Large… | Cybersecurity |
| C | MemDelta | Agentic |
| C | MindClawTheory of Mind (ToM) enables an agent to reason about another actor's beliefs, goals, and intentions, which is… | Embodied |
| C | MMFormalizer: Multimodal Autoformalization in the WildAutoformalization, which translates natural language mathematics into formal statements to enable machine… | Multimodal Understanding |
| C | Molmo2Today's strongest video-language models (VLMs) remain proprietary. The strongest open-weight models either… | Video |
| C | OrgForge-ITSynthetic insider-threat detection benchmark built from OrgForge organizational simulation telemetry with… | Cybersecurity |
| C | PanoWorldMultimodal large laboratory models (MLLMs) still struggle with spatial understanding under the dominant… | Spatial Reasoning |
| C | PerceptionDLM | Multimodal |
| C | PhotoFlow: Agentic 3D Virtual Photography MissionsVirtual photography asks an agent to enter a prepared 3D scene with no preselected camera pose or reference… | Agentic |
| C | RadLIT-9Radiology education retrieval benchmark with 420 multiple-choice questions from 36 open-access review articles… | Retrieval Healthcare |
| C | Relevance as a VulnerabilityAI agents augment large language models with external tools such as web retrieval, enabling grounded and… | Safety |
| C | Reproducing Complex Set-Compositional Information RetrievalComplex information needs may involve set-compositional queries using conjunction, disjunction, and exclusion,… | Retrieval |
| C | ResearchMath-14KThe frontier of mathematics is defined by problems whose solutions are not yet known, yet it remains unclear… | Math |
| C | Right KnowledgeLarge language models can store both outdated facts and newer superseding facts in their parameters, but… | Factuality |
| C | Seg-AgentLanguage-guided segmentation transcends the scope limitations of traditional semantic segmentation, enabling… | Vision |
| C | Send a SCOUT FirstPrompt-injection detectors are heterogeneous: each is strong on a different slice of attacks, and none is… | Cybersecurity |
| C | SenFlowSentence-level AI-generated text detection (S-AGTD) for hybrid documents, where humans and LLMs co-author one… | Classification |
| C | STReasonerSpatio-temporal reasoning in time series involves the explicit synthesis of temporal dynamics, spatial… | Spatial Reasoning |
| C | SynAE | Uncategorized |
| C | The World is Not Mono | Audio |
| C | To Call or Not to CallLLM agents exhibit a consistent tendency to over-call, invoking tools even in situations where none is needed.… | Tool Use |
| C | TSCGProduction agent frameworks (OpenAI Function Calling, Anthropic Tool Use, MCP) transmit tool schemas as JSON,… | Tool Use |
| C | When2SpeakLarge Language Models (LLMs) excel at generating contextually appropriate responses but remain poorly… | Behavioral |
| C | X-Risks Leaderboard | Safety |
| C | MIIT-datasetMulti-image content has become an increasingly prevalent form of visual communication in social media, giving… | |
| B | Global MMLU LiteKaggle reports an average across 16 language dimensions of Global MMLU Lite, with separate culturally… | Multilingual |
| D | Vals Terminal-Bench 1.0 mirrorA Vals-hosted view of Terminal-Bench 1.0 with easy, medium, and hard task splits. | Agentic |
| D | LiveCodeBench Pass@1-COTThis lane contains DeepSeek's LiveCodeBench Pass@1-COT results. The explicit metric and prompting label keeps… | Coding |
| D | Cloudflare Decision Index 0.2.1: GPQA DiamondPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: GSM8KPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: HLEPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: MMLU-ProPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: MMLUPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | SWE-bench ExtraSWE-bench Extra: Evaluates software-engineering agents on realistic issue resolution, repository navigation,… | Coding |
| C | Scale Labs (launched 2026-03-09 | Coding Computer Use+2 |
| D | EMBEvaluating agents on Excel-based financial modeling tasks | Agentic |
| D | Finance Agent v2.1Financial research with tools, revised questions and rubrics, a new judge jury, and stricter treatment of… | Agentic |
| D | EMB v1.1Excel financial modeling from a template or from scratch, with updated grading in v1.1. | Agentic |
| D | Code Migration v1.1 | Coding |
| D | vexp-swe-bench | Coding |
| D | Voyager MinecraftVoyager Minecraft: Measures embodied-agent, navigation, manipulation, or simulated robotics task success. | Agentic |
| C | PostTrain Arena | Coding Computer Use+1 |
| C | Learning from Less: RLVR in Low Data RegimesLearning from Less: RLVR in Low Data Regimes (arXiv 2604.18381; MLSys 2026 per Snorkel) | Coding Computer Use+2 |
| C | ACE (AI Consumer Index) | Coding Enterprise Workflows+1 |
| C | Universal offline sandbox escape disclosureUniversal offline sandbox escape disclosure (https://primeintellect.ai/blog/universal-offline-sandbox-escape) | Coding Enterprise Workflows+2 |
| D | Hermes BenchTests agents on 150 tasks across 25 categories, including Hermes skills, research, diagrams and art, memory,… | Agentic |
| D | VideoQADeepMind's 1H-VideoQA tests recall and reasoning over videos lasting 40 to 90 minutes. The current… | Multimodal |
| D | ProtocolsDetects and fixes errors in molecular-biology protocols using document, code, and web-search tools. | Knowledge |
| D | KnowledgeLiquid AI reports 23.8 skill points for d1-3B and 8.3 for d1-omni-600M on Knowledge with the official Decision… | Decision Models |
| D | LanguageLiquid AI reports 56.4 skill points for d1-3B and 12.9 for d1-omni-600M on Language with the official Decision… | Decision Models |
| D | RetrievalLiquid AI reports 52.8 skill points for d1-3B and 35.0 for d1-omni-600M on Retrieval with the official… | Decision Models |
| D | ToolsLiquid AI reports 74.5 skill points for d1-3B and 15.1 for d1-omni-600M on Tools with the official Decision… | Decision Models |
| D | ArtsLiquid AI reports 36.3 skill points for d1-3B and 6.8 for d1-omni-600M on Arts with the official Decision… | Decision Models |
| C | Teen AI safety study | |
| C | SOTA speculative decoding | Coding |
| C | Morph Prover v0 7B | Coding |
| B | Vals RSI Index | |
| C | computer-1 computer-use harness in Harbor | Coding Computer Use+1 |
| C | GPU Glossary | Coding |
| C | Rubric-based RL papers | Coding Computer Use+2 |
| C | Kubernetes Is the Wrong Primitive for Diverse Workloads | Coding Computer Use |
| C | Evaluations cited in 23 frontier model system cards and reports listed on the vendor homepage (Claude family through Opus/Sonnet 5.5 | |
| D | DECK-BenchMoonshot AI's internal benchmark for presentation and deck-production workflows. | Agentic |
| D | Financial ModelingMoonshot AI's internal benchmark for formula linkage, financial logic, and final workbook quality in… | Agentic |
| D | EEBenchAn electrical-engineering agent benchmark maintained by atopile. | Coding |
| D | De novo protein-binder designDesigns diverse miniprotein binders against target protein structures in a GPU-equipped research sandbox. | Knowledge |
| D | CircaCirca pairs polar questions with indirect answers and labels how each answer should be interpreted. It tests… | Decision Models |
| D | DistressBenchMeasures crisis-support replies against clinician-written rubrics. These published results are excluded from… | Cross-domain |
| D | ExplorationBench AlienCodeTests whether a system can discover a programming language's hidden rules through experiments and apply them… | Agentic |
| D | Toolathlon Verified Pass@3Fraction of Toolathlon Verified tasks solved in at least one of three trials. | Agentic |
| D | Toolathlon Verified Pass³Fraction of Toolathlon Verified tasks solved in all three independent trials. | Agentic |
| D | AA AutomationBenchAn independently evaluated automation benchmark from Artificial Analysis. | Agentic |
| D | AA EnterpriseOps-GymAn independently evaluated enterprise-operations benchmark from Artificial Analysis. | Agentic |
| D | AA Harvey LAB v1.0Archived v1.0 legal-agent results from Artificial Analysis, kept separate from v1.1. | Agentic |
| D | AA ITBenchAn independently evaluated IT-operations benchmark from Artificial Analysis. | Agentic |
| D | AA Agentic IndexA display-only Artificial Analysis agentic index. | Agentic |
| D | LifeValOpenAI's benchmark of practical everyday tasks people want a personal assistant to handle, such as finding… | Agentic |
| D | Kimi Claw 24/7A Moonshot AI internal long-horizon agent benchmark for persistent professional coworking tasks. | Agentic |
| D | Business OperationMoonshot AI's internal benchmark for long-horizon management and decision-making in a simulated business… | Agentic |
| D | Financial ResearchMoonshot AI's internal benchmark for auditable financial research under evidence and cutoff constraints. | Agentic |
| D | DECK-Arena | Agentic |
| D | Design Bench | Agentic |
| D | MLE-Bench LiteA lightweight machine-learning competition benchmark that measures whether models can iteratively train,… | Agentic |
| D | MCP-Tasks | Agentic |
| D | WideResearchA broad research-agent benchmark for open-ended information gathering, synthesis, and answer construction… | Agentic |
| D | MEWCA benchmark that evaluates AI agents on multi-environment web challenges, testing navigation and task… | Agentic |
| D | ACE solvedNumber of advanced cyber-range challenges solved in the joint NIST CAISI and UK AISI preliminary evaluation. | Agentic |
| D | The Last Ones stepsAverage step reached on a 32-step long-horizon cyber range. | Agentic |
| D | FrontierCyber | Agentic |
| D | CyScenarioBench successAverage success rate across realistic, long-horizon cybersecurity scenarios. | Agentic |
| D | Atomic vulnerability research | Agentic |
| D | Atomic evasion | Agentic |
| D | SCONE post-cutoff successShare of SCONE smart-contract vulnerabilities exploited on the 12-task post-cutoff set. | Agentic |
| D | Firefox 147 exploitsShare of patched Firefox 147 JavaScript-engine targets for which the model produced a working… | Agentic |
| D | Harvey's Legal Agent Benchmark v1Archived Vals legal-agent results before the tracked-changes grading fix. | Agentic |
| D | FACTS Search V1The archived FACTS Search V1 protocol measures factual answering with Brave Search and a Gemini 2.0 Flash… | Agentic |
| D | FrontierCode 1.1 MainCognition's 100-task software-engineering benchmark for whether coding agents produce mergeable,… | Coding |
| D | AA Coding Agents | Coding |
| D | VIBE V2A display-only MiniMax provider benchmark for end-to-end coding-agent and product-building tasks. | Coding |
| D | Biomedical image analysisQuantitative biomedical image analysis across segmentation, detection, measurement, classification,… | Multimodal |
| D | BabyVision w/ PythonA Python-assisted BabyVision evaluation for fine-grained visual perception and grounded reasoning. | Multimodal |
| D | ZeroBench w/ Python | Multimodal |
| D | MMMU-Pro w/ PythonTool-augmented MMMU-Pro variant that allows Python assistance during multimodal reasoning. | Multimodal |
| D | Liquid Extract JSON ValidityA display-only Liquid AI extraction metric measuring the share of image-to-JSON outputs that parse as strict… | Multimodal |
| D | Liquid Extract F1A display-only Liquid AI extraction metric measuring field-name agreement between requested schema fields and… | Multimodal |
| D | Liquid Extract VLM JudgeA display-only Liquid AI extraction metric measuring judged agreement between extracted values and the source… | Multimodal |
| D | MStarA general visual question-answering benchmark used in provider tables for real-image reasoning quality. | Multimodal |
| D | ChatCVQAA conversational visual QA benchmark that tests multi-turn grounded answering over images and documents. | Multimodal |
| D | AI2D_TESTA diagram understanding benchmark focused on scientific and educational visual question answering. | Multimodal |
| D | VoiceCodeBenchTests whether speech-to-text models preserve exact structured values in English workplace audio. | Multimodal |
| D | Medicinal chemistrySelects compounds with the highest or lowest measured ADME-related property from molecular structures and… | Knowledge |
| D | Organic chemistry V2Chemistry tasks covering spectroscopy, synthesis planning, reaction prediction, and chemical structure images. | Knowledge |
| D | AA-Omniscience Hallucination RateA display-only Artificial Analysis factuality metric for the rate of incorrect answers among non-correct… | Knowledge |
| D | Long-context translationCohere's long-document translation evaluation: two book chapters translated in a single call, scored per… | Multilingual |
| D | SysBenchTests how well a model follows the rules in a system message across single and multi-turn conversations. | Instruction Following |
| D | KindBenchA behavioral benchmark that tests psychological safety across sixteen adversarial multi-turn conversations… | Instruction Following |
| D | ACCR standardShare of approved advanced-cyber requests completed rather than refused under standard GPT-5.6 Sol safeguards. | Instruction Following |
| D | ACCR Daybreak RedShare of approved advanced-cyber requests completed rather than refused by GPT-5.6 Cyber under Daybreak Red… | Instruction Following |
| D | JevBenchTyped-decision evaluation across 1,624 decisions, combining chance-corrected intelligence, probability… | Decision Models |
| D | Cloudflare Decision Index 0.2.1: BPoMPPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: HumicroeditPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: POP909-CLPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: cfcolorPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ForecastBenchPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: Habermas MachinePublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: New YorkerPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ChessBenchPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: MuSRPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: SATA-BenchPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: CRUXEvalPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: CLadderPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: BBHPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ContractNLIPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ANLIPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: WinoGrandePublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: HellaSwagPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ACOSPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: FinEntityPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: iSarcasmEvalPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: VASTPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: NLI4CTPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: RAGTruthPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: BANKING77Published Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: CLINC150+OOSPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: BRIGHTPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: Amazon ESCIPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: HoVerPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: BFCLPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ToolRetPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: API-BankPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: Home appliance simulatorPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: When2CallPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: RouterBenchPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: SGD/SGD-XPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ARC-EasyPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ARC-ChallengePublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: OpenBookQAPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: CommonsenseQAPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: Support-ticket calibrationPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: Email spam classificationPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | JevBench public hard | Decision Models |
| D | Perplexity Decision PanelPerplexity reports accuracy across 11 decision tasks on a fixed 7,210-row panel. The overall percentage… | Decision Models |
| D | Knowledge and Reasoning | Decision Models |
| D | Language Understanding | Decision Models |
| D | Retrieval and Classification | Decision Models |
| D | Tools and Automation | Decision Models |
| D | Arts and Human Taste | Decision Models |
| D | Decision Index ToolsDecision Index 0.4 publishes chance-corrected tools scores for 209 evaluated systems. | Decision Models |
| D | Decision Index ArtsDecision Index 0.4 publishes chance-corrected arts scores for 209 evaluated systems. | Decision Models |
| D | AA-Omniscience-Decision accuracyAA-Omniscience-Decision accuracy is one component of the Artificial Analysis Decision Index v1.0, run by… | Decision Models |
| D | AA-Omniscience-Decision non-hallucination rateAA-Omniscience-Decision non-hallucination rate is one component of the Artificial Analysis Decision Index… | Decision Models |
| D | KMMLU-ProKorean National Professional Licensure exams evaluating professional-grade knowledge. | Korean |
| D | CLIcKEvaluates Korean culture and linguistics. | Korean |
| D | CAIS Text LeaderboardA Center for AI Safety dashboard view summarizing text capabilities across HLE and TextQuests. | Cross-domain |
| D | Cohere Embed 5 launch: ViDoRe V3 parsed-text average | Cross-domain |
| D | Cohere Embed 5 launch: ViDoRe V3 Finance | Cross-domain |
| D | Cohere Embed 5 launch: Parsed PDF full-suite average | Cross-domain |
| D | Cohere Embed 5 launch: Fused text-image average | Cross-domain |
| D | Cohere Embed 5 launch: Page-image finance average | Cross-domain |
| D | Cohere Embed 5 launch: European-language composite average | Cross-domain |
| D | Cohere Embed 5 launch: Japanese retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Korean retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Arabic retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Farsi retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Hindi retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Bengali retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Telugu retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Indonesian retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Thai retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Document throughput | Cross-domain |
| D | Cohere Embed 5 launch: Shared-space corpus/query pairings | Cross-domain |
| D | Cohere Rerank 5 launch: ViDoRe V3 average | Cross-domain |
| D | Cohere Rerank 5 launch: BEIR average | Cross-domain |
| D | Cohere Rerank 5 launch: BRIGHT average | Cross-domain |
| D | Cohere Rerank 5 launch: nanoMTEB average | Cross-domain |
| D | Cohere Rerank 5 launch: Parsed PDFs average | Cross-domain |
| D | Cohere Rerank 5 launch: Spanish retrieval | Cross-domain |
| D | Cohere Rerank 5 launch: French retrieval | Cross-domain |
| D | Cohere Rerank 5 launch: German retrieval | Cross-domain |
| D | Cohere Rerank 5 launch: Arabic retrieval | Cross-domain |
| D | Cohere Rerank 5 launch: Japanese retrieval | Cross-domain |
| D | Cohere Rerank 5 launch: Korean retrieval | Cross-domain |
| D | Cohere Rerank 5 launch: calibration error vs human graders | Cross-domain |
| D | Cohere Rerank 5 launch | Cross-domain |
| D | A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models | Truthfulness |
| D | AMC 12AMC 12: Measures mathematical reasoning, symbolic problem solving, proof construction, or competition-style… | Math |
| D | An Empirical Study of On-Device Translation for Real-Time Live-Stream Chat on Mobile DevicesDespite its efficiency, there has been little research on the practical aspects required for real-world… | Translation |
| D | Beyond Global ReplanningReal-world computer-use tasks often span multiple applications and devices, requiring agents to coordinate… | Agentic |
| C | Beyond the GUI ParadigmRecent advances in mobile agents are dominated by the GUI paradigm, in which agents perceive UI information… | Mobile |
| D | CiteCheck | Retrieval |
| C | CORTEXThe continuous evolution of large language models drives escalating demands on data scale and quality, and as… | Retrieval |
| D | Cybersecurity CTFsCybersecurity Capture the Flag (CTF) benchmark for evaluating LLMs in offensive security challenges. Contains… | Cybersecurity |
| D | EvoTrace | Coding |
| D | InferenceLatency.comLive provider-level inference latency, throughput, cost, and reliability tracker for hosted language-model… | Inference |
| C | Lumos-NexusConnector-based video unified models have demonstrated strong capability in instruction-grounded video… | Video |
| D | MIMIC-IV-Ext-BenchMIMIC-IV-Ext-Bench: Evaluates clinical, biomedical, medical-exam, coding, or healthcare-document reasoning. | Healthcare |
| D | NodeSynth: Socially Aligned Synthetic Data for AI EvaluationRecent advancements in generative AI facilitate large-scale synthetic data generation for model evaluation.… | Safety |
| D | Open X-Embodiment / RT-XCross-robot embodiment benchmark and dataset collection for evaluating vision-language-action policies across… | Robotics |
| D | RAGASRAGAS: Measures long-context retrieval, needle finding, summarization, factual grounding, or… | Generalization |
| C | Sarashina2.2-TTSWhile large language model (LLM)-based text-to-speech (TTS) systems have achieved high-quality speech… | Speech |
| D | SWE-GymSWE-Gym: Evaluates software-engineering agents on realistic issue resolution, repository navigation, testing,… | Coding |
| D | SWE-smithSWE-smith: Evaluates software-engineering agents on realistic issue resolution, repository navigation,… | Coding |
| C | TailorMindPersonalized content systems depend on available UGC and struggle when suitable content is absent, delayed, or… | Multimodal |
| D | Design Arena — Agentic Game DevelopmentCategory: multimodal. Leader: . Rows: 28. | Multimodal |
| D | Design Arena — HTML SlidesCategory: multimodal. Leader: . Rows: 29. | Multimodal |
| A | BenchLM | General Knowledge |
| D | BBO-PileMost black-box optimization methods require extensive hyperparameter tuning, often limiting their ability to… | Machine Learning |
| C | Beyond Static ToolsThe central challenge of AI for Science is not reasoning alone, but the ability to create computational… | Tool Use |
| D | Decoupling Spatio-Temporal Adapter for Fine-Grained Badminton Action LocalizationTemporal Action Localization (TAL) has been extensively studied in generic video understanding, while… | Video |
| B | FLORES European Languages LeaderboardEuropean-language subset of FLORES-200 machine translation, focusing on translation pairs involving Polish and… | Multilingual |
| B | Galileo Agent Leaderboard | Agentic Business+1 |
| A | GSMA Open Telco Leaderboard | Domain |
| D | HH-RLHFHH-RLHF: Measures model robustness, truthfulness, calibration, bias, harmfulness, jailbreak resistance, or… | Generalization |
| C | Hista and NumcaReinforcement learning (RL) refines large language models (LLMs) by directly optimizing model behavior through… | Machine Learning |
| D | HypersimHypersim evaluates 3D grounding and depth understanding in synthetic indoor scenes. | Spatial Reasoning |
| D | InferBench Text-to-Image | Multimodal |
| C | InfiniteBM ChessHead-to-head LLM game-arena ladder for chess, using InfiniteBM's per-game Bradley-Terry Elo ratings across… | Game |
| D | Integrated Safety Evaluation | Safety |
| D | L-MARSLong-form Multi-hop Agentic Reasoning benchmark for legal search, centered on LegalSearchQA tasks requiring… | Legal |
| D | Large-Scale Aspect-Based Sentiment Analysis with Reasoning-Infused LLMsWe introduce Arctic-ABSA, a collection of powerful models for real-life aspect-based sentiment analysis… | Language-understanding |
| C | LLawCoEmbodied agents operating in decentralized and partially observable environments have attracted growing… | Embodied |
| D | (long-context agent monitoring benchmark — not explicitly named in abstract)Monitoring coding agents for dangerous behavior using language models requires classifying transcripts that… | Long Context |
| C | OMLAB Open Agent Multimodal Leaderboard | Multimodal |
| A | OpenVLM Leaderboard | Multimodal Understanding |
| C | Optimum LLM Perf Leaderboard | Inference |
| D | OS-AtlasGUI grounding benchmark and dataset for evaluating foundation models on operating-system interface… | GUI |
| D | OS-CopilotOS-Copilot: Measures browser, desktop, mobile, or GUI agents operating in interactive environments. | Agentic |
| D | Pokémon Autonomous Completion | Game |
| D | Red QueenCurrent approaches to LLM adversarial testing suffer from coverage gaps: manual red-teaming does not scale,… | Safety |
| D | Sandboxed Coding Agents are Competitive Omni-modal Task SolversAs multimodal LLMs increasingly target video and audio, it is often assumed that such tasks require native… | Agentic |
| D | SeeClickGUI grounding benchmark and model suite for visual GUI agents that must map natural-language instructions to… | GUI |
| D | Self-Ensembling Vision-Language Models for Chart Data ExtractionCharts effectively convey quantitative information, but the underlying data are often locked in image form,… | Document Understanding |
| D | Semantically Aware UAV Landing Site Assessment from Remote Sensing Imagery via Multimodal Large Language ModelsSafe UAV emergency landing requires more than just identifying flat terrain; it demands understanding complex… | Aerospace |
| D | SomaliBench EvalLarge language model safety evaluation remains heavily English-centered, leaving low-resource languages… | Safety |
| D | TAAFExecution traces are a critical source of information for understanding, debugging, and optimizing complex… | Software Engineering |
| D | TabArena All Tasks | Tabular |
| D | TemporalBenchTemporalBench evaluates LLM-based agents on contextual and event-informed time-series tasks spanning multiple… | Agentic |
| D | The Heterogeneous Safety Impacts of Benign Multilingual Fine-TuningFine-tuning a large language model is a ubiquitous method for enhancing its capability on a specific… | Safety |
| C | The Misattribution GapMulti-agent AI pipelines typically assume that agent misconduct originates from model misalignment. We… | Safety |
| D | Tools as Continuous Flow for Evolving Agentic ReasoningLarge Language Models (LLMs) have demonstrated remarkable capabilities in orchestrating tools for reasoning… | Agentic |
| D | TRACERData contamination is a known threat to the reliability of model evaluation. However, it remains underexplored… | Coding |
| C | Ukrainian LLM Leaderboard | Language |
| D | Uncertainty Decomposition for Clarification Seeking in LLM AgentsRecent position papers argue that the classical aleatoric/epistemic uncertainty framework is insufficient for… | Agentic |
| D | Urban Socio-Semantic Segmentation with Vision-Language ReasoningAs hubs of human activity, urban surfaces consist of a wealth of semantic entities. Segmenting these various… | Geospatial |
| C | VisualClawVision language models are serving as general-purpose interfaces for complex multimodal tasks. However,… | Agentic |
| D | Why LLMs Fail at Causal Discovery and How Interventional Agents EscapeCausal discovery is a cornerstone of scientific reasoning, yet whether large language models can perform it… | Reasoning |
| D | WM Bench | Embodied |
| A | YALL Nous Leaderboard | Knowledge |
| C | ZONOS2 Technical ReportWe present ZONOS2 8B, our latest TTS model, which achieves state-of-the-art naturalness, prosody, and voice… | Speech |
| C | Transformer Scalability Crisis | Efficiency |
No entries match. Clear the filter above to see all 3228.
Model rankings
Dedicated view ↗| Rank | Model | Score | Creator | |
|---|---|---|---|---|
| 1 | Gemini 4 Carbon | 88.38 | ||
| 2 | Claude Opus 5.5 | 86.29 | Anthropic | |
| 3 | GPT-6 Astra | 86.07 | OpenAI | |
| 4 | GPT-6.1 Sol | 85.27 | OpenAI | |
| 5 | Gemini 4 Argon | 82.28 | ||
| 6 | Claude Sonnet 5.5 | 81.84 | Anthropic | |
| 7 | Claude Fable 5.1 | 81.08 | Anthropic | |
| 8 | Claude Fable 5 | 80.78 | Anthropic | |
| 9 | GPT-6.1 Terra | 80.46 | OpenAI | |
| 10 | Claude Opus 5 | 79.68 | Anthropic | |
| 11 | GPT-6 Sol | 77.97 | OpenAI | |
| 12 | GPT-5.6 Sol | 74.83 | OpenAI | |
| 13 | Kimi K3.1 Pro | 74.61 | Moonshot AI | |
| 14 | GPT-5.5 Pro | 72.76 | OpenAI | |
| 15 | GPT-5.6 Terra | 69.81 | OpenAI | |
| 16 | Claude Opus 4.8 | 69.63 | Anthropic | |
| 17 | Kimi K3 | 69.34 | Moonshot AI | |
| 18 | Muse Wonder Full | 68.96 | Meta | |
| 19 | GPT-5.5 | 68.82 | OpenAI | |
| 20 | GPT-5.4 Pro | 68.23 | OpenAI | |
| 21 | Claude Opus 4.7 | 68.11 | Anthropic | |
| 22 | Grok 4.6 | 66.62 | xAI | |
| 23 | Gemini 3.7 Flash | 66.53 | ||
| 24 | Claude Haiku 5.5 | 66.42 | Anthropic | |
| 25 | GPT-5.4 | 66.40 | OpenAI | |
| 26 | GPT-5.2 Pro | 66.23 | OpenAI | |
| 27 | Gemini 3.8 Flash | 66.22 | ||
| 28 | Muse Spark 1.2 | 65.41 | Meta | |
| 29 | Ling 3.1 Flash | 65.41 | InclusionAI | |
| 30 | DeepSeek V4.1 Flash 1017 | 65.22 | DeepSeek | |
| 32 | Muse Spark 1.1 | 65.16 | Meta | |
| 33 | Qwen3.8 Max | 64.86 | Alibaba | |
| 34 | Gemini 3.1 Pro | 64.41 | ||
| 35 | dots3-note Preview | 64.35 | Dots Studio | |
| 36 | Claude Sonnet 5 | 64.25 | Anthropic | |
| 37 | GPT-5.6 Luna | 63.80 | OpenAI | |
| 38 | Sage-A1.5 | 62.98 | InternScience | |
| 39 | Grok 4.5 | 62.92 | xAI | |
| 40 | Muse Wonder | 62.78 | Meta | |
| 41 | GPT-5.3 Codex | 62.75 | OpenAI | |
| 42 | DeepSeek V4 Pro 0813 | 62.53 | DeepSeek | |
| 43 | Ornith-1.5-397B | 62.47 | Ornith AI | |
| 44 | Gemini 3.6 Flash | 62.41 | ||
| 45 | Qwen3.7 Max | 62.32 | Alibaba | |
| 46 | GPT-6 Luna | 62.14 | OpenAI | |
| 47 | Claude Opus 4.6 | 61.82 | Anthropic | |
| 48 | Hy4 preview | 61.46 | Tencent | |
| 49 | Gemini 3.5 Flash | 61.40 | ||
| 50 | GLM-5.3 | 61.03 | Z.AI |
No entries match. Clear the filter above to see all 49.