AI benchmarks for LLMs and agents
Pick a benchmark below. Each row opens a page with what it tests, who built it, its domains, what it is built on, and related environments.
S 12 · A 259 · B 630 · C 1,274 · D 839 · snapshot 2026-11-05 · type to filter, click a header to sort
Benchmarks by domain
Each benchmark in the domain its own page or paper says it tests, with the vendor that built it or the lab behind it: S and A tier unless you tick more. The table below lists every benchmark.
/
Benchmarks
The S and A tier benchmarks, each placed in the domain its own page or paper says it tests, with the vendor that built it or the lab behind it. Click one for the quote.
Glow · what it leads with selling
Hold ⌘ or Ctrl and scroll to zoom · drag to rotate · shift-drag or right-drag to move
The galaxy could not load. See every vendor on the board.
How to read it
A company, in the domain its own site leads with. The logo is its own; the glow is what it mainly sells.
Also covers: a ring at a domain's edge for a company that leads with another domain but also covers this one. A faint line runs to where it leads.
Dots around it: the other things it sells. Hollow while it is still expanding into them.
Click any star for the lines from its own website that put it there.
How to read it
A benchmark, in the domain most of its tasks are in. The logo is who built it.
Also covers: a benchmark mostly about another domain that also tests this one.
Lineage: a line from a benchmark to one built on it: its next version, a subset, an extension, or an index that combines it.
Click any star for the line from its own page or paper that put it there, and to light its lineage.
| Tier | Benchmark | Domains |
|---|---|---|
| S | Humanity's Last ExamHumanity's Last Exam measures frontier LLMs' expert-level academic knowledge and reasoning with 2,500… | Coding Computer Use+2 |
| S | Terminal-Bench 2.1Terminal-Bench 2.1 measures a model's ability to navigate and complete difficult tasks in a sandboxed terminal… | Agentic |
| S | SWE-bench VerifiedSWE-bench Verified measures whether AI models can resolve real-world GitHub issues by generating patches that… | Coding |
| S | AIME 2025AIME 2025 measures olympiad-level mathematical reasoning on all 30 problems from the 2025 American… | Math |
| S | Terminal-Bench HardTerminal-Bench Hard measures AI agents' ability to complete challenging terminal tasks such as software… | Coding |
| S | GPQA DiamondGPQA Diamond measures model accuracy on the 198 hardest graduate-level, Google-proof multiple-choice questions… | Reasoning |
| S | GPQA-D | Knowledge |
| S | MMLU-ProMMLU-Pro measures language models' knowledge and reasoning through over 12,000 ten-option multiple-choice… | Knowledge |
| S | DeepSWE 1.1DeepSWE 1.1 measures models' ability to resolve real-world software engineering tasks, scored by pass rate… | Coding |
| S | ProgramBenchProgramBench measures whether AI agents can rebuild complete programs from scratch given only a compiled… | Coding |
| S | Tau2-Bench TelecomTau2-Bench Telecom measures how well conversational agents guide users through telecom technical support in a… | Agentic Business |
| A | Terminal-Bench 3.0Terminal-Bench 3.0 measures whether AI agents can complete difficult, expert-level tasks in command-line… | Coding Enterprise Workflows+1 |
| A | Terminal-BenchTerminal-Bench measures how well AI agents complete hard, realistic tasks in computer terminal / command-line… | Coding Computer Use+2 |
| A | OSWorld 2.0OSWorld 2.0 measures computer-use agents on 108 long-horizon, real-world desktop workflows across everyday and… | Coding Computer Use+2 |
| A | SWE-bench ProSWE-Bench Pro measures AI agents' ability to resolve long-horizon software engineering tasks (bug fixes and… | Multi-Domain Coding |
| A | AIMEAIME scores models on the American Invitational Mathematics Examination's integer-answer competition math… | Math |
| A | Terminal-Bench 4.0Terminal-Bench 4.0 measures the percentage of practical terminal-based tasks AI agents resolve successfully… | Agentic |
| A | OSWorldOSWorld measures how well multimodal agents complete 369 open-ended real-world computer tasks across desktop… | Agentic |
| A | SWE-benchSWE-bench measures whether language models can resolve real-world GitHub issues by generating patches to… | Coding |
| A | Agents' Last ExamAgents' Last Exam evaluates AI agents on long-horizon, economically valuable real-world professional tasks… | Coding Computer Use+2 |
| A | APEX-AgentsAPEX-Agents measures whether AI agents can complete multi-hour, cross-application professional tasks in… | Coding Enterprise Workflows+1 |
| A | APEXAPEX (AI Productivity Index) is Mercor's family of benchmarks measuring economically valuable professional… | Multi-Domain |
| A | DeepSWEDeepSWE measures frontier coding agents on 113 original, long-horizon software engineering tasks across 91… | Coding |
| A | ARC-AGI-1ARC-AGI-1 measures how efficiently AI systems learn and generalize abstract rules from a few examples on 800… | Reasoning |
| A | ARC-AGI-2ARC-AGI-2 measures fluid intelligence and abstract reasoning in AI systems through novel input-output grid… | Reasoning |
| A | Terminal-Bench-Science 0.1Terminal-Bench-Science 0.1 measures AI agents' resolution rate on 70 expert-curated, verifiable scientific… | Agentic |
| A | LiveCodeBenchLiveCodeBench measures LLM coding ability on continuously collected competitive programming problems from… | Coding |
| A | GPQAGPQA measures model accuracy on 448 extremely difficult, Google-proof graduate-level multiple-choice questions… | Knowledge |
| A | HLE-VerifiedHLE-Verified is a verified and revised version of Humanity's Last Exam that evaluates frontier language models… | Knowledge |
| A | SimpleQASimpleQA measures whether language models can accurately answer short, fact-seeking questions with single… | Knowledge |
| A | HELM GSM8KHELM GSM8K scores language models on grade school math word problems within Stanford CRFM's HELM evaluation… | Math |
| A | HELM MMLUHELM's MMLU leaderboard scores language models by exact-match accuracy on the Massive Multitask Language… | Knowledge |
| A | MCP AtlasMCP Atlas measures language models' tool-use competency on tasks run against real Model Context Protocol (MCP)… | Coding Computer Use+2 |
| A | AutomationBenchAutomationBench measures how well AI agents complete cross-application business workflows via REST APIs across… | Agentic |
| A | Finance Agent v2Finance Agent v2 measures how well AI agents perform the work of entry-level financial analysts, answering… | Agentic |
| A | CritPtCritPt measures whether LLMs can solve unpublished, research-level frontier physics challenges and checkpoint… | Reasoning |
| A | ChartographyChartography measures whether frontier models can accurately read and reason over 100 specialized professional… | Multimodal |
| A | AgentBench FCAgentBench FC evaluates LLM agents with function-calling prompts on ALFWorld, database, knowledge graph,… | Agentic |
| A | JobBenchJobBench evaluates AI agents on 130 professional workflow tasks across 35 occupations that experts identify as… | Agentic |
| A | GDPvalGDPval measures how well AI models produce real-world, economically valuable knowledge-work deliverables… | Agentic |
| A | SAGESAGE (Student Assessment with Generative Evaluation) measures how accurately AI models grade handwritten… | Multimodal |
| A | MGSMMGSM measures language models' ability to solve grade-school math word problems translated into 10… | Multilingual |
| A | INCLUDEINCLUDE measures multilingual LLM knowledge and reasoning using 197,243 multiple-choice questions from local… | Multilingual |
| A | MATHMATH measures mathematical problem solving of models on 12,500 challenging competition mathematics problems… | Math |
| A | BFCL-V4BFCL V4 measures how accurately LLMs call functions (tools), covering single-turn, multi-turn, and agentic… | Tool Use |
| A | LOCA-BenchLOCA-bench measures how reliably language agents complete tool-using tasks as their context grows to extreme,… | Long Context |
| A | E-BenchE-Bench measures LLM agents' multi-step tool use on 323 synthetic state-changing tasks across Tencent product… | Agentic |
| A | SWE-LancerSWE-Lancer measures whether frontier LLMs can complete over 1,400 real freelance software engineering tasks… | Coding |
| A | RewardBenchRewardBench evaluates reward models on prompt-chosen-rejected trios spanning chat, reasoning, and safety to… | Alignment |
| A | DRACODRACO measures deep research agents on complex, open-ended research tasks across 10 domains, grading outputs… | Agentic |
| A | AA BriefcaseAA-Briefcase measures frontier agentic capability on long-horizon knowledge work, with models completing… | Agentic |
| A | BankerToolBenchBankerToolBench measures how well AI agents complete end-to-end junior investment banking workflows, producing… | Agentic |
| A | BFCL v3BFCL v3 evaluates how accurately large language models call functions (tools), adding multi-turn and… | Agentic |
| A | Finance Agent v1.1Finance Agent v1.1 measures how well AI agents using tools like SEC search and web search perform entry-level… | Agentic |
| A | ALE-BenchALE-Bench evaluates AI systems on score-based algorithmic programming contests drawn from AtCoder Heuristic… | Agentic |
| A | BigCodeBenchBigCodeBench evaluates LLM code generation on practical, real-world programming tasks, with results tracked on… | Coding |
| A | CursorBenchCursorBench scores models on coding-agent tasks such as edits, refactors, bugfixes, codebase understanding and… | Coding |
| A | AA-LCRAA-LCR measures language models' ability to extract, reason about, and synthesize information from long… | Reasoning |
| A | CharXivCharXiv measures how well multimodal LLMs understand and reason about natural, diverse scientific charts drawn… | Multimodal |
| A | BabyVisionBabyVision measures core visual perception and reasoning abilities of multimodal LLMs independent of… | Multimodal |
| A | CorpFin v2CorpFin v2 measures how accurately models comprehend long credit agreements, including questions over very… | Knowledge |
| A | CaseLaw v2CaseLaw v2 measures how accurately language models answer questions over private, recent Canadian court case… | Knowledge |
| A | GDPval-AAGDPval-AA measures AI models' agentic performance on real-world, economically valuable knowledge-work tasks… | Agentic |
| A | Gert LabsGBENCH by Gert Labs ranks AI models by their performance in complex games, with separate scores for agentic… | Agentic |
| A | AndroidWorldAndroidWorld measures how well autonomous agents complete 116 parameterized tasks across 20 real-world Android… | Agentic |
| A | PinchBenchPinchBench measures how well LLMs perform as the model inside an OpenClaw agent, scoring successful completion… | Agentic |
| A | QwenClawBenchQwenClawBench measures how well OpenClaw agents complete 100 realistic real-user-distribution tasks across… | Agentic |
| A | Harvey's Legal Agent BenchmarkHarvey's Legal Agent Benchmark measures how well AI agents complete agentic legal work using files, scored by… | Agentic |
| A | Vending-Bench 2Vending-Bench 2 measures how well AI models can run a simulated vending machine business over a year, scored… | Agentic |
| A | Tau3-BankingTau3-Banking measures whether conversational agents can retrieve policies from a ~700-document banking… | Agentic |
| A | SWE MultilingualSWE-bench Multilingual measures how well LLM agents resolve real-world GitHub issues across 300 tasks in 42… | Coding |
| A | MLS-Bench LiteMLS-Bench Lite is the official 30-task subset of MLS-Bench, scoring whether AI systems can invent… | Coding |
| A | PaperBenchPaperBench measures AI agents' ability to replicate state-of-the-art AI research papers from scratch,… | Coding |
| A | IOIIOI measures models' ability to solve International Olympiad in Informatics competitive programming problems… | Coding |
| A | MuSRMuSR evaluates language models on multistep soft reasoning tasks posed as long natural-language narratives… | Reasoning |
| A | LisanBenchLisanBench measures LLMs' rule-following, vocabulary knowledge, planning, recall and persistence by scoring… | Reasoning |
| A | GeneBench-ProGeneBench-Pro measures whether AI agents can carry out realistic multi-stage statistical analyses of messy… | Reasoning |
| A | WildBenchWildBench evaluates LLMs on about 1,024 challenging real-user tasks, scoring responses with checklist-guided… | Reasoning |
| A | BenchCAD Vision2CodeBenchCAD Vision2Code measures whether models can generate CadQuery programs from rendered views of mechanical… | Multimodal |
| A | MMMU-ProMMMU-Pro measures multimodal models' expert-level understanding and reasoning over college-level questions… | Multimodal |
| A | OmniDocBench 1.5OmniDocBench 1.5 evaluates how well models parse diverse real-world PDF documents, scoring end-to-end parsing,… | Multimodal |
| A | Video-MMEVideo-MME measures how well multimodal large language models understand and analyze videos. | Multimodal |
| A | MathVisionMathVision (MATH-V) measures large multimodal models' mathematical reasoning on 3,040 competition math… | Multimodal |
| A | ScreenSpot ProScreenSpot-Pro measures how accurately multimodal models locate target GUI elements in high-resolution… | Multimodal |
| A | ZeroBenchZeroBench measures large multimodal models' visual reasoning on a small set of adversarially filtered image… | Multimodal |
| A | Visual-Language UnderstandingVISTA (Visual-Language Understanding) measures multimodal models' complex visual-language understanding on… | Multimodal |
| A | MortgageTaxMortgageTax measures how accurately multimodal models extract semantic fields and calculate annualized tax… | Multimodal |
| A | MathVistaMathVista measures mathematical reasoning of foundation models in visual contexts, such as figures, geometry… | Multimodal |
| A | HealthBenchHealthBench measures how well AI models respond in 5,000 realistic multi-turn, multilingual health… | Knowledge |
| A | HealthBench ProfessionalHealthBench Professional scores language models on real clinician tasks (care consult, writing and… | Knowledge |
| A | SpatialBench VerifiedAnalysis of spatial transcriptomics data across externally validated biological problems. | Knowledge |
| A | SingleCellBenchLatchBio's bioinformatics evaluations measure a model's ability to analyze spatial transcriptomics… | Knowledge |
| A | ProteinGym HardProteinGym Hard is a hard subset of the ProteinGym protein engineering benchmark, scoring how well models… | Knowledge |
| A | LABBench2LABBench2 measures AI systems' real-world capabilities in performing biology research tasks, with nearly 1,900… | Knowledge |
| A | HealthBench HardHealthBench Hard is a 1,000-example subset of HealthBench measuring model performance on difficult health… | Knowledge |
| A | MMMLUMMMLU measures multilingual general knowledge by testing models on the MMLU test set professionally translated… | Knowledge |
| A | C-EvalC-Eval measures the knowledge and reasoning of foundation models in Chinese through 13,948 multiple-choice… | Knowledge |
| A | MedCodeMedCode measures whether AI models can assign correct primary and secondary ICD-10-CM diagnosis codes from… | Knowledge |
| A | MedScribeMedScribe measures whether AI models can write accurate, compliant SOAP clinical notes from patient… | Knowledge |
| A | LegalBenchLegalBench measures large language models' legal reasoning across 162 expert-crafted tasks covering six types… | Knowledge |
| A | TaxEval v2TaxEval v2 measures models' accuracy and stepwise reasoning when answering hard tax-related questions. | Knowledge |
| A | MedQAMedQA measures how well models answer multiple-choice medical questions drawn from professional medical board… | Knowledge |
| A | MILUMILU measures LLM knowledge and language understanding across 11 Indic languages with about 80,000… | Multilingual |
| A | IFBenchIFBench measures how well language models generalize precise instruction following to 58 new, diverse,… | Instruction Following |
| A | WritingBenchWritingBench measures large language models' generative writing ability across 6 core writing domains and 100… | Instruction Following |
| A | IMO 2026IMO 2026 measures a model's ability to write rigorous proofs for the six problems of the 2026 International… | Math |
| A | RiemannBenchRiemannBench measures models on research-level mathematics problems with unique, programmatically verified… | Math |
| A | MATH-500MATH-500 scores models' accuracy on 500 competition math problems covering algebra, probability, trigonometry… | Math |
| A | Apex ShortlistApex-Shortlist is a mathematical reasoning benchmark that scores models with and without tools, reported in… | Math |
| A | USAMO 2026USAMO 2026 evaluates models on proof-based problems from the 2026 USA Mathematical Olympiad. | Math |
| A | ProofBenchProofBench measures whether models can construct Lean 4 formal proofs, accepted by the proof checker, for… | Math |
| A | AA-OmniscienceAA-Omniscience measures factual recall and knowledge calibration across 6,000 questions in 42 economically… | Factuality |
| A | AI Energy ScoreAI Energy Score measures and compares the energy efficiency of AI models across common tasks like text… | Efficiency |
| A | AlpacaEvalAlpacaEval measures how well language models follow general user instructions, using GPT-4-based… | Generalization |
| A | Berkeley Function-Calling LeaderboardBFCL measures how accurately LLMs call functions and tools, including single-turn, multi-turn, relevance… | Agentic |
| A | BFCL_v3_MultiTurnBFCL V3 MultiTurn measures how accurately large language models handle multi-turn, multi-step function (tool)… | Tool Use |
| A | BoolQBoolQ measures models' accuracy at answering yes/no questions about passages, scored by exact match on the… | Knowledge |
| A | CL-benchCL-bench measures language models' ability to learn new knowledge, rule systems, procedures and empirical laws… | Long Context |
| A | ClawEval-MMClawEval-MM is a multimodal agent benchmark, listed under Visual Agent & Coding, that scores models on… | Agentic |
| A | CodeEditorBenchCodeEditorBench measures LLMs' code editing ability across debugging, code translation, code polishing and… | Coding |
| A | NVIDIA ComputeEvalComputeEval measures how well LLMs generate correct and performant CUDA and CUDA Python code across kernels,… | Coding |
| A | Context ArenaContext Arena is a long-context leaderboard that scores models on retrieval and reasoning across increasing… | Reasoning |
| A | Creative Writing v3An LLM-judged benchmark that scores models on creative writing quality using rubric scores and Elo ratings. | Writing |
| A | CursorBench 3.2CursorBench 3.2 is a version of Cursor's agentic coding benchmark that scores models on coding tasks such as… | Coding |
| A | DAXBenchDAXBench measures how well language models write correct, efficient DAX code for Power BI… | Data Business |
| A | Design ArenaDesign Arena is a crowdsourced benchmark where users vote on model-generated designs (websites, web apps,… | Multimodal |
| A | Ego3D-BenchEgo3D-Bench evaluates models' understanding of driving scenes, including dynamic scenes, traffic participants… | Multimodal |
| A | EQ-BenchEQ-Bench measures language models' emotional and social intelligence, currently via multi-turn roleplay chats… | Generalization |
| A | FutureEvalFutureEval measures how accurately AI models and agents forecast future real-world outcomes on Metaculus… | Reasoning Knowledge |
| A | GQAGQA measures compositional visual reasoning and question answering over real-world images, scoring accuracy,… | Knowledge |
| A | Gray Swan IPI — Combined Attack SetMeasures a model's robustness to indirect prompt-injection attacks using Gray Swan's combined attack set,… | Safety Cybersecurity |
| A | HELMHELM is a holistic leaderboard that scores language models across many scenarios such as question answering,… | Knowledge |
| A | HELM AIR-BenchAIR-Bench 2024 measures language model safety refusal across 314 risk categories derived from government… | Generalization |
| A | HELM LiteHELM Lite is a leaderboard that scores language models on ten core scenarios spanning question answering,… | Knowledge |
| A | HELM SafetyHELM Safety is a Stanford CRFM leaderboard that evaluates recent language models on safety-related benchmarks… | Generalization |
| A | HieroglyphBenchHieroglyphBench measures how well vision-language models transcribe photographed ancient Egyptian hieroglyph… | Vision |
| A | JSONSchemaBenchJSONSchemaBench measures how efficiently and completely structured-output engines and LLMs generate JSON that… | Structured Output |
| A | LawBenchLawBench measures how well large language models memorize, understand and apply Chinese legal knowledge across… | Legal |
| A | Matbench DiscoveryMatbench Discovery evaluates machine-learning energy models (e.g. universal interatomic potentials) on… | Materials |
| A | MathArena ApexMathArena Apex measures whether AI models can solve a small set of extremely hard final-answer math… | Math |
| A | MedmarksMedmarks is an open medical AI benchmark suite and leaderboard that scores language models on verifiable and… | Healthcare |
| A | MixEval ChatMixEval Chat reports chat-model scores on MixEval and MixEval-Hard, ground-truth-based dynamic benchmark… | General Knowledge |
| A | MRCR-v2 128kMRCR-v2 128k measures long-context reasoning by asking models to reproduce a specific instance of repeated… | Long Context |
| A | MRCR v2 256KMRCR v2 256K measures long-context reasoning by asking models to reproduce a specific instance of assistant… | Long Context |
| A | $OneMillion-Bench$OneMillion-Bench evaluates language agents on 400 bilingual expert-curated, economically consequential… | Professional |
| A | Open Catalyst OC20OC20 scores machine learning models on predicting catalyst-adsorbate energies and relaxed structures across… | Materials |
| A | Open Catalyst OC22OC22 is a public leaderboard scoring machine learning models on predicting relaxed energies and forces for… | Materials |
| A | ProteinGymProteinGym measures how well models predict protein fitness effects of substitutions on deep mutational… | Science |
| A | ReXrankReXrank is a public leaderboard scoring models on chest X-ray interpretation, covering radiology report… | Healthcare |
| A | Rule2DRCRule2DRC measures whether LLM agents can synthesize correct, executable KLayout DRC scripts from… | Coding |
| A | RuneBenchRuneBench measures how well AI coding agents play RuneScape by writing TypeScript against an emulated game… | Agentic |
| A | ScienceQAScienceQA measures multimodal multiple-choice science question answering with explanations across natural,… | Multimodal |
| A | SEED-BenchSEED-Bench measures generative comprehension of multimodal LLMs using 19K human-annotated multiple-choice… | Knowledge |
| A | SQuAD 2.0SQuAD 2.0 measures reading comprehension: answering questions from Wikipedia passages with text spans and… | Knowledge |
| A | SWE-fficiencySWE-fficiency measures whether language model agents can optimize the runtime of real Python repositories on… | Coding |
| A | E-Bench-CodeE-Bench-Code is the code-execution extension of E-Bench, measuring how reliably LLM agents complete… | Coding |
| A | Hy-Backend 2.0Hy-Backend 2.0 is a Tencent-designed internal agentic coding benchmark that scores models on backend software… | Coding |
| A | Hy-FinmodelBench v2Hy-FinmodelBench v2 is a Tencent internal benchmark scoring models on financial-modeling tasks as part of the… | Finance |
| A | Hy-SWE Max VerifiedHy-SWE Max Verified is Tencent Hunyuan's internal 300-task agentic software-engineering benchmark, scored… | Coding |
| A | TextClass BenchmarkTextClass Benchmark continuously evaluates LLMs and transformers on social-science text classification tasks… | Classification |
| A | VCRVCR measures cognition-level visual commonsense reasoning: models answer multiple-choice questions about… | Knowledge |
| A | AA-Omniscience Net ScoreAA-Omniscience Net Score measures factual knowledge reliability as correct answers minus incorrect answers,… | Behavioral |
| A | AgentDriveAgentDrive measures LLM reasoning for autonomous driving through 300,000 simulated driving scenarios and a… | Agentic |
| A | ArxivMathArxivMath measures models' final-answer accuracy on research-level mathematics problems extracted monthly from… | Math |
| A | BBQBBQ measures demographic social bias in question answering, scoring accuracy on ambiguous and disambiguated… | Language-understanding |
| A | BC-VLBC-VL measures multimodal search and knowledge question answering, where models combine visual inputs with… | Multimodal |
| A | Beam128KBeam128K is a 128K-context long-context evaluation reported under general capabilities in Meta's Muse Glimmer… | Long Context |
| A | BenchCAD Vision2Code-1000BenchCAD Vision2Code-1000 measures how well models convert images of 3D CAD parts into CAD code, scored by… | Multimodal |
| A | BioMysteryBench Human-DifficultBioMysteryBench Human-Difficult measures model accuracy on 23 real-world bioinformatics analysis questions… | Science |
| A | BioMysteryBench Human-SolvableMeasures model accuracy on 76 real-world bioinformatics analysis tasks from BioMysteryBench that at least one… | Science |
| A | BioPipelineBench VerifiedBioPipelineBench Verified measures how well models complete biological data-analysis (bioinformatics pipeline)… | Biology |
| A | BixBenchBixBench measures the ability of LLM-based agents to perform real-world bioinformatics data analysis,… | Science |
| A | BRIDGE Medical LeaderboardBRIDGE measures how well large language models understand real-world clinical practice text across 87 tasks in… | Healthcare |
| A | ChartMuseumChartMuseum measures large vision-language models' visual reasoning on chart question answering over 1,162… | Multimodal |
| A | ChartQAProChartQAPro measures how well large vision-language models answer diverse, challenging questions about… | Multimodal |
| A | CharXiv-DCharXiv-D is the descriptive-questions subset of CharXiv, measuring how well multimodal LLMs extract basic… | Multimodal |
| A | CountQACountQA measures how accurately multimodal models answer counting questions about objects in images, as part… | Multimodal |
| A | EvalVerseEvalVerse is an expert-calibrated evaluation framework that scores generative video models on professional… | Video |
| A | FormationEvalFormationEval measures language models' knowledge of petroleum geoscience and subsurface disciplines through… | Science |
| A | Gaia2Gaia2 is an agentic benchmark scoring models on general autonomous agent task completion, reported in Meta's… | Agentic |
| A | GDPval-MMGDPval-MM is MiniMax's internal agentic evaluation based on the open-source GDPval test set, scoring… | Multimodal |
| A | Graphwalks BFS 1M F1Graphwalks BFS 1M F1 measures long-context reasoning by having models perform breadth-first-search over graphs… | Long Context |
| A | Graphwalks BFS 256k F1Graphwalks BFS 256k F1 measures long-context reasoning by having models perform breadth-first search over… | Long Context |
| A | Graphwalks Parents 1M F1Graphwalks Parents 1M F1 measures long-context reasoning by having models find parent nodes in graphs… | Long Context |
| A | Graphwalks Parents 256k F1Graphwalks Parents 256k F1 measures long-context reasoning by having models find parent nodes in graphs… | Long Context |
| A | HAKARI-BenchHAKARI-Bench measures the retrieval quality and efficiency of embedding, sparse, late-interaction, BM25 and… | Retrieval |
| A | HiPhOHiPhO is a multimodal reasoning benchmark scoring models on challenging visual reasoning involving physical… | Multimodal |
| A | Histopath-CHistopath-C measures the robustness of histopathology vision-language models and test-time adaptation methods… | Biomedicine Healthcare |
| A | HMMT 2025HMMT 2025 scores models on final-answer problems from the Harvard-MIT Mathematics Tournament, a prestigious… | Math |
| A | HotpotQAHotpotQA measures multi-hop question answering over Wikipedia paragraphs, scoring both answer accuracy and… | Knowledge |
| A | IMO 2025IMO 2025 scores models on writing proofs for the six 2025 International Mathematical Olympiad problems, graded… | Math |
| A | JP-TL-BenchJP-TL-Bench measures bidirectional Japanese-English translation quality using anchored pairwise LLM-judged… | Translation |
| A | Kernel Bench L3Kernel Bench L3 measures GPU kernel optimization ability, reporting median speedup over a PyTorch eager… | Coding |
| A | K-MetBenchK-MetBench measures expert meteorological knowledge, reasoning, Korean geo-cultural understanding and… | Weather |
| A | LABBench2 Clinical TrialsLABBench2 Clinical Trials is a subset of LABBench2, a benchmark measuring AI systems' real-world capabilities… | Biology Healthcare |
| A | LABBench2 Database QuestionsLABBench2 Database Questions (DBQA2) is a multiple-choice subset of LABBench2 that measures models' ability to… | Biology |
| A | LABBench2 FigQALABBench2 FigQA measures models' ability to answer multiple-choice questions about scientific figures in… | Biology |
| A | LABBench2 Literature QuestionsLABBench2 Literature Questions (LitQA3 subset) scores models on multiple-choice questions requiring retrieval… | Biology |
| A | LABBench2 Patent QuestionsA subset of LABBench2 that tests AI systems on patent-related questions as part of measuring real-world… | Biology |
| A | LABBench2 Reading TablesLABBench2 Reading Tables is a subset of LABBench2, a benchmark measuring AI systems' ability to perform… | Biology |
| A | LABBench2 Supplementary MaterialsLABBench2 measures AI systems' real-world capabilities in performing biology research tasks, with nearly 1,900… | Biology |
| A | LatchBio BioinformaticsLatchBio evaluations of real-world bioinformatics problem solving. SpatialBench Verified tests analysis of… | Biology |
| A | Lech Mazur WritingMeasures LLMs' creative short-story writing under constrained briefs with ten required elements, scored via… | Writing |
| A | MATH Level 5MATH Level 5 measures models' ability to solve the hardest tier of challenging competition mathematics… | Math |
| A | MCPMark-VerifiedMCPMark-Verified is a human-verified edition of MCPMark that evaluates how well models use MCP tools across… | Agentic |
| A | MEDIC BenchmarkMEDIC evaluates LLMs across five dimensions of clinical competence, including medical QA, clinical… | Healthcare |
| A | MMBCMMBC is a multimodal search-augmented benchmark testing whether models can combine visual inputs with external… | Multimodal |
| A | MMSearchPlusMMSearchPlus measures multimodal models' ability to answer visual questions by combining image inputs with… | Multimodal |
| A | MOODMOOD measures how well LLM safety monitors detect out-of-distribution alignment failures such as jailbreaking,… | Alignment |
| A | MTEB-BRMTEB-BR measures text embedding model quality on 22 native Brazilian-Portuguese tasks across classification,… | Language |
| A | MultiLexNorm++MultiLexNorm++ measures how well models perform lexical normalization of informal social media text, extending… | Multilingual |
| A | NarrativeQANarrativeQA measures reading comprehension and question answering over long narrative texts such as stories,… | Generalization |
| A | Natural QuestionsNatural Questions measures how well language models answer real open-domain questions, scored by F1 in… | Knowledge |
| A | NeoEvalPlusNNeoEvalPlusN scores LLMs on proprietary tests of command following under distraction, expectation-breaking… | Creative |
| A | NYT Connections ExtendedMeasures LLMs' verbal associative reasoning by scoring how many groups they correctly identify in 940 NYT… | Knowledge |
| A | ObviousBenchObviousBench measures whether language models avoid simple, visible mistakes (arithmetic, counting, spelling… | Knowledge |
| A | Open Medical-LLM LeaderboardA leaderboard that scores LLMs by accuracy on medical question-answering datasets including MedQA, MedMCQA,… | Healthcare |
| A | MLE-benchMLE-bench measures how well AI agents perform machine learning engineering on 75 Kaggle competitions, such as… | Self Improvement |
| A | OpenAI MRCR v2 8-needle 128K-256KOpenAI MRCR v2 8-needle 128K-256K measures a model's ability to retrieve and distinguish multiple similar… | Long Context |
| A | OpenAI MRCR v2 8-needle 16K-32KMeasures long-context multi-round coreference retrieval, requiring models to locate eight needles within… | Long Context |
| A | OpenAI MRCR v2 8-needle 256K-512KOpenAI MRCR v2 8-needle 256K-512K measures a model's ability to retrieve and distinguish eight similar needles… | Long Context |
| A | OpenAI MRCR v2 8-needle 32K-64KOpenAI MRCR v2 8-needle 32K-64K measures a model's ability to retrieve multiple needles from long contexts of… | Long Context |
| A | OpenAI MRCR v2 8-needle 4K-8KOpenAI MRCR v2 8-needle 4K-8K measures a model's ability to retrieve and distinguish multiple (8) similar… | Long Context |
| A | OpenAI MRCR v2 8-needle 512K-1MOpenAI MRCR v2 8-needle 512K-1M measures a model's ability to retrieve eight needles from contexts of 512K to… | Long Context |
| A | OpenAI MRCR v2 8-needle 64K-128KOpenAI MRCR v2 8-needle 64K-128K measures a model's ability to retrieve and distinguish multiple similar items… | Long Context |
| A | OpenAI MRCR v2 8-needle 8K-16KOpenAI MRCR v2 8-needle 8K-16K measures a model's ability to retrieve multiple (8) needles across… | Long Context |
| A | Organic chemistryAn Anthropic internal evaluation of fundamental organic chemistry skills such as structure elucidation from… | Chemistry |
| A | OSWorld2.0OSWorld 2.0 measures computer-use agents on 108 long-horizon, real-world desktop workflows across everyday and… | Computer Use |
| A | ProfBenchProfBench scores models on professional work tasks, with a search subset reported for agentic knowledge-work… | Agentic |
| A | Protocol troubleshootingMeasures a model's ability to detect and fix errors in molecular biology protocols, including using web search… | Biology |
| A | Real-World Finance v2Real-World Finance v2 measures a model's ability to complete complex, long-horizon financial-analysis tasks… | Finance |
| A | SearchAgent LeaderboardA standardized leaderboard scoring search-augmented question-answering agents by average exact match across… | Agentic |
| A | SURDSSURDS is an autonomous-driving evaluation measuring multimodal models' understanding of driving scenes,… | Multimodal |
| A | SVFSearchSVFSearch measures how well multimodal models and agents answer knowledge-intensive four-choice questions… | Multimodal |
| A | ThemisThemis-CodeRewardBench evaluates code reward models across five preference criteria and eight programming… | Coding |
| A | TVBenchTVBench is a video understanding benchmark that scores models on reasoning over events, actions and temporal… | Multimodal |
| A | UniGenBenchUniGenBench++ measures how accurately text-to-image models' generated images reflect the semantics of their… | Multimodal |
| A | UniGenBench English LongUniGenBench English Long measures how accurately text-to-image models reflect the semantics of long-form… | Multimodal |
| A | Vals.ai Financial Agent 1.1Vals.ai Financial Agent 1.1 scores models as financial agents, reported with and without web search. | Agentic |
| A | Vibe Code Bench v1.1Vibe Code Bench measures whether AI models can build complete working web applications from scratch, evaluated… | Coding |
| A | VBench Video Generation LeaderboardVBench scores video generative models (text-to-video and image-to-video) across 16 fine-grained dimensions of… | Creative Media Video Generation |
| A | Vectara HHEM Hallucination LeaderboardMeasures how often LLMs hallucinate information not stated in a source document when summarizing it, scored by… | Behavioral |
| A | VideoMME w/o sub.Video-MME measures how well multi-modal LLMs understand and answer questions about videos of varied types and… | Multimodal |
| A | VideoMME w sub.Video-MME evaluates multi-modal LLMs on video understanding across short to hour-long videos, using frames,… | Multimodal |
| A | ViroBenchViroBench evaluates nucleotide foundation models on viral genomics tasks, covering biological understanding… | Biology |
| A | VisFactorVisFactor is a visual reasoning benchmark that scores multimodal models on visual perception, spatial… | Multimodal |
| A | VLADBenchVLADBench evaluates multimodal models on autonomous-driving scene understanding, including dynamic scenes,… | Multimodal |
| A | VNTL LeaderboardVNTL Leaderboard ranks LLMs and translation tools on translating Japanese visual novel lines into English,… | Translation |
| A | WorldVQAWorldVQA measures multimodal models' ability to answer knowledge-seeking visual questions, combining visual… | Multimodal |
| A | YapBenchYapBench measures how much assistant LLMs over-generate (excess response length beyond a minimal baseline) on… | Efficiency |
| A | ZebraLogicZebraLogic measures LLM logical reasoning on programmatically generated logic grid (zebra) puzzles, scoring… | Knowledge |
| B | Senior SWE-benchSenior SWE-bench evaluates coding agents on senior-level software engineering work: under-specified feature… | Coding Computer Use+2 |
| B | ARC-AGI-3ARC-AGI-3 measures whether AI agents can explore novel interactive game environments, acquire goals, build… | Reasoning |
| B | FinanceQAFinanceQA evaluates LLMs on complex numerical financial analysis tasks mirroring real-world investment work,… | Coding Computer Use+1 |
| B | IDE-BenchIDE-Bench evaluates LLMs acting as IDE agents on 80 real-world software engineering tasks across eight… | Coding Computer Use+1 |
| B | Continual Learning BenchContinual Learning Bench measures whether LLM-based agent systems genuinely improve through sequential… | Coding Computer Use+2 |
| B | APEX-Agents 1.1APEX-Agents 1.1 measures how well AI agents perform long-horizon, multi-step professional knowledge-work tasks… | Coding Enterprise Workflows+1 |
| B | APEX-SWEAPEX-SWE measures whether frontier AI models can perform real-world software engineering work, including… | Coding Enterprise Workflows+1 |
| B | APEX-AccountingAPEX-Accounting measures AI agents' ability to complete long-horizon professional accounting tasks across… | Coding Enterprise Workflows+1 |
| B | CodeforcesCodeforces evaluates models on competition-level programming problems from the Codeforces platform, reporting… | Coding |
| B | FinanceBenchFinanceBench measures how well LLMs answer open-book financial questions about publicly traded companies using… | Multi-Domain Coding |
| B | TRAILTRAIL measures how well LLMs can debug and localize errors in long agentic workflow traces, using 148… | Multi-Domain Coding |
| B | SWEMeasures whether AI agents can deploy, debug, migrate, and maintain live production software systems across… | ML Science |
| B | CoreCraftCoreCraft measures whether AI agents can perform multi-step, domain-specific customer support work in a… | Multi-Domain |
| B | MMLU-ProXMMLU-ProX measures LLMs' advanced knowledge and cross-linguistic reasoning using 11,829 identical… | Multilingual |
| B | GSM8KGSM8K measures language models' multi-step mathematical reasoning on 8.5K grade school math word problems. | Math |
| B | SWE-bench LiteSWE-bench Lite measures whether language models can resolve real GitHub issues by editing Python codebases,… | Coding |
| B | Terminal-Bench-ScienceTerminal-Bench-Science measures how well AI agents complete research workflows across scientific domains… | Agentic Science+1 |
| B | Code MigrationCode Migration measures how well models can reimplement existing programs in another language (CLI tools… | Coding |
| B | AIME 2026AIME 2026 scores models on the 30 final-answer math problems from the 2026 American Invitational Mathematics… | Math |
| B | BrowseComp-zhBrowseComp-ZH measures how well LLM agents browse the Chinese web to answer 289 hard multi-hop questions with… | Search |
| B | FrontierMath 2025-02-28 PrivateFrontierMath 2025-02-28 Private measures model performance on research-level mathematics problems at the… | Math |
| B | FrontierMath Tier 4 2025-07-01 PrivateMeasures model accuracy on the hardest, research-level Tier 4 FrontierMath mathematics problems from a private… | Math |
| B | Long-Horizon Terminal-BenchLong-Horizon Terminal-Bench measures how well LLM agents sustain progress over hundreds of steps on 46… | Agentic |
| B | OTIS Mock AIME 2024-2025Measures models on competition-level math problems from the OTIS Mock AIME, evaluating olympiad-level… | Math |
| B | Video SimpleQAVideo SimpleQA measures factual grounding of large video language models using short-form, multi-hop,… | Multimodal |
| B | Remote Labor IndexRemote Labor Index measures how well AI agents complete real-world, economically valuable freelance remote… | Coding Computer Use+2 |
| B | HiL-BenchHiL-Bench measures agents' help-seeking judgment (selective escalation): detecting missing, ambiguous or… | Coding Computer Use+2 |
| B | CEO BenchCEO Bench measures how well models perform executive and departmental decision work inside a simulated… | Coding Enterprise Workflows+1 |
| B | VLMBenchVLMBench measures vision-language models on open-ended, image-based real-world business and STEM tasks… | Coding |
| B | CWE-bench | Security Enterprise Workflows |
| B | WebBenchWebBench measures how effectively browser agents complete realistic read and write web workflows across 2,454… | Finance Enterprise Workflows |
| B | SpreadsheetBench 2SpreadsheetBench 2 measures how well LLM agents complete end-to-end business spreadsheet workflows… | Coding Computer Use+1 |
| B | SkillsBenchSkillsBench measures whether curated Agent Skills (procedural knowledge packages) improve LLM agent pass rates… | Coding Computer Use+1 |
| B | Realm: Pathology-report reasoning benchmark | Enterprise Workflows |
| B | GDP.pdfGDP.pdf measures whether models can parse, understand, and synthesize complex real-world professional PDFs… | Agentic |
| B | ToolathlonToolathlon measures how well language agents complete realistic, long-horizon, multi-app tasks using 604 tools… | Agentic |
| B | OfficeQA ProOfficeQA Pro measures how well models and agents answer hard grounded-reasoning questions by finding and… | Multimodal |
| B | FrontierSWEFrontierSWE measures how well coding agents handle ultra-long-horizon, open-ended software engineering tasks… | Coding |
| B | UI-BenchUI-Bench measures the visual design quality of websites and apps generated by AI text-to-app tools through… | Coding Computer Use+1 |
| B | Market-BenchMarket-Bench evaluates LLMs on introductory quantitative trading by having them write executable backtesters… | Coding Computer Use+1 |
| B | YC-BenchYC-Bench measures whether LLM agents can maintain long-term planning and consistent execution by running a… | Coding Computer Use+2 |
| B | VERITASVeriTaS is a dynamic, quarterly-updated benchmark that scores automated fact-checking systems on real-world… | Coding Computer Use+2 |
| B | Vending-BenchVending-Bench measures an LLM agent's long-term coherence by having it run a simulated vending machine… | Computer Use Long-Horizon |
| B | Crosby-micro1 RedlineBenchCrosby-micro1 RedlineBench measures how well models act as negotiation participants in multi-turn SaaS MSA… | Enterprise Workflows |
| B | RealmRealm is micro1's benchmark series evaluating multi-step domain reasoning, notably long-horizon legal… | Multi-Domain |
| B | AppBenchApp-Bench measures how much required functionality AI coding assistants and web-app builders deliver when… | Multi-Domain |
| B | Cua-BenchCua-Bench measures AI agents' computer-use mastery on desktop and mobile tasks, including expert KiCad… | Computer Use |
| B | Code IndexCode Index is a live coding leaderboard scoring models and agents on contamination-free private task sets… | Coding |
| B | BinaryAuditBinaryAudit measures AI agents' ability to find backdoors and timebombs hidden in stripped compiled binaries… | Security |
| B | unix-ctfunix-ctf procedurally generates capture-the-flag tasks in Linux containers to train and evaluate Unix… | ML |
| B | AutomationBench-AAAutomationBench-AA measures how well AI agents complete cross-application business workflow tasks via REST… | Agentic Business |
| B | τ-benchτ-bench measures how reliably language agents use domain-specific API tools and follow policy guidelines in… | Agentic |
| B | BOOMBOOM measures how well time-series forecasting models predict real-world observability and production… | Data |
| B | CHOICECHOICE measures the remote sensing perception and reasoning capabilities of vision-language models on 10,507… | Geospatial |
| B | CLEARCLEAR is an evaluation framework that measures how answer-option count, abstention options and semantic… | Biomedicine Healthcare |
| B | DoneStandard embodied evaluations do not independently score whether an agent correctly commits to task completion… | Embodied |
| B | FACTS GroundingFACTS Grounding measures how accurately LLMs generate long-form responses fully grounded in a provided… | Knowledge |
| B | ForgeForge (OPT-BENCH) evaluates LLMs on NP-hard optimization problems, scoring both solution feasibility (Success… | Reasoning |
| B | HEARHEAR evaluates general-purpose audio embedding models on nineteen downstream tasks spanning speech,… | Audio |
| B | PostTrainBenchPostTrainBench measures how well LLM agents can autonomously post-train small base language models under a… | Agentic |
| B | GBA EvalGBA Eval measures whether frontier AI coding agents can write a complete Game Boy Advance emulator in… | Coding |
| B | LeapLEAPBench measures the in-context learning efficiency of LLMs on iterative scientific design optimization… | Multi-Domain Tool Use |
| B | AttuneAttuneBench measures LLM emotional intelligence by having models predict real participants' self-labeled… | Multi-Domain Tool Use |
| B | AutoCAD-BenchAutoCAD Bench measures whether multimodal agents can produce correct, editable 2D drawings and 3D models in… | Computer Use Tool Use |
| B | Toloka ArenaToloka Arena measures agentic intelligence via multi-turn tool-use tasks in simulated customer service and… | Multi-Domain |
| B | Long-Horizon PlanningLong-Horizon Planning measures an agent's long-horizon planning by its trading performance (PnL, Sharpe,… | Long-Horizon ML |
| B | CAD ArenaCAD Arena measures how well AI agents reconstruct real manufactured parts from engineering drawings as… | Chip Design |
| B | TB3 resultsTerminal-Bench 3.0 (TB3) measures how well AI agents complete hard coding tasks in terminal environments,… | Coding Long-Horizon |
| B | Math RL TasksMath RL Tasks scores models on 20 constructive mathematics problems where an executable verifier checks a… | Math Science |
| B | DABstepDABstep measures how well LLM agents perform multi-step data analysis, answering ~450 questions that require… | Agentic Business |
| B | t2-benchτ2-bench evaluates conversational AI agents in a dual-control environment (e.g., telecom technical support)… | Agentic |
| B | TRACETRACE evaluates conversational tourism recommender systems on multi-turn dialogues for recommending the right… | Tool Use |
| B | TRACKTRACK measures how well LLMs propagate updated knowledge that conflicts with their parametric knowledge… | Reasoning |
| B | TRAPTRAP measures the trade-off between agents' task accuracy when using private information in tool calls and… | Agentic |
| B | VeRAVeRA converts reasoning benchmark problems (e.g., AIME, BeyondAIME, AMO-Bench math problems) into executable… | Reasoning |
| B | TRACESTRACES measures whether LLMs recognize and reject flawed premises from retracted, fraudulent, and… | |
| B | CyberGymCyberGym measures AI agents' real-world cybersecurity capabilities by having them generate proof-of-concept… | Agentic |
| B | DeepSearchQADeepSearchQA measures how well deep research agents execute multi-step web search plans to produce exhaustive,… | Agentic |
| B | Claw-EvalClaw-Eval measures how well autonomous LLM agents complete real-world multi-step tasks across service… | Agentic |
| B | DeepPlanningDeepPlanning measures whether agentic LLMs can do long-horizon planning, such as multi-day travel and… | Agentic |
| B | CyberBenchCyberBench measures whether AI agents can craft proof-of-concept inputs that trigger real OSS-Fuzz… | Agentic |
| B | EnigmaEvalEnigmaEval measures LLMs' creative, multi-step reasoning and multimodal processing on complex puzzle-hunt… | Reasoning |
| B | DynaMathDynaMath measures the robustness of vision-language models' mathematical reasoning by evaluating them on… | Multimodal |
| B | CC-OCRCC-OCR measures the OCR and literacy capabilities of large multimodal models across multi-scene text reading,… | Multimodal |
| B | Blueprint-Bench 2Blueprint-Bench 2 measures AI agents' spatial reasoning by having them convert ~20 apartment photographs per… | Multimodal |
| B | AudioMC – Text OutputMeasures conversational intelligence of spoken dialogue and audio-language models in multi-turn human-speech… | Multimodal |
| B | DocVQADocVQA measures how well models answer questions asked about document images, testing document understanding. | Multimodal |
| B | CMMLUCMMLU measures the knowledge and reasoning of large language models in Chinese across many academic subjects,… | Knowledge |
| B | CRUXEvalCRUXEval measures code reasoning, understanding and execution by asking models to predict inputs and outputs… | Decision Models |
| B | APEX-Agents-AAAPEX-Agents-AA is Artificial Analysis' implementation of APEX-Agents, scoring AI agents by pass@1 on… | Agentic |
| B | SWE-Atlas RefactoringSWE Atlas - Refactoring measures a coding agent's ability to restructure production code across real… | Agentic |
| B | ResearchClawBenchResearchClawBench measures whether AI agents can autonomously conduct end-to-end scientific research from… | Agentic |
| B | SWE-RebenchSWE-rebench measures how well LLM-based software engineering agents resolve fresh, decontaminated real-world… | Coding |
| B | Vibe Code BenchVibe Code Bench measures whether AI models can build complete, deployable web applications from natural… | Coding |
| B | SWE-Atlas Codebase QnASWE-Atlas Codebase QnA measures AI coding agents' deep code comprehension by having them explore, run and… | Coding |
| B | NL2RepoNL2Repo-Bench measures whether coding agents can generate a complete, installable Python library from a single… | Coding |
| B | React Native EvalsReact Native Evals measures the success rate, token usage and cost of AI coding agents on React Native code… | Coding |
| B | ReactBenchReactBench measures how well coding agents implement and fix realistic React tasks, checking correctness plus… | Coding |
| B | KernelBench HardKernelBench Hard measures how well frontier coding agents write and optimize GPU kernels (CUDA) for… | Coding |
| B | GameDevBenchGameDevBench measures how well coding agents complete 333 multimodal game-development tasks in the Godot… | Coding |
| B | SWE Atlas – Test WritingSWE Atlas – Test Writing measures how well AI coding agents can autonomously explore real production… | Coding |
| B | WeirdMLWeirdML scores language models on unusual machine learning tasks that test their adaptability. | Coding |
| B | BBHBBH (BIG-Bench Hard) measures language model performance on 23 challenging BIG-Bench tasks, many requiring… | Reasoning |
| B | HellaSwagHellaSwag measures commonsense natural language inference by asking models to select the most plausible… | Reasoning |
| B | WinoGrandeWinoGrande measures commonsense reasoning through 44k adversarially filtered Winograd-style pronoun resolution… | Reasoning |
| B | LongBench v2LongBench v2 measures LLMs' deep understanding and reasoning over long contexts (8k to 2M words) via 503… | Reasoning |
| B | OCRBench V2OCRBench v2 measures large multimodal models' OCR abilities, including text recognition, localization,… | Multimodal |
| B | OmniDocBenchOmniDocBench measures how well models parse diverse real-world PDF documents, scoring end-to-end parsing,… | Multimodal |
| B | PerceptionBenchPerceptionBench measures atomic visual perception in multimodal LLMs across ten capabilities such as counting,… | Multimodal |
| B | RealWorldQARealWorldQA measures basic real-world spatial understanding of multimodal models using over 700 real-world… | Multimodal |
| B | MMLongBench-DocMMLongBench-Doc measures how well large vision-language models understand and answer questions about long,… | Multimodal |
| B | ERQAERQA measures multimodal models' embodied reasoning via 400 multiple-choice visual questions on spatial… | Multimodal |
| B | MLVUMLVU evaluates multimodal large language models on long video understanding across diverse video genres,… | Multimodal |
| B | LVBenchLVBench measures how well multimodal models understand and extract information from extremely long videos… | Multimodal |
| B | MMVUMMVU measures expert-level, knowledge-intensive video understanding of multimodal foundation models across 27… | Multimodal |
| B | TIR-BenchTIR-Bench evaluates multimodal models' agentic thinking-with-images reasoning across 13 tasks requiring tool… | Multimodal |
| B | MedXpertQAMedXpertQA measures expert-level medical knowledge and reasoning with 4,460 text and multimodal exam questions… | Multimodal |
| B | SuperGPQASuperGPQA measures LLMs' graduate-level knowledge and reasoning with 26,529 multiple-choice questions across… | Knowledge |
| B | OpenBookQAOpenBookQA measures whether models can answer elementary-level science multiple-choice questions by combining… | Knowledge |
| B | MultiNRCMultiNRC measures LLM reasoning on native, culturally grounded questions in French, Spanish and Chinese across… | Knowledge |
| B | PRBench FinancePRBench Finance measures LLMs' high-stakes professional reasoning on expert-written finance tasks, scored… | Knowledge |
| B | PRBench LegalPRBench Legal measures LLMs on high-stakes legal professional reasoning tasks written by lawyers, scored… | Knowledge |
| B | SciPredictSciPredict measures whether LLMs can forecast the outcomes of real, recently published experiments in physics,… | Knowledge |
| B | TutorBenchTutorBench measures how well LLMs perform core tutoring tasks—adaptive explanations, feedback on student work,… | Knowledge |
| B | Gray Swan IPIMeasures how often indirect prompt-injection attacks, hidden in content like web pages, emails or documents,… | Instruction Following |
| B | IFEvalIFEval measures how well large language models follow verifiable natural language instructions, such as… | Instruction Following |
| B | IMOAnswerBenchIMO-AnswerBench tests models on 400 diverse International Mathematical Olympiad-level problems with verifiable… | Math |
| B | LiveBenchLiveBench is a frequently updated, contamination-limited LLM benchmark with objectively scored tasks across… | Cross-domain |
| B | AdvBenchAdvBench measures whether models follow harmful instructions, scoring their tendency to produce harmful… | Generalization |
| B | Age of LLMAge of LLM measures LLMs' strategic reasoning, diplomacy and action reliability by pitting two models against… | Game |
| B | AgentDojoAgentDojo evaluates the robustness of tool-using LLM agents against prompt injection attacks and the… | Generalization |
| B | AgentFloorAgentFloor is a deterministic 30-task benchmark organized as a six-tier capability ladder measuring how well… | Agentic |
| B | AgentHarmAgentHarm measures whether LLM agents refuse explicitly malicious multi-step agentic tasks and how well… | Generalization |
| B | Aider PolyglotAider Polyglot measures how well LLMs edit code to solve 225 hard Exercism exercises across C++, Go, Java,… | Coding |
| B | AMA-BenchAMA-Bench evaluates long-horizon memory of LLM agents over agent-environment interaction trajectories using… | Agentic |
| B | Android BenchAndroid Bench measures how well LLMs and coding agents resolve real Android development tasks, including… | Coding |
| B | LifeSci DiamondLifeSci Diamond measures how well AI systems produce free-text regulatory affairs deliverables for drugs,… | Biomedicine Business+1 |
| B | Arena AI Image-to-VideoA crowdsourced pairwise human-preference leaderboard ranking image-to-video generation models by votes on… | Multimodal |
| B | Arena-HardArena-Hard-Auto scores instruction-tuned LLMs on hard real-world user queries and creative writing prompts… | Generalization |
| B | Arena-Hard v2Arena-Hard v2 measures instruction-tuned LLMs on 500 challenging real-world user queries plus 250 creative… | General Knowledge |
| B | ASD-BenchASD-Bench evaluates ML, deep learning and foundation models for autism spectrum disorder screening from AQ-10… | Biomedicine Healthcare |
| B | ATLASATLAS measures long-context language model capabilities across eight dimensions with length-aware scoring over… | Long Context |
| B | Auto-CompAuto-Comp is an automated synthetic pipeline and set of benchmarks probing compositional reasoning (color… | Multimodal Understanding |
| B | AutoBenchAutoBench is a dynamic LLM benchmark that generates virtual agentic business environments with multiple LLMs… | Agentic |
| B | BabyAIBabyAI is a gridworld platform with levels that test how sample-efficiently agents learn to understand and… | Agentic |
| B | BacktestBenchBacktestBench measures how well LLMs automate quantitative trading strategy backtesting across metrics… | Finance |
| B | BalrogBALROG measures the agentic reasoning and decision-making capabilities of LLMs and VLMs across a diverse set… | Knowledge |
| B | BehaviorBenchBehaviorBench evaluates foundation models on behavioral science tasks (behavior prediction and simulation,… | Behavioral |
| B | BenchCADBenchCAD measures how well multimodal models understand and generate executable parametric CadQuery programs… | Coding |
| B | BIG-benchBIG-bench is a collaborative benchmark of more than 200 tasks probing large language models' broad… | Knowledge |
| B | BIRD-SQLBIRD-SQL evaluates text-to-SQL systems on converting natural language questions into SQL over large… | Data |
| B | BizFinBenchBizFinBench measures how well LLMs handle real-world, business-driven financial tasks such as anomalous event… | Finance Business |
| B | BLINKGBLINKG measures how well LLMs map heterogeneous data source schemas to ontology terms to construct knowledge… | Data |
| B | C3-Bench: A Context-Aware Change Captioning BenchmarkC3-Bench measures how well change captioning models and large multimodal models describe changes between image… | Vision |
| B | CADGenBenchCADGenBench measures how well AI systems generate and edit valid, geometrically correct 3D mechanical part… | Multimodal |
| B | CanLegalRAGBenchCanLegalRAGBench measures retrieval-augmented generation systems on Canadian legal question answering, using… | Legal |
| B | CardioLensCardioLens evaluates multimodal LLMs on multi-sequence cardiac MRI interpretation, covering image… | Biomedicine Healthcare |
| B | CFLUECFLUE evaluates large language models on Chinese financial language understanding through knowledge assessment… | Finance |
| B | ChartographerChartographer evaluates vision-language models' visual reasoning on chart question answering using… | Reasoning |
| B | CheckMIABenchCheckMIABench evaluates membership inference attacks against language models, using intermediate checkpoints… | Safety |
| B | ChemBenchChemBench evaluates the chemical knowledge, reasoning, and intuition of language and multimodal models against… | Science |
| B | CHI-BenchCHI-Bench evaluates long-horizon, policy-rich U.S. healthcare workflow agents across provider prior… | Agentic Healthcare |
| B | Chinese Labor Law Large Language Model BenchmarkLabourLawBench measures how well LLMs handle Chinese labor-law tasks such as provision citation, legal QA,… | Legal |
| B | CiteVQACiteVQA measures whether multimodal LLMs answer questions about long real-world PDFs correctly while citing… | Document Understanding |
| B | ClaimDBClaimDB measures how well LLMs verify factual claims whose evidence comes from compositions of millions of… | Factuality |
| B | ClassEvalClassEval measures how well LLMs generate class-level Python code, scored by class- and method-level test pass… | Coding |
| B | ClawBenchClawBench measures whether AI browser agents can complete everyday online tasks such as purchases, bookings… | Agentic |
| B | ClawProBenchClawProBench measures how models perform as OpenClaw agents when they must reason, plan, use tools, and stay… | Agentic |
| B | CLUBenchCLUBench evaluates 24 clustering algorithms, spanning conventional, deep learning-based and foundation… | Machine Learning |
| B | CommunityBenchCommunityBench evaluates how well large language models model community-specific preferences and values,… | Alignment |
| B | COMPL-AICOMPL-AI is an open-source benchmarking suite that scores LLMs against technical requirements derived from the… | Safety |
| B | ComplexMCPComplexMCP measures how well LLM agents complete interdependent commercial software automation tasks using… | Tool Use Business |
| B | ConStory-BenchConStory-Bench measures narrative consistency errors (contradictions in facts, characters, timeline, world… | Long Context |
| B | ContextEchoContextEcho measures persona drift in frontier language models over long, tool-using agentic-coding sessions… | Agentic |
| B | Continual Model Routing in Evolving Model HubsCMRBench measures how well routing methods select the right pre-trained model from a continually expanding hub… | Machine Learning |
| B | ContractBenchContractBench measures whether tool-augmented LLM agents preserve observation contracts, i.e., the temporal… | Agentic |
| B | ControBenchControBench measures how well graph neural networks, pretrained language models and LLMs predict users'… | Domain |
| B | Convex Coding Evals — No GuidelinesMeasures whether language models can write working Convex backends and client integrations without… | Coding |
| B | CorpBench Work 1.3CorpBench Work 1.3 evaluates AI agents on 100 deterministic business workflows across finance, marketing,… | Agentic Business |
| B | CTFBenchCTFBench evaluates AI-powered smart contract auditors on test contracts each containing one vulnerability,… | Generalization |
| B | CursorBench 3.1CursorBench 3.1 measures coding agents on ambiguous, multi-file tasks sourced from real Cursor sessions,… | Coding |
| B | CV-ArenaCV-Arena evaluates instruction-guided image editing on 12K high-resolution real-image instruction pairs across… | Vision |
| B | Data Agent BenchmarkUC Berkeley EPIC Data Lab benchmark for data agents answering complex real-world data tasks across 12… | Agentic |
| B | DataComp-VLMDataComp-VLM is a data-curation benchmark where participants curate training data for vision-language models,… | Multimodal |
| B | Deep20BenchDeep20Bench measures how many yes/no-style questions language models need to identify a hidden subject in a… | Game Reasoning+1 |
| B | DeepResearchEvalDeepResearchEval scores deep research systems on persona-driven, multi-source web research tasks, measuring… | Agentic |
| B | DentalGPT Dental Eval SuiteMeasures multimodal models' understanding of dental images, including panoramic X-ray understanding, dental… | Healthcare |
| B | DocScopeDocScope measures whether multimodal LLMs can produce verifiable reasoning trajectories (evidence pages,… | Document Understanding |
| B | Dr-CiKDr-CiK measures whether agents can retrieve forecasting-relevant context from a noisy document corpus, filter… | Agentic |
| B | DrVD-BenchDrVD-Bench measures whether vision-language models reason like clinicians in medical image diagnosis across… | Biomedicine Healthcare |
| B | DuelLab GameBench 2DuelLab GameBench 2 measures how well AI models write game-playing programs, which are compiled and run… | Game |
| B | EbisuEbisu measures how well LLMs understand native Japanese financial language, via implicit commitment/refusal… | Finance |
| B | EgoCoT-BenchEgoCoT-Bench measures how well multimodal LLMs perform fine-grained, grounded and verifiable operation-centric… | Reasoning |
| B | EHRBenchEHRBench measures LLM clinical decision-making on nearly 1M EHR-grounded QA items covering diagnosis,… | Biomedicine Healthcare |
| B | EmoHarborEmoHarbor evaluates how well LLMs provide personalized emotional support by simulating users' internal worlds… | Behavioral |
| B | EscherVerseEscherVerse measures teleo-spatial reasoning in vision-language models: reasoning about physical dynamics,… | Spatial Reasoning |
| B | EvalPlusEvalPlus measures the functional correctness of LLM-generated code using test-augmented benchmarks HumanEval+… | Coding |
| B | ExploitBench v8-benchExploitBench v8-bench measures how far AI agents progress in exploiting 41 patched V8 bugs, graded across 16… | Cybersecurity |
| B | FAM-BenchFAM-Bench measures whether language and vision-language models can judge and rank dishes for suitability to… | Healthcare |
| B | FCMBenchFCMBench evaluates vision-language models on perception and credit-specific reasoning tasks over financial… | Finance |
| B | FEV Benchfev-bench measures time series forecasting models (pretrained, statistical, baseline) on 100 real-world… | Data |
| B | Fin-BiasFin-Bias measures how LLM investment decisions (Bullish/Neutral/Bearish ratings) from long firm analyst… | Finance |
| B | Finetuning with Scientific Data Increases HallucinationsSciFactCheck measures factuality hallucinations (unverifiability, overclaim, attribution) of LLMs, including… | Factuality |
| B | FinVerBenchFinVerBench measures whether LLMs can correctly verify the numerical consistency of corporate financial… | Finance |
| B | Flat-Pack BenchFlat-Pack Bench measures fine-grained spatio-temporal video understanding of large vision-language models on… | Video |
| B | FollowTableFollowTable measures how well retrieval models follow fine-grained content and schema instructions when… | Retrieval |
| B | ForecastBench-SimForecastBench-Sim is a contamination-free, simulated-world benchmark that scores LLMs' distributional… | Finance |
| B | FormalRewardBenchFormalRewardBench measures how well reward models distinguish correct from incorrect Lean 4 formal proofs… | Math |
| B | FrontierCodeFrontierCode measures whether model-generated code changes on real open-source repositories meet maintainers'… | Coding |
| B | Hindsight LLM Memory LeaderboardA leaderboard ranking LLMs, rerankers and embedding models on Hindsight agent-memory operations (retain fact… | Agentic |
| B | ImpermanentImpermanent measures how well time-series forecasting models generalize over time by scoring their forecasts… | Data |
| B | IneqMathIneqMath measures LLMs' ability to solve Olympiad-level inequality problems via bound estimation and relation… | Math |
| B | InvisibleBenchInvisibleBench evaluates AI models in multi-turn caregiver-support conversations for safety (crisis detection,… | Safety Healthcare |
| B | LatencyGridLatencyGrid measures latency of voice AI infrastructure across providers, including LLM time-to-first-token,… | Audio |
| B | LEXamLEXam measures LLM legal reasoning on 7,537 English and German law exam questions, both long-form open… | Legal |
| B | LiveSecBenchLiveSecBench is a dynamic safety leaderboard that scores LLMs, mainly in Chinese-language applications, on… | Safety |
| B | LiveSQLBenchLiveSQLBench measures how well LLMs and agents solve real-world text-to-SQL tasks, including… | Text TO Sql |
| B | MageBench Season 1MageBench Season 1 rates LLMs by playing Magic: The Gathering games across formats, reporting ratings, blunder… | Game |
| B | MBPP+MBPP+ measures LLM ability to synthesize short Python programs from natural language descriptions, scored by… | Coding |
| B | McEvalMcEval measures code LLMs on code generation, completion, and understanding tasks across 40 programming… | Coding |
| B | MCP-UniverseMCP-Universe measures how well LLMs and agents complete realistic, long-horizon tasks by interacting with… | Agentic |
| B | MCP-BenchMCP-Bench evaluates LLM agents on realistic multi-step tool-use tasks across 28 live MCP servers, scoring… | Agentic |
| B | MCPMarkMCPMark measures how well AI models use MCP servers to complete challenging, verifiable tasks across Notion,… | Agentic |
| B | MedAraBenchMedAraBench measures how well large language models answer Arabic multiple-choice medical questions across 19… | Biomedicine |
| B | Medical STT BenchmarkMeasures speech-to-text accuracy and latency on medical audio, including medical word error rate, drug-term… | Audio |
| B | METR Task-Completion Time Horizons 1.1METR Time Horizon 1.1 measures the human-expert task duration at which AI agents are predicted to succeed with… | Agentic Autonomy |
| B | MIR-SafetyBenchMIR-SafetyBench measures how vulnerable multimodal large language models are to safety attacks built on… | Safety |
| B | MMAUMMAU measures how well audio-language models understand and reason about speech, environmental sounds, and… | Multimodal |
| B | MMBench-CNMMBench-CN is the Chinese-translated version of MMBench, scoring vision-language models on multiple-choice… | Knowledge |
| B | MMSI-BenchMMSI-Bench measures multimodal LLMs' multi-image spatial reasoning through 1,000 human-designed… | Multimodal |
| B | MolecularIQMolecularIQ measures LLMs' chemical reasoning over molecular graphs via symbolically verifiable counting,… | Chemistry |
| B | MoralChoiceMoralChoice measures how language models handle moral decision scenarios, scored by accuracy and… | Generalization |
| B | NanoGPT Speedrun FrontierA leaderboard measuring how much autonomous coding agents can improve the nanoGPT speedrun training recipe,… | Agentic |
| B | OK-VQAOK-VQA measures visual question answering on images where answering requires outside knowledge beyond the… | Knowledge |
| B | Omni-MATHOmni-MATH measures LLMs' olympiad-level mathematical reasoning on 4428 competition problems across 33… | Math |
| B | PolyBenchPolyBench measures how well language models reason about polymer design tasks, from simple to complex… | Materials |
| B | POPEPOPE measures object hallucination in large vision-language models by polling yes/no questions about whether… | Knowledge |
| B | Professional Reasoning Bench - LegalProfessional Reasoning Bench Legal evaluates frontier LLMs on complex legal reasoning tasks drawn from… | Legal |
| B | ProductWebGenProductWebGen measures how well multimodal generative models create product showcase webpages (renderable HTML… | Multimodal Business |
| B | PutnamBenchPutnamBench measures how well theorem-proving systems can formally prove Putnam competition math problems in… | Math |
| B | AgentWorldBenchAgentWorldBench measures how faithfully language world models predict the next environment observation after… | Agentic |
| B | RealToxicityPromptsRealToxicityPrompts measures how likely language models are to generate toxic continuations (neural toxic… | Generalization |
| B | RedSage-BenchRedSage-Bench measures LLM cybersecurity knowledge, offensive skills, and security tool expertise through 30K… | Cybersecurity |
| B | RefusalBenchRefusalBench measures frontier LLM refusal behaviour and safety calibration on matched biological research… | Safety |
| B | RepairBenchRepairBench is a leaderboard that scores frontier models on zero-shot program repair of 574 single-function… | Coding |
| B | RuVerBenchRuVerBench measures how reliably LLM judges verify whether long agentic outputs (deep research reports and… | Agentic |
| B | SaaS-BenchSaaS-Bench measures whether computer-use agents can complete long-horizon professional workflows by driving a… | Software Engineering |
| B | scBenchscBench measures AI agents' ability to perform single-cell RNA-seq analysis tasks on real data snapshots,… | Science |
| B | SciKnowEvalSciKnowEval measures large language models' multi-level scientific knowledge (memory, comprehension,… | Science |
| B | SecCodeBenchSecCodeBench measures the security of code generated and repaired by AI models and agentic coding tools, with… | Cybersecurity |
| B | SEED-Bench-2SEED-Bench-2 measures the hierarchical capabilities of multimodal large language models, including image-text… | Knowledge |
| B | SimpleBenchSimpleBench measures LLMs on over 200 multiple-choice questions covering spatio-temporal reasoning, social… | Reasoning |
| B | SingGuard-BenchSingGuard-Bench measures how well multimodal guardrail models classify safety risks across 80+ risk types in… | Uncategorized |
| B | SPACENUM: Revisiting Spatial Numerical Understanding in VLMsSpaceNum measures whether vision-language models can ground numerical values (action magnitudes, coordinates)… | Spatial Reasoning |
| B | SpatialBenchSpatialBench measures whether AI agents can extract biological insight from real-world spatial transcriptomics… | Science |
| B | SpeakerSleuthSpeakerSleuth measures whether large audio-language models can reliably judge speaker consistency across… | Audio |
| B | SpiderSpider measures cross-domain text-to-SQL semantic parsing, scoring how well models generate complex SQL… | Data |
| B | StoryAlignStoryAlign (with its StoryRMB benchmark) measures how well reward models select human-preferred stories, using… | Writing |
| B | Structured Output BenchmarkSOB measures how accurately language models produce schema-compliant, value-correct JSON from text-normalized… | Structured Output |
| B | SuperGLUESuperGLUE measures model performance on a set of difficult English language understanding tasks via a… | Knowledge |
| B | SustainBenchSustainBench measures machine learning models on satellite and street-level imagery tasks for monitoring the… | Geospatial |
| B | SWT-BenchSWT-Bench measures whether models or agents can write tests that reproduce real-world user-reported GitHub… | Coding |
| B | TableBenchTableBench measures large language models' table question answering across fact checking, numerical reasoning,… | Knowledge |
| B | TAT-QATAT-QA measures question answering with numerical reasoning over hybrid tabular and textual content from… | Data |
| B | TaxBenchTaxBench measures how accurately and reliably AI models handle real-world tax tasks—tax knowledge and… | Finance Business |
| B | TDC AMESTDC AMES is a leaderboard scoring molecular machine learning models by AUROC on binary prediction of Ames… | Science |
| B | TDC BBB_MartinsTDC BBB_Martins is a Therapeutics Data Commons leaderboard scoring molecular models by AUROC on binary… | Science |
| B | TDC Bioavailability_MaTDC Bioavailability_Ma is a leaderboard that scores molecular models by AUROC on binary prediction of oral… | Science |
| B | TDC Caco2_WangTDC Caco2_Wang is a leaderboard benchmark scoring molecular models on regression of Caco-2 cell effective… | Science |
| B | TDC Clearance_Hepatocyte_AZA Therapeutics Data Commons ADMET leaderboard scoring molecular models by Spearman correlation on predicting… | Science |
| B | TDC Clearance_Microsome_AZA TDC ADMET leaderboard scoring molecular models by Spearman correlation on predicting microsomal clearance of… | Science |
| B | TDC CYP2C9_Substrate_CarbonMangelsA Therapeutics Data Commons ADMET leaderboard task scoring molecular models by AUPRC on predicting whether… | Science |
| B | TDC CYP2C9_VeithTDC CYP2C9_Veith is a Therapeutics Data Commons ADMET leaderboard scoring models by AUPRC on binary prediction… | Science |
| B | TDC CYP2D6_Substrate_CarbonMangelsA Therapeutics Data Commons ADMET leaderboard task scoring models by AUPRC on binary prediction of whether… | Science |
| B | TDC CYP2D6_VeithTDC CYP2D6_Veith is a Therapeutics Data Commons ADMET leaderboard scoring molecular models by AUPRC on binary… | Science |
| B | TDC CYP3A4_Substrate_CarbonMangelsA Therapeutics Data Commons ADMET leaderboard scoring molecular models by AUROC on binary prediction of… | Science |
| B | TDC CYP3A4_VeithTDC CYP3A4_Veith is a Therapeutics Data Commons leaderboard scoring molecular models by AUPRC on binary… | Science |
| B | TDC DILITDC DILI is a Therapeutics Data Commons leaderboard that scores molecular models by AUROC on binary prediction… | Science |
| B | TDC DRD3 DockingA molecule generation benchmark that scores methods on docking scores against the DRD3 target under limited… | Science |
| B | TDC Half_Life_ObachTDC Half_Life_Obach is a leaderboard benchmark scoring molecular models on regression of drug half-life… | Science |
| B | TDC hERGTDC hERG is a Therapeutics Data Commons leaderboard scoring molecular models by AUROC on binary prediction of… | Science |
| B | TDC HIA_HouTDC HIA_Hou is a leaderboard benchmark scoring molecular machine learning models by AUROC on binary prediction… | Science |
| B | TDC LD50_ZhuTDC LD50_Zhu is a Therapeutics Data Commons leaderboard that scores molecular models on regression of acute… | Science |
| B | TDC Lipophilicity_AstraZenecaA Therapeutics Data Commons ADMET leaderboard scoring molecular models by MAE on regressing lipophilicity… | Science |
| B | TDC Pgp_BroccatelliTDC Pgp_Broccatelli is a Therapeutics Data Commons leaderboard that scores molecular models on binary… | Science |
| B | TDC PPBR_AZTDC PPBR_AZ is a Therapeutics Data Commons leaderboard that scores models by mean absolute error when… | Science |
| B | TDC Solubility_AqSolDBTDC Solubility_AqSolDB is a Therapeutics Data Commons ADMET leaderboard that scores molecular models by MAE on… | Science |
| B | TDC VDss_LombardoTDC VDss_Lombardo is a Therapeutics Data Commons leaderboard scoring molecular models on regression of drug… | Science |
| B | TextArena Balanced SubsetTextArena Balanced Subset is a leaderboard slice that ranks LLMs by TrueSkill ratings from competitive play… | Game |
| B | TRLawBenchTRLawBench measures how well large language models handle Turkish law across foundational multiple-choice,… | Legal |
| B | VariantBenchVariantBench measures how well AI agents perform genetic variant discovery and interpretation across 118… | Science Biology+1 |
| B | Visual Aesthetic BenchmarkVisual Aesthetic Benchmark measures whether multimodal models can match expert aesthetic judgment by selecting… | Multimodal |
| B | VitaBenchVitaBench measures how well LLM agents complete multi-turn, tool-heavy consumer-service tasks across food… | Agentic |
| B | VQAv2VQAv2 measures how well models answer open-ended natural-language questions about images, using a balanced… | Knowledge |
| B | VTBVisualToolBench measures how well multimodal LLMs use image-manipulation and other tools to reason with visual… | Multimodal |
| B | WildClawBenchWildClawBench measures how well autonomous AI agents complete 60 hand-crafted real-world tasks inside a live… | Agentic |
| B | WolfBenchWolfBench measures how consistently AI models and coding-agent combinations solve Terminal-Bench 2.0 tasks,… | Agentic |
| B | Workspace-BenchWorkspace-Bench measures how well AI agents operate in realistic workspaces with large-scale file… | Agentic Business |
| B | AGC-BenchAGC-Bench measures the creativity of LLMs across 78 datasets spanning brainstorming, problem solving, STEM,… | |
| B | KernelBench CUDAKernelBench CUDA scores coding agents on writing fast, correct CUDA GPU kernels for hard tasks like fused MoE,… | |
| B | KernelBench MegaKernelBench Mega measures whether coding agents can write a single fused GPU megakernel (e.g., Kimi-Linear… | |
| B | SPIKE-BenchSPIKE-Bench measures whether LLMs comply with toxin-design prompts and generate biologically plausible,… | |
| B | URSAURSA benchmarks retrosynthesis systems and LLMs on synthesis planning for novel drug-like target molecules,… | |
| B | ACEBenchACEBench measures how well large language models use tools across normal, special (ambiguous or incomplete… | Tool Use |
| B | Advancing ESG IntelligenceA three-level benchmark built from 310 corporate sustainability reports that evaluates LLMs and agents on ESG… | Finance |
| B | AfroBench-LiteAfroBench measures how well large language models perform across 64 African languages on understanding,… | Multilingual |
| B | Agent IslandAgent Island measures language-model agents' skill in a multiplayer winner-take-all game of cooperation,… | Agentic |
| B | AgentCIBenchAgentCIBench measures whether computer-use agents leak contextually inappropriate personal information across… | Agentic |
| B | Agentic Skills Evaluation FrameworkEvaluates how well LLM agent-model configurations use agent skills, scoring instruction-following and goal… | Agentic |
| B | AgentRxAgentRx evaluates single- and multi-agent LLM systems on multimodal clinical risk prediction tasks using… | Healthcare |
| B | AGL1KAGL1K measures how well audio-language models can infer the geographic location (continent, country, city,… | Audio |
| B | AICD BenchAICD Bench measures how well detectors identify AI-generated source code, covering robust binary… | Coding |
| B | AlignBenchAlignBench measures how well large language models are aligned in Chinese, using 683 real-scenario queries… | General Knowledge |
| B | ALL Bench LLMALL Bench LLM is a composite leaderboard that aggregates and cross-verifies LLM scores across knowledge,… | General Knowledge |
| B | Altered RiddlesAltered Riddles measures how often language models fall back on memorized answers (Conditioned Override Rate)… | Knowledge |
| B | ArchSIBenchArchSIBench measures the architectural spatial intelligence of vision-language models across perception,… | Spatial Reasoning |
| B | ASPECTSIMASPECTSIM evaluates how well LLMs and embedding models measure document similarity conditioned on an… | Language-understanding |
| B | AsyncToolAsyncTool measures how well LLM-based agents perform asynchronous function calling, coordinating multiple… | Agentic |
| B | AuAuAuAu measures the risk that LLMs generate responses with authoritarian tendencies, using psychometric… | Alignment |
| B | Audio Reasoning ChallengeThe Audio Reasoning Challenge scores single models and agents on deep audio reasoning over speech, sound,… | Audio |
| B | BABILongBABILong measures how well language models extract and reason over facts distributed within extremely long… | Long Context |
| B | BanglaMedVQABanglaMedVQA measures how well LLMs and vision-language models answer clinically validated Bangla medical… | Biomedicine Healthcare |
| B | Benchmarks are Not EnoughRAMP measures LLM software engineering agents on a long-horizon, serially dependent compiler-construction… | Agentic |
| B | Beyond QuestionsBeQu measures the parametric knowledge LLMs surface in response to open-ended elicitation prompts about 10,000… | Factuality |
| B | BIG-Bench Extra HardBIG-Bench Extra Hard (BBEH) measures broad general reasoning in LLMs with harder replacements for each… | Knowledge |
| B | BigCodeBench-HardBigCodeBench-Hard measures LLM code generation on a 148-task subset of BigCodeBench aligned with real-world… | Coding |
| B | BLINKBLINK measures core visual perception abilities of multimodal LLMs via 3,807 multiple-choice questions… | Multimodal |
| B | BLURBBLURB measures biomedical natural language understanding and reasoning across thirteen PubMed-based datasets… | Healthcare |
| B | BOLDBOLD measures the tendency of language models to produce biased completions across demographic groups. | Generalization |
| B | BTZSCBTZSC measures zero-shot text classification performance (macro F1) of cross-encoders, embedding models,… | Classification |
| B | C4STYLIC4STYLI measures LLMs' recognition and generation of culturally resonant aesthetic stylistics in translated… | Language |
| B | CalArena: A Large-Scale Post-Hoc Calibration BenchmarkCalArena is a standardized benchmark of nearly 2000 experiments that compares post-hoc calibration methods for… | Classification |
| B | CareQA-VisionCareQA-Vision measures vision-language models on medical and nursing image-based questions drawn from Spain's… | Healthcare |
| B | CC-OCR V2CC-OCR V2 measures how well large multimodal models handle real-world document OCR tasks: text recognition,… | Ocr |
| B | CFMMECFMME evaluates large vision-language models on perception, understanding, reasoning and cognition across… | Finance |
| B | CG-BenchCG-Bench measures multimodal LLMs' long-video understanding through clue-grounded question answering, scoring… | Video |
| B | ChartBenchChartBench measures multimodal LLMs' chart comprehension and data reliability through complex visual reasoning… | Knowledge |
| B | Clembench Text v3.0Clembench Text v3.0 scores chat-optimized language models as conversational agents playing text games,… | Agentic |
| B | ClinicalMCClinicalMC measures LLM performance on multi-course clinical decision-making as a patient's condition evolves… | Healthcare |
| B | ComplexFuncBenchComplexFuncBench measures LLMs' ability to perform multi-step and constrained function calling across five… | Tool Use |
| B | CrossViewBenchCrossViewBench measures multimodal LLMs' cross-view spatial understanding across correspondence,… | Spatial Reasoning |
| B | CSimpleQAChinese SimpleQA measures the factuality of language models answering short Chinese factual questions across 6… | General Knowledge |
| B | CULTURE-MTCULTURE-MT measures how well models translate social media user-generated content while preserving cultural… | Translation |
| B | CURE-MedCUREMED-BENCH measures the logical correctness and language consistency of LLMs on open-ended, verifiable… | Healthcare |
| B | CVGEvalKitCVGEvalKit evaluates language models on criminal court view generation from case facts and charge prediction… | Legal |
| B | DataBenchDataBench measures how well LLMs answer questions over real-world tabular data, using 80 datasets and 1,822… | Data |
| B | DatBenchDatBench measures vision-language model capabilities across nine areas (charts, counting, documents, general… | Multimodal Understanding |
| B | DeepTumorVQADeepTumorVQA measures how well medical vision-language models and tool-augmented agents answer 3D abdominal CT… | Biomedicine Healthcare |
| B | Defects4JDefects4J is a collection of 854 reproducible real-world Java bugs with supporting infrastructure, used to… | Coding |
| B | DermoBenchDermoBench evaluates multimodal LLMs on 11 dermatology tasks across morphology, diagnosis, reasoning, and… | Healthcare |
| B | DiffCap-BenchDiffCap-Bench evaluates multimodal large language models on image difference captioning, describing… | Vision |
| B | DisastQADisastQA measures how accurately LLMs answer multiple-choice and open-ended disaster-management questions… | Question Answering |
| B | DIYHealth SuiteDIYHealth Suite includes DIYHealthBench, a benchmark evaluating foundation models on 11 home-based health care… | Healthcare |
| B | DocAtlasDocAtlas-Bench measures multilingual document OCR and understanding (text, tables, formulas, reading order)… | Document Understanding |
| B | EComStageEComStage evaluates LLMs as e-commerce agents across stage-wise reasoning (perception, planning, action) on… | Business |
| B | EgoMemReasonEgoMemReason measures how well multimodal models and agentic frameworks reason over week-long egocentric video… | Reasoning |
| B | EgoSchemaEgoSchema measures very long-form video understanding through multiple-choice questions on three-minute… | Knowledge |
| B | ENAMELENAMEL measures how efficient LLM-generated code is, using HumanEval-derived problems, expert reference… | Coding |
| B | EnterpriseArenaEnterpriseArena measures LLM agents' ability to perform long-horizon resource allocation under uncertainty as… | Finance Business |
| B | EpiQALEpiQAL measures how well LLMs answer epidemiological questions grounded in research articles, testing factual… | Healthcare |
| B | EstGraphEstGraph measures LLMs' reasoning ability to estimate properties of very large, partially accessible graphs… | Reasoning |
| B | Facet-ProbeFacet-Probe audits whether multimodal large language models change their answers when order-irrelevant… | Multimodal |
| B | Fiction.LiveBenchFiction.LiveBench measures how well models comprehend and reason over fiction/narrative text, with models… | Long Context |
| B | Fin-RATEFin-RATE measures how well LLMs perform financial analyst tasks over U.S. SEC filings, including detail… | Finance |
| B | FinChainFinChain evaluates models' financial chain-of-thought reasoning, scoring them with ChainEval plus ROUGE and… | Finance |
| B | FinEvalFinEval measures large language models' Chinese financial domain knowledge and practical abilities across… | Finance |
| B | FraudBenchFraudBench measures how well MLLMs, AI-generated image detectors, and humans detect AI-generated fake damage… | Multimodal |
| B | FreshQAFreshQA is a regularly updated question-answering benchmark that measures the factuality of LLM answers on… | Generalization |
| B | FullStackBench enFullStack Bench measures LLMs' full-stack coding ability across application domains and 16 programming… | Coding |
| B | FullStackBench zhFullStackBench zh is the Chinese subset of FullStack Bench, which evaluates LLMs on full-stack programming… | Coding |
| B | FutureOmniFutureOmni measures how well multimodal LLMs forecast future events from audio-visual video context via… | Multimodal Understanding |
| B | GenAI-BenchGenAI-Bench measures how well multimodal LLMs judge AI-generated images, image edits and videos, scored by… | Multimodal |
| B | GeneBenchGeneBench measures whether AI agents can carry out realistic multi-stage scientific data analysis in genetics… | Science |
| B | GEO-Bench 2GEO-Bench-2 evaluates geospatial foundation models on Earth Observation tasks including classification,… | Vision |
| B | GeoCode LeaderboardA leaderboard scoring large language models on geospatial code generation for Google Earth Engine using… | Geospatial |
| B | GeoSym-BenchGeoSym-Bench is an expert-curated set of 511 complex multimodal geometry problems for rigorously evaluating… | Math |
| B | GIFT-EvalGIFT-Eval measures zero-shot time series forecasting performance of foundation and other models across 23… | Data |
| B | GPF-LiveNewsGPF-LiveNews is a streaming evaluation protocol and benchmark that audits group-conditioned framing bias in… | Factuality |
| B | CORE-Bench HardCORE-Bench Hard measures whether AI agents can computationally reproduce published scientific papers' results… | Coding |
| B | Online Mind2WebOnline Mind2Web measures how well web agents complete 300 realistic tasks on 136 live websites, scored with… | Agentic |
| B | Hallucinations LeaderboardThe Hallucinations Leaderboard evaluates LLMs on hallucination-related benchmarks covering factuality and… | Behavioral |
| B | HalluScoreHalluScore is an Arabic question answering benchmark of 827 questions that evaluates hallucination behavior in… | Factuality |
| B | HardMTBenchHardMTBench measures bidirectional Chinese-English machine translation quality on difficult,… | Translation |
| B | HardSecBenchHardSecBench measures whether LLMs generate secure hardware code, scoring functional and security correctness… | Cybersecurity |
| B | HarmVideoBenchHarmVideoBench measures how well large multimodal models understand harmful videos across observable evidence,… | Safety |
| B | Harvey LAB-AAHarvey LAB-AA measures how well AI agents complete real-world legal work across 120 private tasks in 24… | Legal |
| B | HealthBench-BR and PCDT-QAHealthBench-BR and PCDT-QA measure LLM recall of Brazilian SUS official clinical guideline knowledge via… | Biomedicine Healthcare |
| B | HEIM: Holistic Evaluation of Text-to-Image ModelsHEIM evaluates text-to-image generation models across 12 aspects including image-text alignment, quality,… | Creative Media Image Generation |
| B | HMMT February 2026Measures LLM performance on final-answer math problems from the February 2026 Harvard-MIT Mathematics… | Math |
| B | HREFHREF measures how well language models follow instructions, using human response-guided automatic evaluation… | Instruction Following |
| B | HUMAINEHUMAINE is a leaderboard ranking LLMs based on detailed human feedback about their performance in real-world… | Human-preference |
| B | Hunt Instead of WaitHunt Instead of Wait introduces DDR-Bench, a checklist-based benchmark measuring how well LLM agents… | Agentic |
| B | HyperTrackHyperTrack evaluates vision-language model agents on offline mobile GUI navigation across over 16000… | GUI |
| B | IdioLinkIdioLink measures whether embedding/retrieval models can link idiomatic expressions to conceptually equivalent… | Retrieval |
| B | IFMTBenchIFMTBench measures how well models follow translation instructions (format preservation, glossaries, context,… | Multilingual |
| B | IHBenchIHBench measures how well voice agents recover after user interruptions in multi-step, state-machine-driven… | Speech Healthcare+1 |
| B | INCLUDE-base-44 European LanguagesMeasures multilingual LLMs' regional knowledge and reasoning via multiple-choice questions from local exams,… | Multilingual |
| B | IndoBiasIndoBias measures social bias and stereotypes of LLMs in Indonesian and three local languages (Javanese,… | Safety |
| B | IndustryBenchIndustryBench measures LLM knowledge and safety-aware correctness on industrial procurement QA grounded in… | Domain Business |
| B | InfiniteBM Heads-Up No-Limit Hold'emA head-to-head LLM arena ladder that rates models with Bradley-Terry Elo on heads-up no-limit hold'em poker… | Game |
| B | InfographicVQAInfographicVQA measures models' ability to answer natural language questions about infographic images,… | Knowledge |
| B | IntentGraspIntentGrasp measures how well LLMs understand the intent behind speech, conversation, and writing, using test… | Knowledge |
| B | Inverse Turing BenchInverse Turing Bench measures how accurately models can tell which of two paired multi-turn dialogue… | Classification |
| B | ITBench-AAITBench-AA measures AI agents' ability to perform Kubernetes incident root-cause analysis (SRE) from offline… | Agentic |
| B | J1-ENVSJ1-ENVS measures how well LLM agents perform interactive Chinese legal tasks, from consultation and document… | Legal |
| B | JMed48kJMed48k measures how well vision-language models answer Japanese national healthcare licensing exam questions,… | Biomedicine Healthcare |
| B | K-12EduBenchK-12EduBench evaluates LLMs' subject-level knowledge, subject-specific problem solving, and educational goal… | Education |
| B | Korean Canonical Legal BenchmarkKCL measures language models' Korean legal reasoning independently of domain knowledge, using multiple-choice… | Legal |
| B | Ko-WideSearchKo-WideSearch measures how well web agents exhaustively enumerate the members of a closed set from Korean web… | Agentic |
| B | LegalCiteBenchLegalCiteBench measures how reliably language models recover, complete, verify and match U.S. case law… | Legal |
| B | LingLanMiDianLingLanMiDian (LingLan) evaluates large language models on Traditional Chinese Medicine knowledge recall,… | Healthcare |
| B | LingOly-TOOLingOly-TOO measures language models' reasoning on obfuscated Linguistics Olympiad problems, separating… | Knowledge |
| B | LiveMedBenchLiveMedBench evaluates large language models on time-stamped real-world medical consultation cases using… | Healthcare |
| B | LLM-WikiRaceLLM-WikiRace measures how well language models plan and reason over world knowledge by navigating Wikipedia… | Agentic |
| B | LOGICAL-COMMONSENSEQALOGICAL-COMMONSENSEQA measures whether language models can judge the joint plausibility of pairs of… | Reasoning |
| B | LogicScoreLogicScore evaluates the global logical reasoning integrity (completeness, essentiality, determinateness) of… | Reasoning |
| B | LongBench ProLongBench Pro measures long-context understanding of LLMs on 1,500 natural bilingual (English and Chinese)… | Long Context |
| B | LongVideoBenchLongVideoBench measures how well large multimodal models retrieve and reason over detailed information in… | Long Context |
| B | M3MAD-BenchM3MAD-Bench evaluates multi-agent debate methods across knowledge, mathematics, medicine, natural sciences and… | Agentic |
| B | MacAgentBenchMacAgentBench measures how well computer use agents complete 676 real-world tasks across 25 macOS… | GUI |
| B | Magis-BenchMagis-Bench measures LLMs on magistrate-level legal writing tasks, including discursive legal analysis and… | Legal |
| B | Math-VRMath-VR measures how well VLMs, unified models and LLMs solve bilingual math problems requiring visual… | Multimodal |
| B | MechVQAMechVQA measures how well multimodal LLMs understand mechanical engineering drawings across recognition,… | Document Understanding |
| B | Med-RewardBenchMed-RewardBench measures how well reward models and judges evaluate medical multimodal LLM responses across… | Biomedicine Healthcare |
| B | MedAgentGymMedAgentGym measures LLM agents' code-centric reasoning on biomedical data science tasks in executable sandbox… | Biomedicine |
| B | MedConsultBenchMedConsultBench evaluates medical consultation agents across the full online consultation cycle, from history… | Healthcare |
| B | MedCTAMedCTA measures how well multimodal medical tool agents select and use clinical tools across 107… | Healthcare |
| B | MedGUIDEMedGUIDE measures whether LLMs can make clinical decisions consistent with NCCN cancer guideline decision… | Healthcare |
| B | MedLayXPlainMedLayXPlain measures whether medical vision-language models can generate accurate expert and… | Biomedicine Healthcare |
| B | Memory Architectures for Multi-Turn Text-to-SQLEnterpriseMem-Bench measures how well models with different memory architectures answer multi-turn Text-to-SQL… | Text TO Sql Business |
| B | Merge-BenchMerge-Bench measures how well language models resolve real-world version control merge conflicts, using 7938… | Coding |
| B | MHGraphBenchMHGraphBench measures how well LLMs recognize mental-health entities, judge relations, and perform two-hop… | Biomedicine Healthcare |
| B | MindEdit-BenchMindEdit-Bench measures whether vision-language models can perform object-level counterfactual spatial… | Spatial Reasoning |
| B | MineExplorerMineExplorer measures how well multimodal LLM agents sustain open-world exploration in Minecraft, using… | Game |
| B | MINTEvalMINTEval measures how well memory-augmented agents and long-context LLMs recall and aggregate information over… | Agentic |
| B | MixReaMixRea measures whether LLMs attend to subtle implicit contextual cues while following explicit task… | Reasoning |
| B | MMBench-VideoMMBench-Video measures large vision-language models' video understanding, including temporal reasoning, using… | Multimodal |
| B | MMDocBenchMMDocBench measures fine-grained visual perception, reasoning and grounding of large vision-language models on… | Document Understanding |
| B | MMEBMMEB measures universal multimodal embedding models across classification, visual question answering,… | Multimodal |
| B | MMGistMMGist is a curated vision-language benchmark of 7,262 items across seven capability dimensions that measures… | Multimodal |
| B | MMTUMMTU measures how well models understand, reason over, and manipulate real tables across 25 expert-level table… | Data |
| B | MolDeToxMolDeTox evaluates LLMs and VLMs on stepwise fragment-level editing of molecules to reduce toxicity (molecular… | Biomedicine |
| B | MolRecBench-WildMolRecBench-Wild measures how well models translate real-world molecular diagrams from chemistry papers into… | Chemistry |
| B | MPDocBench-ParseMPDocBench-Parse measures how well models parse multi-page, visually rich documents in English and Chinese… | Document-ai |
| B | MRBenchMRBench measures grain blindness in universal multimodal retrieval models, scoring how well they handle… | Retrieval |
| B | MSRVTT-QAMSRVTT-QA measures how well models answer natural-language questions about the content of short videos drawn… | Knowledge |
| B | MT-BenchMT-Bench measures chat assistants' quality on challenging multi-turn open-ended questions, scored by strong… | Generalization |
| B | MT-JailBenchMT-JailBench is a modular evaluation framework that benchmarks multi-turn jailbreak attacks on large language… | Safety |
| B | MTRefSeg-21KMTRefSeg-21K measures how well vision-language models segment language-described temporal changes across… | Vision |
| B | MuirBenchMuirBench measures robust multi-image understanding of multimodal LLMs via 2,600 multiple-choice questions… | Multimodal |
| B | MulTaBenchMulTaBench measures predictive performance of tabular learners on 40 multimodal tabular datasets combining… | Tabular |
| B | Multi-LCBMulti-LCB measures LLM code-generation ability on contamination-aware competitive programming tasks across… | Coding |
| B | MultiCaptionMultiCaption measures how well models detect contradictions between multilingual claims about the same image… | Multimodal Understanding |
| B | MultiSoc-4DMultiSoc-4D is a Bengali social media benchmark of 58K+ comments annotated for category, sentiment, hate… | Human-preference |
| B | MuseBenchMuseBench measures how well multimodal LLMs understand creative intent in audiovisual arts (cinema, visual… | Creative Media |
| B | Natural2CodeNatural2Code is a code-generation benchmark that scores models on writing code from natural-language prompts;… | Coding |
| B | NeuroAtlasNeuroAtlas measures how well EEG and time-series foundation models perform on clinical EEG (epilepsy, sleep,… | Biomedicine Healthcare |
| B | OccuBenchOccuBench measures how well AI agents complete professional occupational tasks across 10 industries and 65… | Domain |
| B | OCR-RobustOCR-Robust measures how robust vision-language models' OCR reasoning is under controlled visual perturbations… | Ocr |
| B | OCRBenchOCRBench measures the OCR capabilities of large multimodal models across text recognition, scene-text VQA,… | Ocr |
| B | OCRROCRR measures how quickly classification systems recover novel-class accuracy and retain original-distribution… | Classification |
| B | OJBenchOJBench measures competitive-level code reasoning of LLMs using 232 programming competition problems from NOI… | Knowledge |
| B | OmniGAIAOmniGAIA measures how well omni-modal AI agents solve multi-hop tasks requiring deep reasoning and multi-turn… | Agentic |
| B | OPT-BENCHOPT-BENCH measures whether LLM agents can iteratively self-improve solutions from environmental feedback… | Self Improvement |
| B | OR-SpaceOR-Space measures whether LLM agents can build, revise, and explain industrial operations research… | Agentic Business |
| B | P3B3P3B3 measures LLM bias toward European versus Brazilian Portuguese varieties and controllability of variety in… | Multilingual |
| B | PAGE BenchPAGE Bench measures GUI agents' ability to perform point-precise geometric construction actions in continuous… | GUI |
| B | PALM-BenchPALM-Bench evaluates large audio-language models on personalized tasks: recognizing personal concepts (e.g.,… | Audio |
| B | PandaBenchPandaBench measures LLM safety against jailbreak attacks, evaluating attack success rates across 49 LLMs under… | Safety |
| B | Patient-Similarity Cohort Reasoning in Clinical Text-to-SQLCLINSQL measures how well language models generate executable SQL for clinical questions over MIMIC-IV EHR… | Text TO Sql |
| B | PEC-HomePEC-Home is a simulated smart-home dataset that scores how well LLM home assistants interpret progressively… | Language-understanding |
| B | Perception or PrejudiceMeasures whether multimodal LLMs perceive Big Five personality from videos through grounded behavioral… | Behavioral |
| B | From Perception to ActionCHAIN is an interactive 3D physics-driven benchmark that evaluates whether vision-language models can… | Embodied |
| B | PerfCodeBenchPerfCodeBench measures whether LLMs can produce correct and efficient system-level high-performance code… | Coding |
| B | PersistBenchPersistBench measures safety risks of long-term memory in LLM assistants, namely cross-domain leakage and… | Safety |
| B | PersonaArenaPersonaArena evaluates LLMs' persona-level role-playing through multi-turn simulated social interactions… | Agentic |
| B | Physical AI Bench GenerationPhysical AI Bench's generation track evaluates video generative models on predicting physically plausible… | Multimodal |
| B | Physical AI Bench UnderstandingPhysical AI Bench (PAI-Bench) evaluates multimodal LLMs and video generative models on perception and… | Multimodal |
| B | PhysicsFinalsPHYSICS (PhysicsFinals) measures foundation models' ability to solve 1,297 expert-annotated university-level… | Science |
| B | PIQAPIQA measures whether language models can answer physical commonsense questions, choosing the right solution… | Language |
| B | PL-MTEBPL-MTEB evaluates text embedding models on Polish-language classification, clustering, pair classification,… | Language |
| B | PlanningBenchPlanningBench measures LLMs' ability to produce complete, verifiable plans for realistic planning problems… | Reasoning |
| B | PlantMarkerBenchPlantMarkerBench measures how well language models interpret literature-grounded evidence for… | Biology |
| B | PLawBenchPLawBench evaluates LLMs on realistic legal practice tasks (legal consultation, case analysis, document… | Legal |
| B | PodBenchPodBench measures how well LLMs generate podcast scripts from long source documents under complex… | Audio |
| B | POLAR-BenchPOLAR-Bench measures privacy-utility trade-offs of LLM agents that must follow a user's privacy policy while… | Agentic |
| B | PolySpeech-100PolySpeech-100 measures native-level speech comprehension of Speech-LLMs across 110 languages and dialects,… | Speech |
| B | ProactBenchProactBench measures conversational proactivity: whether LLM assistants notice and act on user needs that are… | Agentic |
| B | PROBE Binding AffinityPROBE Binding Affinity measures how well protein representation methods estimate protein-protein binding… | Science |
| B | PROBE Function PredictionPROBE Function Prediction measures how well protein representation models predict ontology-based protein… | Science |
| B | ProsaProsa measures LLM quality on 1,000 real multi-turn Brazilian Portuguese user chats from WildChat, scored with… | Language |
| B | PTEBPTEB (Paraphrasing Text Embedding Benchmark) evaluates the robustness of sentence embedding models by… | Language |
| B | PubMedQAPubMedQA measures a model's ability to answer biomedical research questions with yes/no/maybe using the… | Biomedicine Healthcare |
| B | Qiskit QuantumKatasQiskit QuantumKatas measures how well LLMs write Qiskit quantum computing code across 350 tasks spanning… | Coding |
| B | RankJudgeRankJudge evaluates and ranks LLM judges on their ability to identify the worse of paired multi-turn,… | Agentic |
| B | RealityTestRealityTest measures whether AI text and speech models disclose that they are AI when users probe their… | Safety |
| B | RealStreamEvalRealStreamEval is a frame-level multi-turn evaluation protocol that scores video-language models on streaming… | Video |
| B | ReasonTabQAReasonTabQA measures LLM table question answering and reasoning over complex industrial tables with… | Tabular |
| B | RedactionBenchRedactionBench measures how well models redact personally identifiable information in context, across 200… | Safety |
| B | ReEfBench: Quantifying the Reasoning Efficiency of LLMsReEfBench evaluates the reasoning efficiency of LLMs via a neuro-symbolic, process-centric evaluation of… | Efficiency |
| B | RepoQARepoQA measures LLMs' long-context code understanding by having them find functions in repositories from… | Coding |
| B | RIFT-Bench: Dynamic Red-teaming For Agentic AI SystemsRIFT-Bench evaluates the security of agentic AI systems through automated dynamic red-teaming, discovering… | Agentic |
| B | RPC-BenchRPC-Bench measures how well foundation models comprehend computer science research papers through 15K… | Document Understanding |
| B | RTEBRTEB measures the retrieval accuracy of embedding models on open and private datasets across enterprise… | Retrieval |
| B | RTL-OPTRTL-OPT measures how well LLMs optimize RTL hardware code for power, performance, and area while preserving… | Coding |
| B | RWGBenchRWGBench measures how well systems generate related work sections for computer science papers, scoring… | Writing |
| B | RxEvalRxEval measures LLM inpatient prescribing capability via multiple-choice questions requiring selection of… | Healthcare |
| B | SaaSBenchSaaS-Bench measures whether computer-use agents can complete long-horizon professional workflows by driving a… | Software Engineering Business |
| B | SABER-MathSABER-Math measures how well retrieval and embedding models rerank documents by fine-grained mathematical… | Retrieval |
| B | SaLADSaLAD measures whether multimodal LLMs give safe, informative safety-warning responses to 2,013 real-world… | Safety |
| B | SceneFunRISceneFunRI measures whether vision-language models can infer the 2D locations of invisible (occluded)… | Spatial Reasoning |
| B | SciHorizon-GENESciHorizon-GENE measures how well LLMs reason from gene-level knowledge to functional understanding,… | Biology |
| B | ScreenSpotScreenSpot measures GUI grounding: locating the screen element matching a natural-language instruction across… | Agentic |
| B | Script SensitivityScript Sensitivity benchmarks open-source language models via perplexity on Unicode, Romanized and… | Multilingual |
| B | SDGBiasBenchSDGBiasBench measures decision-level and estimation-level biases of vision-language models in Sustainable… | Reasoning |
| B | Seal-0Seal-0 measures factual accuracy and reasoning of search-augmented language models on the hardest fact-seeking… | Search |
| B | SeePhys ProSeePhys Pro measures whether multimodal models keep the same physics reasoning ability as critical problem… | Reasoning |
| B | Sefaria Rabbinic Embedding LeaderboardMeasures how well embedding models retrieve and align Rabbinic Hebrew/Aramaic texts with their English… | Retrieval |
| B | SenseBenchSenseBench measures how well vision-language models perceive and describe low-level degradations in remote… | Vision |
| B | ShoppingMMLUShopping MMLU measures how well LLMs act as online shop assistants across 57 Amazon-derived tasks covering… | Business |
| B | SiT-BenchSiT-Bench measures the spatial intelligence of LLMs from coordinate-aware textual scene descriptions, without… | Spatial Reasoning |
| B | SkillRetSkillRet measures how well retrievers select the right skill from large libraries of public agent skills given… | Retrieval |
| B | SonicBenchSonicBench measures whether large audio language models can perceive 12 fundamental physical audio attributes… | Audio |
| B | SpaceDGSpaceDG measures the robustness of multimodal LLMs' spatial reasoning on indoor-scene visual questions under… | Spatial Reasoning |
| B | SpatialUAVSpatialUAV measures spatial intelligence of vision-language models on real low-altitude UAV imagery, covering… | Spatial Reasoning |
| B | Spider 2.0Spider 2.0 measures how well language models and agents solve real-world enterprise text-to-SQL workflow… | Data Business |
| B | StakeBenchStakeBench evaluates whether LLMs understand language grounded in market commitment, using prediction-market… | Finance |
| B | STALESTALE measures whether LLM agents detect and act on implicit conflicts where later observations invalidate… | Agentic |
| B | StemBindStemBind is a diagnostic benchmark that measures where multimodal LLMs fail in abstract visual reasoning by… | Reasoning |
| B | STEMVerseSTEMVerse is a diagnostic evaluation framework that scores LLMs' STEM reasoning across over 20,000 problems… | Reasoning |
| B | StereoTalesStereoTales measures social bias and harmful stereotypes emerging in open-ended LLM story generation across 10… | Safety |
| B | Stick To Your Role!Stick To Your Role! measures how stably LLMs maintain simulated personas' personal values across many… | Alignment |
| B | STT-ArenaSTT-Arena measures whether LLM agents can detect mid-task spatio-temporal disruptions and adaptively replan in… | Agentic |
| B | SwordBenchSwordBench evaluates methods for steering image representations of vision models in concept removal tasks,… | Vision |
| B | SycoEval-EMSycoEval-EM measures whether LLMs acquiesce to adversarial patient persuasion for guideline-discordant care in… | Healthcare |
| B | TableVistaTableVista measures foundation models' multimodal table reasoning across 30,000 rendered table images that… | Reasoning |
| B | TADDLE BenchmarkAn expert-annotated benchmark of 1,800 LLM-generated peer reviews on 50 ICLR 2025 papers, scoring systems on… | Classification |
| B | Tau2 AirlineTau2 Airline is the airline customer-service domain of τ2-bench, which evaluates conversational agents that… | Agentic Business |
| B | TAU3-Benchτ³-Bench evaluates conversational customer-support agents completing multi-step tasks with tools, including… | Agentic |
| B | TheoremQATheoremQA measures how well AI models apply university-level theorems to solve challenging problems in math,… | Math |
| B | TIDETIDE is an agent-agnostic, environment-agnostic diagnostic evaluation framework that scores LLM agents'… | Agentic |
| B | TOBenchTOBench (also called MM-ToolBench) measures omni-modal tool-using agents on 100 executable MCP-based tasks in… | Tool Use Business |
| B | Trip+Trip+ measures how well language-model agents plan and revise personalized, minute-level travel itineraries… | Agentic |
| B | TriViewBenchTriViewBench measures how multimodal LLMs handle multi-view structural visual reasoning (local decisions,… | Reasoning |
| B | Trust No ToolTRUST-Bench measures whether LLM agents and defenses can detect and avoid harmful final actions when a… | Tool Use |
| B | TuRTLe Code CompletionTuRTLe Code Completion evaluates LLMs on completing RTL (Verilog) hardware code, scoring syntax,… | Coding |
| B | TuRTLe Line CompletionTuRTLe Line Completion scores LLMs on completing lines of RTL (Verilog) hardware design code, as one task in… | Coding |
| B | TuRTLe Module CompletionTuRTLe Module Completion scores LLMs on completing RTL hardware modules drawn from over 1000 real-world Tiny… | Coding |
| B | TuRTLe Spec-to-RTLTuRTLe Spec-to-RTL scores LLMs on generating hardware RTL (Verilog) from specifications, measuring syntax,… | Coding |
| B | UAVBenchUAVBench measures LLM reasoning for autonomous UAV systems using 50,000 validated flight scenarios and 50,000… | Agentic |
| B | UbuntuGuardUbuntuGuard measures how well guardian models and LLMs enforce culturally grounded, context-specific safety… | Safety |
| B | Uncertainty Quantification for Computer-Use AgentsArgus benchmarks post-hoc uncertainty quantification methods for vision-language computer-use agents on… | GUI |
| B | URIAL BenchURIAL Bench scores base (untuned) language models aligned only through in-context prompting on multi-turn… | Instruction Following |
| B | Poker AgentPoker Agent measures how well language models play 10-seat no-limit Texas Hold'em against each other, ranked… | Agentic |
| B | VAREX-BenchVAREX measures how accurately multimodal foundation models extract structured JSON data from government forms… | Document Understanding |
| B | VerdictBenchVerdictBench measures how well models adjudicate e-commerce transaction disputes from multi-round, multimodal… | Agentic |
| B | VibeCodingBenchVibeCodingBench evaluates AI coding agents on production-oriented full-stack development tasks, scoring… | Coding |
| B | Video-HolmesVideo-Holmes measures multimodal LLMs' complex video reasoning by requiring them to locate and connect visual… | Multimodal |
| B | Video Understanding Reward ModelingVURB measures how well reward models judge preference pairs of responses to video understanding tasks,… | Video |
| B | VisualWebBenchVisualWebBench measures how well multimodal LLMs understand and ground web pages across seven tasks built from… | Coding |
| B | VitaBench 2.0VitaBench 2.0 measures how well LLM agents infer, use and update user preferences and act proactively across… | Agentic |
| B | VLMbench PerformanceVLMbench Performance measures inference throughput and latency (tokens/sec, TTFT, TPOT) of vision-language… | Efficiency |
| B | VoiceBenchVoiceBench measures LLM-based voice assistants on spoken instructions across knowledge, instruction following,… | Speech |
| B | VRBenchVRBench evaluates large models' multi-step reasoning over long narrative videos, scoring both final… | Multimodal |
| B | What Users Leave UnsaidHAERAE-Vision measures how well vision-language models answer real-world, under-specified visual questions… | Multimodal Understanding |
| B | When Seekers Are Hard to HelpEvaluates emotional support dialogue systems under worst-case interactions with hard-to-help simulated… | Reasoning |
| B | When Simulation LiesRobustBench-TC (from the paper When Simulation Lies) measures the robustness of tool-use language agents under… | Tool Use |
| B | WikiSQLWikiSQL measures how accurately models translate natural language questions into SQL queries over relational… | Knowledge |
| B | WikiVQABenchWikiVQABench measures how well vision-language models answer knowledge-grounded multiple-choice visual… | Multimodal |
| B | WildRoadBenchWildRoadBench measures how well vision-language models and autonomous LLM agents can localise road damage in… | Vision |
| B | WMT24++WMT24++ measures machine translation quality of MT systems and LLMs from English into 55 languages and… | Translation |
| B | Your One-Stop Solution for AI-Generated Video DetectionAIGVDBench measures how well detectors and vision-language models can distinguish AI-generated videos from… | Video |
| B | ZEROBench-SubZEROBench-Sub is a subset of ZeroBench, a difficult visual reasoning benchmark that tests large multimodal… | Multimodal |
| B | Avalon-ToM-BenchAvalon-ToM-Bench measures fine-grained Theory of Mind in language models through perspective-constrained… | |
| B | ClawTrackClawTrack evaluates LLM-based autonomous agents on 320 multi-step tasks across 8 domains, scoring both task… | |
| B | EASELEASEL measures dexterous visual tool use in multimodal agents, chiefly via reference-guided painting of a… | |
| B | Gestalt Grouping behavioral batteryA behavioral battery that scores vision models against human perception data on four Gestalt grouping tasks… | |
| B | MedPRESSMedPRESS measures whether LLMs abandon safe medical advice and sycophantically agree with patients under… | |
| B | MemeBenchMemeBench measures how well large vision-language models interpret culture-dependent Chinese and English… | |
| B | MMJailBenchMMJailBench measures the jailbreak vulnerability of multimodal large language models by systematically varying… | |
| B | quantum-api-driftMeasures whether LLMs generate quantum SDK (Qiskit) code that executes correctly on the specific requested SDK… | |
| B | RRS-10KRRS-10K measures how well vision-language models interpret rare, military-related remote sensing images across… | |
| B | From Sounds to ScenesMeasures whether large audio language models can understand auditory scenes by integrating speech, acoustic… | Audio |
| B | FINESSE-BenchFINESSE-Bench evaluates LLMs' financial domain knowledge and technical analysis across eight hierarchical… | Finance |
| B | Overview of the PsyDefDetect Shared Task at BioNLP 2026PsyDefDetect is a shared task that scores systems on classifying help-seeker utterances in emotional support… | Biomedicine |
| B | PhysicsMindPhysicsMind measures whether multimodal LLMs and video world models reason about and generate motion… | Science |
| C | Terminal-Bench 2.0 | Long-Horizon |
| C | SWE-Bench Pro v1 (2025-09-19) and v2SWE-Bench Pro measures whether AI agents can solve long-horizon software engineering tasks. | Coding Computer Use+2 |
| C | WebArenaWebArena measures whether autonomous language agents can complete diverse, long-horizon tasks on self-hosted,… | Agentic |
| C | MMLUMMLU measures a text model's multitask accuracy across 57 multiple-choice subjects, testing world knowledge… | Knowledge |
| C | Multi-SWE-BenchMulti-SWE-bench measures how well LLMs resolve real software issues by generating patches to codebases across… | Coding |
| C | GDP.xlsx | Enterprise Workflows Long-Horizon |
| C | AIME 2024AIME 2024 measures a model's ability to solve challenging competition mathematics problems from the 2024… | Math |
| C | MMLU-ReduxMMLU-Redux measures broad multitask knowledge of language models using 5,700 manually re-annotated MMLU… | Knowledge |
| C | FrontierMathFrontierMath tests AI models on unpublished, highly challenging mathematics problems ranging from… | Math |
| C | MAGIMAGI measures how well AI agents complete ambiguous, long-horizon enterprise work tasks that interleave GUI… | Computer Use |
| C | SWE-bench LiveSWE-bench Live measures how well LLMs and agents resolve real, recent GitHub issues by generating patches that… | Coding |
| C | SWE-bench FullSWE-bench Full measures whether AI systems can resolve 2,294 real GitHub issues from 12 popular Python… | Coding |
| C | DrugDiscoveryBench with PhyloDrugDiscoveryBench evaluates whether coding agents can assist with early-stage drug discovery tasks. | Coding Computer Use+2 |
| C | READY or Not enterprise agent deployment benchmarksREADY measures whether AI agents can be reliably deployed on enterprise workflows by jointly assessing… | Coding Computer Use+2 |
| C | RSI BenchRSI Bench evaluates whether AI agents can autonomously perform AI R&D tasks such as post-training, alignment,… | Coding Computer Use+2 |
| C | PRBenchPRBench measures high-stakes professional reasoning of language models in legal and finance domains using… | Coding Computer Use+2 |
| C | DROPDROP measures English reading comprehension requiring discrete reasoning (addition, counting, sorting) over… | Reasoning |
| C | BrowseComp Long Context 128kMeasures the ability of AI browsing agents to persistently navigate the internet and locate hard-to-find,… | Search |
| C | BrowseComp Long Context 256kBrowseComp Long Context 256k is a variant of BrowseComp, which measures how well AI agents can browse the web… | Search |
| C | CrafterCrafter is an open-world 2D survival game benchmark that scores agents by success rates on 22 achievements and… | Creative Media |
| C | FrontierMath Tier 4FrontierMath Tier 4 measures model accuracy on 43 exceptionally difficult, research-level mathematics problems… | Math |
| C | SWE-bench Verified MiniSWE-bench Verified Mini measures AI agents' ability to resolve real-world GitHub issues in Python… | Coding |
| C | TAU-bench AirlineTAU-bench Airline evaluates AI agents on airline customer-service tasks, such as changing or finding flights,… | Agentic |
| C | HumanEval-MulHumanEval-Mul measures functional correctness of code generated by models across multiple programming… | Coding |
| C | LiveCodeBench-PlusLiveCodeBench-Plus is a 91-problem coding benchmark of evolved and difficult original LiveCodeBench v6 tasks… | Coding |
| C | MMLU Medical GeneticsMMLU Medical Genetics is the medical genetics subject of the MMLU test, scoring models on multiple-choice… | Healthcare |
| C | MMLU-STEMMMLU-STEM is the STEM subset of MMLU, a multiple-choice test of a text model's multitask accuracy and… | General Knowledge |
| C | MultiMedia-TerminalBenchMMTB evaluates terminal agents on 105 tasks across 5 meta-categories that require directly operating on audio… | Agentic |
| C | OSWorld-MCPOSWorld-MCP measures computer-use agents' tool invocation via MCP, GUI operation, and decision-making on real… | Agentic |
| C | Westworld v1 simulated-web environments (2025-11-23): | Computer Use Enterprise Workflows |
| C | BrowserBench (2025-11-22): | Computer Use Enterprise Workflows |
| C | CyberStrikeCyberStrike measures whether frontier agents can complete realistic long-horizon cybersecurity work—defensive… | Coding Enterprise Workflows+1 |
| C | SciCode++SciCode++ measures models on computationally intensive scientific coding tasks across physics, materials… | Coding Enterprise Workflows+1 |
| C | Open-MM-RLOpen-MM-RL measures PhD-level multimodal STEM reasoning over figures, diagrams and graphs with single… | Coding Enterprise Workflows+1 |
| C | SDLCbenchSDLCbench evaluates models on long-horizon software-engineering work on private enterprise codebases. | Coding Enterprise Workflows+1 |
| C | CompanyBenchCompanyBench evaluates enterprise financial-analyst agents working on a simulated consumer-fintech neobank. | Coding Enterprise Workflows+1 |
| C | EKWbenchEKWbench measures models on multi-hour operational knowledge work such as sourcing reviews, cost roll-ups,… | Coding Enterprise Workflows+1 |
| C | Medical BenchMedical Bench measures longitudinal clinical reasoning over a single dermatology patient's chart read at nine… | Coding Enterprise Workflows+1 |
| C | OdysseysMegaOdysseysMega measures computer-use agents on long-horizon browser tasks on the live web, judged rubric by… | Coding Enterprise Workflows+1 |
| C | SWE-Bench Pro - Private DatasetSWE-Bench Pro Private Dataset measures AI agents' resolve rate on long-horizon software engineering tasks… | Coding Long-Horizon+1 |
| C | gba-eval-attempts and per-model gba-eval-attempt-* reposGBA Eval measures whether frontier AI coding agents can write a complete GBA emulator with WebAssembly support… | Coding Private Codebases |
| C | Legora BARLegora BAR (Benchmark for Agentic Reasoning) evaluates AI agents on end-to-end legal tasks drawn from real… | Coding Computer Use+1 |
| C | AutoResearchExamAutoResearchExam measures how quickly AI agents can improve and generalize hidden-test scores on 29 open-ended… | Long-Horizon |
| C | data-eng-bench | Long-Horizon |
| C | CWE-bench v1 | Coding Computer Use+2 |
| C | TraitBasis / Impatient Users Confuse AI AgentsTraitBasis/τ-Trait measures how robust conversational AI agents are when simulated users show traits like… | Coding Computer Use+2 |
| C | Vending-Bench 2 / Vending-Bench ArenaVending-Bench 2 measures how well AI models stay coherent while running a simulated vending machine business… | Computer Use Long-Horizon |
| C | Blueprint-Bench 2, 2026-05Blueprint-Bench 2 measures AI agents' spatial reasoning by having them convert ~20 apartment interior photos… | Computer Use Long-Horizon |
| C | Drone-Bench, with Anthropic Project Pilot, 2026-07Drone-Bench measures how well AI models can write code to perform a simple surveillance task (reconstruction,… | Computer Use Long-Horizon |
| C | ClawsBenchClawsBench measures the task success and safety of LLM productivity agents working across mock Gmail, Slack,… | Coding Computer Use+1 |
| C | Robo UseRobo Use measures how well coding agents solve robot benchmark tasks by driving a simulated robot from the… | Coding Computer Use+1 |
| C | FrontierPhysics v0.1FrontierPhysics evaluates AI agents on end-to-end frontier physics research tasks. | Coding Computer Use+1 |
| C | Benchmarking Agents in Insurance Underwriting EnvironmentsUNDERWRITE is an expert-designed, multi-turn insurance underwriting benchmark that evaluates AI agents using… | Coding Computer Use+2 |
| C | BigLaw Bench: ResearchBigLaw Bench: Research measures whether AI models can use search tools to research US case law and produce… | Coding Computer Use+2 |
| C | Measuring Autonomous AI ResearchMeasures how well frontier models can autonomously conduct AI research by running multi-day agent runs on the… | Coding Enterprise Workflows+2 |
| C | General Agent synthetic environmentA self-evolving synthetic agent environment of 4,504 stateful database tool-use tasks across 1,040 domains,… | Coding Enterprise Workflows+2 |
| C | Realm: Legal reasoning benchmarkRealm evaluates the legal reasoning ability of AI systems, scoring models on a Best@3 leaderboard. | Enterprise Workflows |
| C | Realm: Financial reasoning benchmarkRealm Financial measures how well AI agents complete 103 expert-authored real-world finance tasks (IFRS… | Enterprise Workflows |
| C | Democratizing Diplomacy harness paperAn evaluation harness that lets any LLM play full-press Diplomacy without fine-tuning, measuring strategic… | Long-Horizon |
| C | Humor / Bad Cards Arena (LOL Arena) leaderboardMeasures humor alignment: how often a model picks the same funniest card as the human judge across thousands… | Long-Horizon |
| C | DAYJOB (Healthcare/Finance) - arXiv:2610.01306DAYJOB measures whether AI agents can complete long-horizon professional healthcare and finance tasks in… | Enterprise Workflows Long-Horizon |
| C | ComplexConstraints and BeyondComplexConstraints is an expert-curated instruction-following benchmark of 75 public prompts scored against… | Enterprise Workflows Long-Horizon |
| C | EnterpriseBench CoreCraft - arXiv:2602.16179EnterpriseBench CoreCraft measures whether AI agents can perform multi-step, domain-specific customer support… | Enterprise Workflows Long-Horizon |
| C | The Hierarchy of Agentic Capabilities - arXiv:2601.09032Evaluates frontier AI models on 150 workplace tasks in a realistic e-commerce RL environment, analyzing… | Enterprise Workflows Long-Horizon |
| C | Antidote and Hemingway-bench leaderboards | Enterprise Workflows Long-Horizon |
| C | OfficeQAOfficeQA measures how well models and agents perform grounded reasoning by finding and combining figures from… | Multimodal |
| C | Westworld Finance Diligence Bench (2026-08-11): | Computer Use Enterprise Workflows |
| C | DealTrace Bench (2026-05/06): | Computer Use Enterprise Workflows |
| C | VAmoS Pro Bench / 'VAmoS Part Deux'VAmoS Pro Bench / 'VAmoS Part Deux' (arXiv 2609.38512, Sep 2026) - https://arxiv.org/abs/2609.38512 | Enterprise Workflows |
| C | VAmoS Bench: Voice Agent Simulation BenchVAmoS Bench: Voice Agent Simulation Bench (arXiv 2607.27453, Jul 2026) - https://arxiv.org/abs/2607.27453 | Enterprise Workflows |
| C | Public leaderboards VAmoS and VAmoS Pro | Enterprise Workflows |
| C | BankingBench | Enterprise Workflows |
| C | SantaBench | Enterprise Workflows |
| C | KellyBenchKellyBench measures how well language-model agents make long-horizon sequential decisions by betting on a… | Coding Long-Horizon |
| C | VADERVADER measures how well LLMs assess, detect, explain, and remediate real-world software vulnerabilities, with… | Coding Computer Use+1 |
| C | Blueprint-BenchBlueprint-Bench measures spatial reasoning of AI models by having them convert apartment photographs into… | Computer Use Long-Horizon |
| C | Butter-BenchButter-Bench measures the practical intelligence of LLMs acting as high-level controllers of robots in the… | Computer Use Long-Horizon |
| C | LibraryDesignBenchLibraryDesignBench measures how well coding agents design full-featured software libraries from… | Coding Computer Use+2 |
| C | SlopCodeBenchSlopCodeBench is a community benchmark measuring code erosion in AI-generated code. | Coding Computer Use+2 |
| C | SWE-Marathon-ExtSWE-Marathon-Ext measures whether AI models can build complete SaaS product clones from scratch given a… | Coding Enterprise Workflows+1 |
| C | Realm: Tax reasoning benchmarkRealm: Tax measures whether AI agents can prepare complete, accurate federal Form 1040 tax returns with… | Enterprise Workflows |
| C | CortexRetrievalBenchCortexRetrievalBench evaluates how well AI retrieval systems find and rank authoritative sources for financial… | Enterprise Workflows |
| C | LongExtractionBenchLongExtractionBench measures how accurately, completely and robustly document-extraction systems and frontier… | Enterprise Workflows |
| C | Diplomacy Arena leaderboardDiplomacy Arena scores models on multi-agent strategic negotiation by having each play 20 games of AI… | Long-Horizon |
| C | AI_Diplomacy open-source harnessAI_Diplomacy is an open-source harness in which AI models play full games of Diplomacy against each other,… | Long-Horizon |
| C | Chartography - arXiv:2608.10677Chartography measures whether frontier multimodal models can correctly read and reason over professional,… | Enterprise Workflows Long-Horizon |
| C | HANDBOOK.md - arXiv:2607.25398HANDBOOK.md measures whether LLM agents follow long, binding company policy documents (20-124 page SOPs) while… | Enterprise Workflows Long-Horizon |
| C | GDP.pdf - arXiv:2607.11192GDP.pdf measures whether multimodal models can answer realistic, expert-authored questions grounded in… | Enterprise Workflows Long-Horizon |
| C | Riemann-Bench - arXiv:2604.06802Riemann-Bench measures AI systems on private, expert-curated research-level mathematics problems with… | Enterprise Workflows Long-Horizon |
| C | Assemble Bench (robotics | Computer Use Enterprise Workflows |
| C | SheetBench-50 (with Sepal AI) - 2025-10-01SheetBench-50 evaluates agents on 50 financial analyst spreadsheet tasks performed in a browser-based… | Computer Use Enterprise Workflows |
| C | Cua-Bench EDA Leaderboard | Computer Use |
| C | Gemini 3.5 Flash computer-use evaluation on Cua-BenchEvaluates Gemini 3.5 Flash and other frontier models as computer-use agents on Cua-Bench's KiCad EDA suite of… | Computer Use |
| C | Windows Agent ArenaWindows Agent Arena measures how well multi-modal agents complete 150+ realistic tasks by operating… | Computer Use |
| C | IPI ArenaIPI Arena measures how vulnerable LLM agents are to concealed indirect prompt injection attacks across tool… | |
| C | Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration TestingEvaluates AI agents against human cybersecurity professionals on penetration testing of a live university… | |
| C | D-REX deceptive reasoning benchmarkD-REX evaluates whether models and safety mechanisms can detect deceptive reasoning, where a model's internal… | |
| C | Agent Red Teaming (ART) benchmarkART measures how robust LLM-powered AI agents are against curated high-impact prompt-injection attacks that… | |
| C | NIST CAISI blog on the UK AISI x Gray Swan agent red-teaming competitionA large-scale public agent red-teaming competition measuring how often attacks (direct and indirect prompt… | |
| C | BlenderBench | Coding Computer Use+1 |
| C | InsureBench: insurance underwriting and claims benchmark | Coding Computer Use+3 |
| C | CorpLaw (measured 2026-08-18; access on request): | Coding Enterprise Workflows+1 |
| C | AUDITSAUDITS measures how robustly image manipulation detection and localization methods detect diffusion-based… | Vision |
| C | ConvertedCoding agents increasingly act as codebase-scale collaborators that can assist with codebase conversion, but… | Coding |
| C | EnterpriseBenchEnterpriseBench measures how well LLM agents complete 500 tasks in a simulated enterprise environment spanning… | Agentic Business |
| C | Frontier-BenchFrontier-Bench (presented as Terminal-Bench) measures frontier AI agents on a diverse, difficult, evolving set… | Agentic |
| C | FrontierFinanceFrontierFinance measures how well AI systems answer 220 open-ended financial research queries across the… | Finance |
| C | ActBenchActBench measures behavioral safety of cowork agents by scoring execution trajectories on matched benign and… | |
| C | CoREBCoREB measures code retrieval and reranking models across text-to-code, code-to-text, and code-to-code search… | Retrieval |
| C | FinQAFinQA measures models' ability to answer expert-written numerical reasoning questions over financial reports… | Finance |
| C | FRAMESFRAMES measures LLMs' ability in end-to-end retrieval-augmented generation: factual responses, retrieval, and… | Search |
| C | IndicaIndica measures LLMs' knowledge of region-specific cultural commonsense across five Indian regions and their… | General Knowledge |
| C | MOREMORE measures how well OCR and document parsing models recover text, formulas, tables, code blocks, catalogs… | Document Understanding |
| C | NESTNEST measures how well vision-language models understand narrative event structure in full-length movies, via… | Video |
| C | PARSEPARSE measures reasoning-focused open-domain question answering by LLMs in Persian across 10,800 Boolean,… | Reasoning |
| C | PlantLarge Language Model (LLM) agents remain vulnerable to safety threats from the external environment, where… | Safety |
| C | PortBenchPortBench measures how well LLMs perform correlation-aware portfolio management across six asset classes, via… | Finance |
| C | PRIMEPRIME evaluates how LLMs resolve conflicting meta-instructions across response length, output format, and… | Instruction Following |
| C | REALREAL evaluates LLMs' memory, comprehension, reasoning and hallucination in housing transactions and real… | Business |
| C | ROSEROSE measures whether multimodal LLMs can turn fixed visual evidence into context-specific actions via coupled… | Reasoning |
| C | SODESODE evaluates the social dynamics of LLM agents in behavioral game theory settings: direct reciprocity,… | Behavioral |
| C | STAGESTAGE measures models' narrative understanding over full-length movie screenplays via knowledge graph… | Long Context |
| C | TASTETASTE is a designer-annotated multi-dimensional preference dataset for AI-generated graphic design, used to… | Creative |
| C | RFT case study with Savi Security, Qwen3 4B scam detection | Enterprise Workflows |
| C | DDBenchDDBench measures how well LLM coding agents repair historical bugs in open-source distributed systems, with… | |
| C | FinReportBenchFinReportBench measures whether LLMs can generate institution-grade bilingual financial research reports,… | |
| C | ToolFailBenchToolFailBench measures tool-use failure modes of LLM agents (tool-skip, result-ignore, output fabrication,… | |
| C | TREKTREK measures whether tool-using LLM agents can synthesize feasible travel itineraries that are… | |
| C | CybenchCybench measures language model agents' cybersecurity capabilities on 40 professional-level Capture the Flag… | Agentic |
| C | ExploitGymExploitGym measures whether AI agents can extend vulnerability-triggering inputs into working exploits across… | Agentic |
| C | EdgeBenchEdgeBench measures how autonomous AI agents learn from real-world environment feedback over 12+ hour runs… | Agentic |
| C | ExploitBenchMeasures how far AI agents progress along the Chromium V8 exploitation ladder, scoring 16 exploit capability… | Agentic |
| C | CorpusQA 1MCorpusQA 1M measures a model's accuracy at answering questions over roughly million-token long-context… | Reasoning |
| C | CountBenchCountBench is an image-text benchmark that evaluates whether vision-language models understand object counts… | Multimodal |
| C | Design2CodeDesign2Code measures how well multimodal LLMs convert screenshots of real-world webpages into front-end code… | Multimodal |
| C | ContractNLIContractNLI measures document-level natural language inference on contracts: classifying whether hypotheses… | Decision Models |
| C | BelebeleBelebele measures multilingual machine reading comprehension via four-option multiple-choice questions on… | Decision Models |
| C | WebVoyagerWebVoyager measures how well multimodal web agents complete open-ended user tasks end-to-end on 15 popular… | Agentic |
| C | ZClawBenchZClawBench measures how well models perform as general-purpose agents completing end-to-end OpenClaw-style… | Agentic |
| C | SWE-sweepSWE-sweep measures whether coding agents can autonomously discover and fix as many unspecified bugs as… | Agentic |
| C | InferenceBenchInferenceBench measures whether autonomous coding agents can build and optimize an OpenAI-compatible LLM… | Agentic |
| C | MobileWorldMobileWorld measures how well autonomous mobile agents complete long-horizon, cross-application tasks on… | Agentic |
| C | GAIAGAIA measures how well AI assistants answer real-world questions that are simple for humans but require… | Agentic |
| C | Time Horizon Index: KSPMeasures how far an AI agent gets through a fixed ladder of 30 Kerbal Space Program missions when given five… | Agentic |
| C | SRE BenchSRE-Bench measures how well AI agents reverse engineer compiled binaries of private, real-world-scale programs… | Agentic |
| C | Public Benefits Bench v1Public Benefits Bench v1 measures how accurately AI models answer SNAP benefits navigation questions,… | Agentic |
| C | DeepResearch Bench IIDeepResearch Bench II evaluates deep research systems on producing long-form research reports from web… | Agentic |
| C | FLTEvalFLTEval scores models on Lean 4 formal proof tasks derived from the Fermat's Last Theorem project, applying… | Coding |
| C | VIBE-ProVIBE-Pro is a repo-level code generation benchmark that scores models on end-to-end full project delivery… | Coding |
| C | KernelBenchKernelBench measures how well language models write correct, fast GPU kernels (CUDA/Triton), scoring speedup… | Coding |
| C | MRCRv2MRCR v2 measures long-context reasoning by asking models to reproduce a specific instance of assistant output… | Reasoning |
| C | Graphwalks Parents 128KGraphwalks Parents measures whether a language model can, given a long list of directed graph edges in… | Reasoning |
| C | MRCR 1MMRCR 1M measures a model's ability to retrieve and resolve multi-round coreferences across contexts of about… | Reasoning |
| C | ODINW13A visual detection and grounding benchmark slice used to compare zero-shot object understanding across diverse… | Multimodal |
| C | VideoMMMUVideo-MMMU measures large multimodal models' ability to acquire and apply knowledge from expert-level… | Multimodal |
| C | Vision2WebVision2Web measures whether multimodal coding agents can build static webpages, interactive frontends and… | Multimodal |
| C | SimpleVQASimpleVQA measures the factuality of multimodal large language models answering short natural-language… | Multimodal |
| C | AudioMC – Audio OutputAudioMultiChallenge Audio Output evaluates spoken dialogue systems producing native speech on multi-turn… | Multimodal |
| C | ChartQAChartQA measures models' ability to answer complex questions about charts requiring visual and logical… | Multimodal |
| C | InfoVQAInfographicVQA measures visual question answering over infographic images, requiring joint reasoning over… | Multimodal |
| C | TextVQATextVQA measures whether models can read and reason about text in images to answer visual questions. | Multimodal |
| C | AGIEvalAGIEval measures foundation models on human-centric standardized exams such as college entrance exams, law… | Knowledge |
| C | TruthfulQATruthfulQA measures whether language models answer 817 questions across 38 categories truthfully rather than… | Knowledge |
| C | TriviaQATriviaQA measures reading comprehension and question answering on trivia questions authored by enthusiasts,… | Knowledge |
| C | NOVA-63NOVA-63 is a multilingual benchmark evaluating LLM capability across 63 languages. | Multilingual |
| C | PolyMathPolyMATH evaluates multi-modal LLMs' cognitive and mathematical reasoning on 5,000 image-based textual and… | Multilingual |
| C | MAXIFEMaXIFE measures LLMs' instruction-following capabilities in multilingual and cross-lingual settings across 23… | Multilingual |
| C | TabFactTabFact measures whether models can verify if natural-language statements are entailed or refuted by Wikipedia… | Decision Models |
| C | HANDBOOK.md AgentsHANDBOOK.md Agents measures whether agents can follow a long (100-page) company handbook while completing… | Enterprise Workflows Tool-use+1 |
| C | A^3-BenchA3-Bench measures memory-driven scientific reasoning in LLMs, scoring accuracy and anchor/attractor memory… | Science |
| C | A2RBenchA2RBench measures LLMs' abstract reasoning ability, their capacity to extract and apply abstract rules, using… | Reasoning |
| C | ABC-BenchABC-Bench measures whether coding agents can explore real repositories, edit code, configure environments,… | Coding |
| C | AcademiClaw: When Students Set Challenges for AI AgentsAcademiClaw measures how well AI agents complete 80 complex, long-horizon, bilingual academic tasks sourced… | Agentic |
| C | ActiveSciBenchActiveSciBench measures closed-loop scientific discovery with active, budget-constrained experiment selection,… | Science |
| C | AcuityBenchAcuityBench measures whether language models identify the appropriate urgency of medical care from user health… | Healthcare |
| C | AEQ-BenchAEQ-Bench measures the empathy of omni-modal large models, both generating empathetic responses from audio and… | Audio |
| C | AfriEconQAAfriEconQA measures how well LLM and RAG systems answer complex African economic questions requiring numerical… | Finance |
| C | AfriScience-MTAfriScience-MT measures machine translation quality of scientific plain-language summaries from English into… | Translation |
| C | Afrispeech SemanticsAfrispeech Semantics measures how well audio language models perform semantic and paralinguistic reasoning… | Audio |
| C | AfroBenchAfroBench measures how well large language models perform across 64 African languages on 15 tasks spanning… | Multilingual |
| C | AgencyBenchAgencyBench measures autonomous LLM agents on 138 long-horizon real-world tasks across 32 scenarios spanning… | Agentic |
| C | Agent Security LeagueAgent Security League measures whether AI coding agent and model combinations produce functionally correct and… | Coding |
| C | Agent-ValueBenchAgent-ValueBench evaluates the values steering autonomous agents' behavior through value-conflict tasks in… | Alignment |
| C | AgentBoardAgentBoard evaluates LLMs as generalist agents across 9 diverse multi-turn, partially-observable tasks,… | Agentic |
| C | AgentClinicAgentClinic measures how well LLM agents perform sequential diagnostic decision-making in simulated multimodal… | Healthcare |
| C | AgentHijackAgentHijack measures how robust multimodal computer-use agents are on desktop tasks when exposed to nine… | Agentic |
| C | AgentickAgentick measures sequential decision-making agents (RL, LLM, VLM, hybrid, human) on 37 procedurally generated… | Agentic |
| C | AgenticVBenchAgenticVBench measures how well multimodal AI agents complete 100 expert-authored real-world video… | Domain |
| C | AgentLeakAgentLeak measures privacy leakage of sensitive data across internal channels (inter-agent messages, shared… | Safety |
| C | Agentset Reranker LeaderboardAgentset's Reranker Leaderboard evaluates reranking models in a RAG pipeline across six retrieval datasets… | Retrieval |
| C | AgingBenchAgingBench measures how the reliability of deployed, memory-based AI agents degrades over many sessions across… | Agentic |
| C | AgroBenchAgroBench evaluates vision-language models on seven expert-annotated agricultural question-answering tasks… | Science |
| C | AgroVGAgroVG measures how well vision-language models localize agricultural targets (crops, weeds, fruit, pests,… | Vision |
| C | AI Snitches Get GlitchesMeasures whether LLM agents covertly monitor and report users (agentic surveillance) across over 300… | Agentic |
| C | AI2DAI2D measures diagram understanding through multiple-choice question answering over grade-school science… | Knowledge |
| C | Aider Refactoring BenchmarkMeasures an LLM's ability to refactor 89 large methods from large Python classes, outputting long code chunks… | Coding |
| C | AIM-BenchAIM-Bench measures the inventory decision-making quality and behavioral biases of LLM agents acting as… | Agentic Business |
| C | AIR-BenchAIR-Bench evaluates large audio-language models on understanding speech, natural sounds and music via… | Knowledge |
| C | AlexandriaAlexandria measures how well machine translation systems and LLMs translate between English and 13 regional… | Translation |
| C | Ambig-DSAmbig-DS measures whether data-science agents recognize and handle underspecified task framings (ambiguous… | Agentic |
| C | Android Control High_EMMeasures how accurately agents complete high-level natural-language tasks in Android apps, scored by exact… | Agentic |
| C | Android Control Low_EMAndroid Control Low_EM measures exact-match accuracy of UI control agents executing low-level human… | Agentic |
| C | Animation2CodeAnimation2Code measures whether vision-language models can reconstruct executable HTML/CSS/JavaScript web… | Coding |
| C | ANLIANLI measures natural language understanding via adversarially collected natural language inference… | Language |
| C | AppellateGenAppellateGen measures how well models draft second-instance (appellate) legal judgments by reasoning over… | Legal |
| C | ARC ChallengeARC Challenge measures models' ability to answer difficult grade-school multiple-choice science questions that… | Knowledge |
| C | ARC EasyARC Easy measures a model's ability to answer grade-school multiple-choice science questions, forming the… | Knowledge |
| C | ArchBench Serverless Component GenerationArchBench's serverless component generation task measures how well LLMs and coding agents generate serverless… | Coding |
| C | Are LLM Decisions Faithful to Verbal Confidence?RiskEval evaluates whether LLMs adjust their abstention decisions in line with their verbal confidence under… | Safety |
| C | ArtifactsBenchArtifactsBench measures LLMs' ability to generate visual and interactive code artifacts, scoring rendered… | Coding |
| C | Artificial Analysis Speech-to-TextA leaderboard measuring speech-to-text transcription accuracy (word error rate), speed factor and price across… | Audio |
| C | ASPIASPI measures how much more vulnerable LLM agents are to prompt injection attacks when they enter a… | Safety |
| C | ASR-FairBenchASR-FairBench is a leaderboard that scores automatic speech recognition models on accuracy (WER) and fairness… | Audio |
| C | AssertLLM2AssertLLM2 measures how well LLMs generate formal SystemVerilog Assertions from hardware design… | Coding |
| C | AstroMindAstroMind measures how well LLMs reason about spacecraft behavior, covering intent inference, maneuver… | Aerospace |
| C | AtelierEvalAtelierEval measures the prompting proficiency of humans and multimodal LLMs as text-to-image prompters across… | Multimodal |
| C | ATOM-BenchATOM-Bench evaluates real-world robot manipulation policies on atomic motor and instruction skills and on… | Robotics |
| C | AudioAgentBenchAudioAgentBench measures voice agent models on six multi-turn spoken workflows (appointment, assistant,… | Speech |
| C | AudioBenchAudioBench evaluates audio large language models on speech, sound, music, ASR, audio QA, translation, and… | Knowledge |
| C | AutoEval-VideoAutoEval-Video measures how well large vision-language models answer open-ended questions about videos across… | Multimodal |
| C | AutoMedBenchAutoMedBench measures how well autonomous coding agents run medical AI research pipelines (plan, setup,… | Healthcare |
| C | AVBenchAVBench measures the quality of human-centric audio-video generative models across ten dimensions covering… | Creative Media |
| C | Bayesian Adaptation GymBayesian Adaptation Gym measures how well Bayesian low-rank adaptation methods for multi-modal language models… | Machine Learning |
| C | BCoughBenchBCoughBench evaluates respiratory acoustic foundation models on cough-based disease, demographic and age tasks… | Audio Healthcare |
| C | Benchmarking Autonomous Agents against TemporalA3S-Bench measures how vulnerable LLM-based autonomous agents are to temporal, spatial, and semantic evasion… | Agentic |
| C | Benchmarking Robot Memory Under InterferenceMeasures how well robot memory systems (memory-augmented VLAs) retain relevant information across sessions as… | Robotics |
| C | Beyond ContinuityMeasures whether LLMs detect topic pivots versus refinements in multi-turn dialogue and shortlist relevant… | Reasoning |
| C | Beyond the All-in-One AgentEntCollabBench measures how well role-specialized LLM agents collaborate in a simulated permission-isolated… | Agentic Business |
| C | Beyond Value BenchmarksMeasures how closely LLMs' value structures align with human reference structures by having models and humans… | Alignment |
| C | BFCL v2BFCL v2 measures how accurately large language models call functions (tools), using live enterprise and… | Tool Use |
| C | BIM-EditBIM-Edit measures how well LLMs perform natural-language editing of IFC-format Building Information Models,… | Document Understanding |
| C | BioCoderBioCoder measures how well large language models generate bioinformatics-specific code in Python, Java and… | Biology |
| C | BioMedHopBioMedHop measures LLM biomedical multi-hop reasoning over multi-source evidence (knowledge graphs, documents,… | Biomedicine |
| C | Biopharma BenchBiopharma Bench measures whether AI agents can complete pharmaceutical regulatory and CMC assignments inside… | Biomedicine Agentic+1 |
| C | BioSecBench-FunctionIt measures whether AI agents can analyze real experimental data (biophysical assays, structures, sequences)… | Biology Safety |
| C | BioSecBench-SurveillanceBioSecBench-Surveillance measures whether AI agents can turn raw pathogen sequencing data into correct genomic… | Biology Agentic+1 |
| C | BlueFin: Benchmarking LLM Agents on Financial SpreadsheetsBlueFin measures how well LLM agents perform synthesis, manipulation, and comprehension tasks over spreadsheet… | Finance Business |
| C | Boiling the FrogBoiling the Frog measures whether tool-using AI agents in corporate and office settings can be led into unsafe… | Agentic |
| C | BrowserARTBrowserART measures whether LLM-based browser agents attempt 100 harmful browser-related behaviors on… | Safety |
| C | CadEvalCadEval scores models on CAD and technical code generation tasks. | Coding |
| C | CalBenchCalBench measures how well multiple LLM agents coordinate calendar scheduling with private calendars, scoring… | Agentic |
| C | CaseHOLDCaseHOLD measures legal reasoning by asking models to pick the correct holding of a cited case from five… | Legal |
| C | CaseReportBenchCaseReportBench measures how well LLMs perform dense information extraction from clinical case reports on… | Healthcare |
| C | Cattle TradeCattle Trade measures LLM agents' strategic reasoning in a long multi-agent economic card game combining… | Game |
| C | CCBenchCCBench measures how well AI coding agents implement features in small, private, human-written codebases not… | Coding |
| C | CEO-Bench: Can Agents Play the Long Game?CEO-Bench measures whether language model agents can sustain long-horizon, adaptive decision-making by… | Agentic Business |
| C | ChaosBenchChaosBench evaluates ML weather emulators on subseasonal-to-seasonal climate prediction using reanalysis data… | Climate |
| C | ChartFIChartFI measures how faithfully and insightfully multimodal large language models describe complex charts,… | Document Understanding |
| C | Checkup2ActionCheckup2Action measures how well language models generate safe, prioritised, patient-oriented structured… | Healthcare |
| C | Chess-World-ModelChess-World-Model measures whether models can track state exactly, predicting the board state reached after a… | Game |
| C | CheXperceptCheXpercept measures expert-level lesion perception of vision-language models on chest X-rays across coarse… | Biomedicine Healthcare |
| C | ChildAgentEvalChildAgentEval is a WISC-inspired interactive benchmark that compares the reasoning of multimodal LLM-based… | Agentic |
| C | ChLogicChLogic measures whether language models preserve logical reasoning performance when the same logical… | Reasoning |
| C | ChronoMedKGChronoMedKG is a temporal biomedical knowledge graph released with the ChronoTQA benchmark, which scores LLMs… | Biomedicine Healthcare |
| C | CIPHERCIPHER measures how often LLM-generated Python code contains cryptographic vulnerabilities under insecure,… | Cybersecurity |
| C | Cited but Not VerifiedEvaluates the source attribution of LLM deep research agents by parsing inline citations in generated reports… | Factuality |
| C | ClawForgeClawForge measures how well command-line agents complete executable, multi-step workflows over persistent… | Agentic |
| C | ClinConsensusClinConsensus measures how well LLMs cover physician-authored, case-specific rubric criteria in open-ended… | Healthcare |
| C | Clotho-AQAClotho-AQA measures audio question answering: answering natural language questions about environmental audio… | Knowledge |
| C | CLQTCLQT is a closed-loop, cost-aware trading environment that diagnoses LLM portfolio-management agents'… | Finance |
| C | CoffeeBenchCoffeeBench measures how much net income an LLM agent can generate as a coffee roaster over a 90-day simulated… | Business |
| C | CogScaleCogScale is a benchmark of 14 scalable synthetic tasks that evaluates the cognitive and memory abilities of… | Reasoning |
| C | CogSense-BenchCogSense-Bench is a visual question answering benchmark that scores multimodal LLMs on five cognitive… | Multimodal Understanding |
| C | CommonsenseQACommonsenseQA measures how accurately models answer multiple-choice questions that require commonsense world… | Knowledge |
| C | CommonWhyCommonWhy measures LLMs' entity-based causal commonsense reasoning by having them answer 15,000 why questions… | Reasoning |
| C | Composite-BenchComposite-Bench measures whether models can complete long-horizon enterprise workflows in deterministic… | Agentic Business |
| C | Confirming CorrectEffective tutoring requires distinguishing optimal, valid but suboptimal, and incorrect student solutions, a… | Education |
| C | ConsumerSimBenchConsumerSimBench measures whether LLMs can reconstruct the concrete reaction patterns real consumers express… | Behavioral |
| C | Contextual StereoSetContextual StereoSet measures how robust LLM stereotype-bias evaluations are by holding stereotype content… | Safety |
| C | CooperBenchCooperBench measures how well pairs of coding agents coordinate to implement separate, potentially conflicting… | Agentic |
| C | Counter Turing TestCounter Turing Test (CT2) is a shared-task benchmark that scores systems on detecting AI-generated text… | Factuality |
| C | CRMArenaCRMArena evaluates LLM agents on realistic customer relationship management tasks in a sandboxed Salesforce… | Agentic Business |
| C | CrossCult-KIBenchCrossCult-KIBench measures how well multimodal LLMs can have cultural knowledge inserted for a target culture… | Multimodal Understanding |
| C | CrystalXRD-BenchCrystalXRD-Bench measures whether vision-language models can read powder XRD pattern images and identify the… | Materials |
| C | CUADCUAD measures how well models highlight salient clauses in commercial legal contracts for contract review,… | Legal |
| C | CUDABeaver: Benchmarking LLM-Based Automated CUDA DebuggingCUDABeaver measures whether LLMs can truly repair failing CUDA code from real failing workspaces while… | Coding |
| C | Cultural Fidelity in English-to-Hindi TranslationMeasures whether English-to-Hindi translation systems preserve the recoverability of explicitly encoded gender… | Translation |
| C | CulturaQACulturaQA is an LRM-generated, human-curated Greek question answering dataset used to train and evaluate LLMs… | Question Answering |
| C | CutVerseCutVerse measures how well autonomous GUI agents complete long-horizon media post-production editing tasks… | GUI |
| C | CyberSecEvalCyberSecEval measures LLM cybersecurity risks, including propensity to generate insecure code and compliance… | Generalization |
| C | DBESDBES is a diagnostic benchmark and metric suite that measures expert specialization in Mixture-of-Experts… | Machine Learning |
| C | DEEPSYNTHDEEPSYNTH measures how well LLM agents gather information from multiple web sources, synthesize it, and reason… | Agentic |
| C | DeluluDelulu measures whether code LLMs produce hallucinated completions (invented methods, invalid parameters,… | Coding |
| C | DevBenchDevBench evaluates LLMs across the full software development lifecycle: software design, environment setup,… | Coding |
| C | DexHoldemDexHoldem is a real-world benchmark evaluating dexterous ShadowHand manipulation policies, agentic perception… | Embodied |
| C | DiagFlowBenchDiagFlowBench measures how language models grounded in industrial diagnostic flowcharts recognise and handle… | Instruction Following |
| C | DialDeferDialDefer measures dialogic deference: how LLM judges shift verdicts on identical claims when framed as a… | Behavioral |
| C | DiffSpotDiffSpot measures whether vision-language models can spot and describe fine-grained visual differences between… | Vision |
| C | DiscoverPhysicsDiscoverPhysics measures whether LLM agents can discover the altered laws of motion of simulated N-body worlds… | Reasoning |
| C | Do Coding Agents Understand Least-Privilege Authorization?AuthBench measures whether coding agents can infer least-privilege file-level read/write/execute permission… | Agentic |
| C | DocOSDocOS measures whether GUI agents can autonomously search a web browser for online documentation, understand… | GUI |
| C | DR-ArenaDR-Arena is an automated, dynamic evaluation framework that ranks deep research agents via Elo battles on… | Agentic |
| C | Drift-BenchDrift-Bench measures how LLM agents handle faulty user inputs (implicit intent, missing parameters, false… | Agentic |
| C | Drive-P2DDrive-P2D evaluates vision-language models on autonomous driving through 6,650 questions that progress from… | Embodied |
| C | DriveBenchDriveBench evaluates the reliability and visual grounding of vision-language models on autonomous driving… | Embodied |
| C | DSAEvalDSAEval measures how well LLM-based data science agents solve 641 real-world data science problems over… | Data |
| C | Dynamic-SUPERBDynamic-SUPERB is a leaderboard benchmark that scores speech and audio language models on… | Speech |
| C | EgoPro-BenchEgoPro-Bench evaluates multimodal LLMs on personalized proactive interaction in streaming egocentric videos,… | Video |
| C | EHR-ComplexEHR-Complex measures how well medical agents perform interactive clinical database reasoning over MIMIC-IV… | Biomedicine Healthcare |
| C | EmoSEmoS measures how well multimodal language models recognize fine-grained emotions in bilingual (Chinese and… | Multimodal Understanding |
| C | Enhancing Pathological VLMs with Cross-scale ReasoningScale-VQA measures pathology vision-language models' cross-scale reasoning via 4,685 multiple-choice questions… | Reasoning Healthcare |
| C | EpiBenchEpiBench evaluates LLMs on five epilepsy clinical reasoning tasks: clinical decision QA, EEG report… | Healthcare |
| C | EQUITRIAGEEQUITRIAGE audits gender bias in LLM-based emergency department triage by measuring how often models change… | Healthcare |
| C | ERP-BenchERP-Bench measures how well AI agents complete 300 long-horizon procurement and manufacturing workflow tasks… | Agentic Business |
| C | EUDAIMONIA: Evaluating Undesirable Dynamics in AIEUDAIMONIA measures whether LLMs align with user welfare in social interactions, checking for violations such… | Safety |
| C | EVA-BenchEVA-Bench evaluates voice agents in simulated spoken multi-turn conversations across enterprise domains,… | Agentic Business |
| C | EvoCode-BenchEvoCode-Bench measures whether coding agents can keep their own codebase working across multi-round evolving… | Software Engineering |
| C | Faithfulness Metrics Don't Measure FaithfulnessBonaFide measures how well chain-of-thought faithfulness metrics detect unfaithful reasoning, using 3,066 CoTs… | Alignment |
| C | FBHMFBHM measures vision-language models' ability to detect hateful memes across 25 rhetorical functionalities and… | Safety |
| C | FinBenFinBen is an evaluation benchmark that scores large language models on financial understanding and prediction… | Finance |
| C | FineBenchFineBench measures how well vision-language models understand fine-grained human activity in long-form videos,… | Video |
| C | FORTIS: Benchmarking Over-Privilege in Agent SkillsFORTIS measures over-privilege in LLM agent skills: whether models select the minimally sufficient skill from… | Safety |
| C | FraudSMSWalkerFraudSMSWalker measures whether web agents can accurately and evidence-groundedly judge fraud in bilingual… | Safety |
| C | From Laboratory to Real-World ApplicationsRepoReason measures LLM agents' repository-level code reasoning via white-box abductive assertion verification… | Software Engineering |
| C | From Scenes to ElementsGranuVistaVQA, introduced in the paper From Scenes to Elements, measures how well multimodal RAG systems… | Multimodal Understanding |
| C | FrontierChallengeFrontierChallenge measures whether AI agents can complete end-to-end scientific workflows across six domains… | Agentic Science |
| C | FullHomeFullHome evaluates whether household agents can infer executable task structure from a complete household… | Robotics |
| C | GameCraft-BenchGameCraft-Bench measures whether coding agents can turn natural-language game specifications into complete,… | Game |
| C | GavelGavel evaluates LLMs on long-context multi-document legal case summarization (32K-512K tokens) using a 26-item… | Legal |
| C | GitTaskBenchGitTaskBench measures how well code agents solve 54 real-world tasks by leveraging GitHub code repositories,… | Coding |
| C | Global PIQAGlobal PIQA evaluates culturally specific physical commonsense reasoning of language models across over 100… | Knowledge |
| C | GPTNTGPTNT measures real-time, asymmetric collaboration between multimodal agents who must communicate to defuse… | Agentic |
| C | Hack-Verifiable TextArenaHack-Verifiable TextArena measures whether and how language model agents exploit reward hacking opportunities… | Agentic |
| C | HarmActionsEvalHarmActionsEval measures how often LLM agents avoid executing harmful tool actions under adversarial pressure,… | Safety |
| C | Hex DataBenchHex DataBench measures how well AI agents perform realistic, often vague agentic analytics tasks in a messy… | Agentic Data+1 |
| C | HiMed: Incentivizing Hindi Reasoning in Medical LLMsHiMed is a Hindi medical reasoning benchmark suite evaluating LLMs on Western and Indian systems of medicine… | Healthcare |
| C | HLV-1KHLV-1K measures how well multimodal models understand hour-long videos through time-specific QA and… | Multimodal |
| C | ITSMBenchITSMBench measures whether AI agents can resolve real enterprise IT service management tickets end-to-end… | Agentic |
| C | LexGLUELexGLUE measures model performance on seven English legal language understanding tasks, including court case… | Legal |
| C | LLM-AggreFactLLM-AggreFact measures how well models detect grounded factuality errors (hallucinations) by checking claims… | Safety |
| C | LLMEval-LogicLLMEval-Logic measures LLMs' natural-language logical reasoning on Chinese propositional and first-order logic… | Reasoning |
| C | Look-Ahead-BenchLook-Ahead-Bench measures look-ahead bias in LLMs used in practical financial trading workflows by comparing… | Finance |
| C | MAS-PromptBenchMAS-PromptBench measures how much system-prompt optimization improves multi-agent LLM systems across tasks… | Uncategorized |
| C | MedAgentBenchMedAgentBench measures how well LLM agents complete 300 physician-written clinical tasks by interacting with a… | Healthcare |
| C | MedHELMMedHELM evaluates large language models on 121 clinical tasks across 35 benchmarks spanning healthcare… | Healthcare |
| C | Minerva-EgoMinerva-Ego measures complex multi-step egocentric video reasoning via multiple choice questions over HD-EPIC… | Video |
| C | MiniWoB++MiniWoB++ is a collection of over 100 web interaction environments testing agents' ability to complete simple… | Agentic |
| C | MLAgentBenchMLAgentBench measures how well language agents can autonomously perform machine learning experimentation,… | Agentic |
| C | MMBenchMMBench evaluates multimodal (vision-language) models on image understanding and cross-modal reasoning, with… | Knowledge |
| C | MMErroRMMErroR measures whether vision-language models can detect erroneous reasoning chains and classify the error… | Multimodal Understanding |
| C | MMStarMMStar measures vision-indispensable multimodal capabilities of large vision-language models across 1,500… | Multimodal |
| C | MultiMedQAMultiMedQA measures large language models' medical question answering across professional medical exams,… | Healthcare |
| C | NeedleBenchNeedleBench measures LLM retrieval and reasoning in bilingual long-context tasks with key information embedded… | Generalization |
| C | ODinWODinW measures how well object detection models transfer zero-shot and few-shot to 13 diverse real-world… | Vision |
| C | OdysseyBenchOdysseyBench measures how well LLM agents complete long-horizon office workflows across Word, Excel, PDF,… | Agentic Business |
| C | Omi SOAP Note Safety BenchmarkMeasures how safely models generate clinical SOAP notes from doctor-patient dialogues, scoring unsupported… | Healthcare |
| C | OpenEarthMapOpenEarthMap is a benchmark for global sub-meter level land cover mapping (semantic segmentation of… | Geospatial |
| C | OpenHuEvalOpenHuEval measures how well large language models handle Hungarian language and Hungary-specific knowledge… | Language |
| C | PaintBenchPaintBench measures how precisely multimodal image editing models perform single-answer visual edits across 20… | Multimodal |
| C | PassBenchPassBench measures how well LLMs generate graph compiler passes that speed up long-tail fusible subgraphs,… | Coding |
| C | PeerRankPeerRank is an autonomous LLM evaluation framework where models generate questions, answer them with web… | Meta |
| C | PersonalAlignPersonalAlign (evaluated via its AndroidIntent benchmark) measures how well GUI agents resolve vague mobile… | GUI |
| C | PhyGroundPhyGround measures whether video generation models produce videos that follow 13 physical laws across… | Video |
| C | PhysicianBenchPhysicianBench measures how well LLM agents complete long-horizon physician tasks in a FHIR-based EHR… | Healthcare |
| C | PRISMPRISM is a name shared by two benchmarks: one scores language models on generating spatially correct… | Reasoning |
| C | PRiSM: Benchmarking Phone Realization in Speech ModelsPRiSM benchmarks phone recognition systems and speech models on phonetic perception via intrinsic… | Speech |
| C | PrivacyGLUEPrivacyGLUE measures language models' general language understanding of privacy policies across 7… | Legal |
| C | Progress-BenchProgress-Bench measures whether vision-language models can estimate how far a robot manipulation task has… | Multimodal Understanding |
| C | QASCQASC measures multi-hop question answering by composing facts to answer 8-way multiple-choice grade school… | Knowledge |
| C | QFBenchQFBench measures whether AI agents can write and execute correct quantitative finance code (derivatives… | Finance |
| C | R2MEDR2MED measures how well retrieval systems find medical evidence that requires reasoning, across Q&A reference… | Healthcare |
| C | Recovering Policy-Induced ErrorsGUI-RobustEval measures GUI agents' ability to recover from their own policy-induced errors across 1,216… | GUI |
| C | RedBenchRedBench measures LLM robustness against adversarial red-teaming attack prompts and refusal behavior across 22… | Safety |
| C | ReMMDReMMDBench measures how well models and agents detect multimodal misinformation in real-world multilingual,… | Factuality |
| C | RoboBenchRoboBench evaluates multimodal large language models as embodied brains for robot manipulation across… | Embodied |
| C | RoboWitsRoboWits measures robot policies' cognitive reasoning, creative tool use, and robustness to unexpected… | Robotics |
| C | Rogo Big Finance BenchBig Finance Bench measures how frontier AI agents perform on 928 rubric-graded finance workflow questions such… | Finance |
| C | RTSGameBenchRTSGameBench measures strategic reasoning of vision-language models through gameplay and diagnostic mini-games… | Game |
| C | RXL-RADSetRXL-RADSet measures how validly and accurately language models assign RADS risk categories from 1,600… | Healthcare |
| C | SALT-KGSALT-KG evaluates how well models perform semantics-aware prediction on multi-table enterprise transactional… | Tabular |
| C | SCICONVBENCHSCICONVBENCH measures how well LLMs conduct multi-turn clarification dialogues to formulate well-posed… | Science |
| C | SciFig-BenchSciFig-Bench measures how well systems generate methodology figures for scientific papers from methodology… | Creative Media |
| C | SemEval-2026 Task 7SemEval-2026 Task 7 evaluates how well LLMs and NLP systems answer everyday cultural knowledge questions… | Multilingual |
| C | SIN-BenchSIN-Bench measures whether multimodal LLMs can understand long-form scientific papers by constructing explicit… | Document Understanding |
| C | SparkleSparkle-Bench evaluates instruction-guided video background replacement models on instruction following,… | Video |
| C | SpeechEQSpeechEQ measures the emotional intelligence of speech-language models by testing whether they select socially… | Speech |
| C | SpeedrunBench: SuperTux 1-1SpeedrunBench: SuperTux 1-1 measures how quickly LLM agents can complete the first level of the platformer… | Agentic Multimodal |
| C | StrategyQAStrategyQA measures a model's ability to answer yes/no questions that require inferring implicit multi-step… | Knowledge |
| C | StructEvalStructEval measures LLMs' ability to generate and convert between structured output formats, both… | Structured Output |
| C | SWE-TogetherSWE-Together measures how well coding agents complete repository-level tasks in multi-turn interactive… | Software Engineering |
| C | TAU2-bench RetailTAU2-bench Retail measures how well conversational agents complete multi-turn retail customer-service tasks… | Agentic Business |
| C | TDC BindingDB Patent DTI-DGMeasures how well models predict drug-target binding affinity on BindingDB patent data under domain… | Science |
| C | TDC TCHard TCR-EpitopeMeasures how well models predict TCR-epitope binding interactions, scored by AUPRC on leaderboards under NA… | Science |
| C | TeleCom-BenchTeleCom-Bench measures how well LLMs understand telecom knowledge (3GPP, 5G, proprietary product docs) and… | Domain |
| C | TerminalWorldTerminalWorld measures how well AI agents complete real-world terminal tasks reverse-engineered from… | Agentic |
| C | TinyGrafixBenchTinyGrafixBench measures how precisely image editing models can perform data visualization edits (bar,… | Multimodal |
| C | TriBench-KoTriBench-Ko is a Korean benchmark that evaluates LLM task performance and deployment risks (inaccuracy, bias,… | Legal |
| C | Video-MME v2Video-MME-v2 evaluates multimodal models' video understanding across a tri-level hierarchy of visual… | Multimodal |
| C | VideoDRVideoDR measures multimodal models' ability to answer video-conditioned open-domain questions by extracting… | Multimodal |
| C | VoiceAssistant-EvalVoiceAssistant-Eval measures how well AI voice assistants handle listening, speaking, and viewing tasks across… | Speech |
| C | VoxParadoxVoxParadox measures whether audio LLMs understand paralinguistic speech cues (age, gender, emotion, pitch,… | Audio |
| C | WBenchWBench evaluates interactive video world models over multi-turn cases covering navigation, subject action,… | Video |
| C | WebLINXWebLINX measures how well agents perform conversational web navigation, controlling a browser to follow… | Agentic |
| C | WebWorld-BenchWebWorld-Bench measures how well language world models predict browser state transitions on real-web… | Agentic |
| C | WorkArena-L1WorkArena-L1 measures how well LLM-based web agents complete atomic knowledge-work tasks in the ServiceNow… | Agentic Business |
| C | WorkArena-L2WorkArena-L2 measures web agents' ability to complete compositional, realistic knowledge-work workflows on the… | Agentic Business |
| C | YouCook2YouCook2 scores video-language models on cooking instructional videos via text-to-video retrieval, video… | Knowledge |
| C | AnyGroundBenchAnyGroundBench measures how well vision-language models perform and adapt at spatio-temporal video grounding… | |
| C | EduArtEduArt measures multimodal LLMs' art-historical knowledge and visual reasoning using 871 human-authored… | |
| C | EgoArgusEgoArgus measures whether vision-language models can act as egocentric daily assistants, arbitrating between… | |
| C | S-EMBERS-EMBER measures streaming episodic memory retrieval in long-form egocentric smart-glasses video, requiring… | |
| C | VWE-BENCHVWE-BENCH measures how well multimodal agents construct and refine interactive 3D open worlds from user… | |
| C | 1GC-7RC1GC-7RC measures how well autonomous AI coding agents can design, implement, and train ML models from scratch… | Agentic |
| C | 3DCodeBench3DCodeBench measures how well vision-language model agents translate text and image references into procedural… | Coding |
| C | ABRA: Agent Benchmark for Radiology ApplicationsABRA measures how well (V)LLM agents operate a medical imaging viewer and DICOM server via function-calling… | Biomedicine Healthcare |
| C | Absurd WorldAbsurd World is a benchmarking framework that tests whether LLMs reason logically in altered, absurd versions… | Reasoning |
| C | ACC-OOD benchmarkACC-OOD evaluates frozen noisy-label-learning image classifier checkpoints on both closed-set accuracy and… | Machine Learning |
| C | ActTraitBenchActTraitBench measures personality consistency in LLMs, quantifying the gap between explicit persona… | Behavioral |
| C | ADBenchAD-Bench measures how well LLM agents answer real-world advertising and marketing analytics requests using… | Agentic Business |
| C | AdversaRiskQAAdversaRiskQA measures whether LLMs detect and resist confidently framed misinformation injected into prompts… | Factuality |
| C | AfriVox-v2AfriVox-v2 measures speech recognition accuracy of speech models on in-the-wild African-language audio across… | Speech |
| C | AgentRewardBenchAgentRewardBench measures how well LLM judges and automatic evaluators assess the success, side effects, and… | Agentic |
| C | AgentEHRAgentEHR measures how well LLM agents perform complex clinical decision-making tasks such as diagnosis and… | Healthcare |
| C | AgentEvalBenchAgentEvalBench measures how well AI systems can automatically generate agent evaluations (metrics, executable… | Agentic |
| C | AgentMeterAgentMeter measures how well model-CLI configurations perform on local command-line tasks, using a… | Agentic |
| C | AgentTrapAgentTrap measures whether LLM agents using third-party skills in a sandbox resist malicious runtime behavior… | Agentic |
| C | AGGBenchAGGBench evaluates whether systems can answer entity-level aggregation queries over large unstructured text… | Retrieval |
| C | AGORAAGORA measures how well LLM agents locate sparse evidence across large archives of workplace documents,… | Agentic |
| C | AHa-BenchAHa-Bench measures auditory hallucinations in large audio-language models across semantic, acoustic, and… | Audio |
| C | AIPatient ArenaAIPatient Arena evaluates LLMs in multi-turn, EHR-grounded clinical consultations across eight dimensions of… | Biomedicine Healthcare |
| C | AirNavAirNav measures how well UAV agents follow natural-language instructions to navigate realistic urban aerial… | Embodied |
| C | AIRTBenchAIRTBench measures language models' ability to autonomously discover and exploit AI/ML security… | Cybersecurity |
| C | AITZ_EMAITZ_EM scores GUI agents on Android smartphone tasks by checking whether their predicted screen actions… | Agentic |
| C | ALARBALARB evaluates LLM reasoning in the Arabic legal domain using Saudi commercial court cases, with tasks such… | Legal |
| C | ALFREDALFRED measures how well models map natural language instructions and egocentric vision to action sequences… | Embodied |
| C | ALFWorldALFWorld measures how well agents complete household tasks in interactive text environments aligned with… | Agentic |
| C | ambiguous emotion recognition benchmark for ALMs with TTSA benchmark evaluating how well audio-language models recognize ambiguous emotions in speech under various… | Audio |
| C | AMVICCAMVICC measures visual reasoning failure modes of multimodal LLMs (via VQA) and image generation models (via… | Multimodal Understanding |
| C | AOR-BenchAOR-Bench measures whether large audio language models over-refuse benign pseudo-harmful audio queries, using… | Audio |
| C | API-BankAPI-Bank evaluates tool-augmented LLMs on planning, retrieving, and calling APIs across 314 annotated tool-use… | Tool Use |
| C | ArguAnaArguAna measures how well embedding and retrieval models retrieve the best counterargument for a given… | Retrieval |
| C | Artificial Intelligence Quotient (AIQ) BenchmarkThe AIQ Benchmark measures psychometric cognitive abilities of generative AI models (verbal, quantitative,… | Knowledge |
| C | ASR Code-Switch BenchmarkMeasures commercial automatic speech recognition accuracy (WER and BERTScore) on code-switched speech across… | Speech |
| C | ASTRA-QAASTRA-QA measures how well RAG systems answer abstract questions that require synthesizing information across… | Question Answering |
| C | AstroAlertBenchAstroAlertBench evaluates multimodal LLMs on reviewing real astronomical transient alerts from ZTF through… | Domain |
| C | AstroReason-BenchAstroReason-Bench evaluates LLM agents' planning on physics-constrained space planning problems such as ground… | Aerospace |
| C | ATC Safety-Oriented Language Understanding EvalMeasures how reliably LLMs interpret air traffic control utterances, using consequence-aware scoring that… | Aerospace |
| C | ATRBenchATRBench measures whether long-lived LLM agents proactively ask for reusable user preferences that a later… | Agentic |
| C | AttaQAttaQ measures how vulnerable large language models are to adversarial questions designed to provoke harmful… | Safety |
| C | AttuneBenchAttuneBench measures LLM emotional intelligence in real multi-turn human-model conversations, scoring emotion… | Behavioral |
| C | AutoLogiAutoLogi is a bilingual benchmark of automatically generated open-ended logic puzzles with program-based… | Knowledge |
| C | AutoRO-BenchAutoRO-Bench evaluates how well LLMs reformulate uncertain robust optimization models into tractable… | Math |
| C | AVGen-BenchAVGen-Bench scores text-to-audio-video generation models on uni-modal quality, cross-modal alignment, and… | Video |
| C | AWARE-USAWARE-US measures whether tool-calling conversational agents querying a structured car database can… | Tool Use |
| C | BabelJudgeBabelJudge measures the reliability of LLM-as-a-judge models across languages and agent trajectories, scoring… | Multilingual |
| C | BenGERBenGER evaluates LLM systems on subsumption-based legal reasoning in German law using exam-style free-text… | Legal |
| C | BenHalluEvalBenHalluEval measures how well LLMs detect hallucinations in Bengali across generative QA, Bangla-English… | Factuality |
| C | Beyond Binary SuccessBeyond Binary Success (MetaFine) is a diagnostic meta-evaluation framework that scores vision-language-action… | Embodied |
| C | Beyond Function CallingBeyond Function Calling (ToolBench-X) measures how well tool-using LLM agents complete executable multi-step… | Tool Use |
| C | Beyond Static DialoguesRHELM (Beyond Static Dialogues) measures LLM long-term memory over realistic, evolving dialogues integrated… | Long Context |
| C | BilliardPhys-BenchBilliardPhys-Bench measures multimodal LLMs' intuitive physical reasoning in synthetic billiards scenes,… | Reasoning |
| C | BioSecBench-RefusalBioSecBench-Refusal measures whether AI agents refuse legitimate life-science research tasks and whether they… | Safety |
| C | BIRD-PythonBIRD-Python measures how reliably models translate natural-language data questions into executable… | Coding |
| C | BizBenchBizBench measures LLMs' quantitative reasoning on realistic business and financial problems, including… | Finance |
| C | BLUEX v2BLUEX v2 measures LLM performance in Portuguese on open-ended, free-form questions from second-phase Brazilian… | Multilingual |
| C | BossesCommunities can sustainably manage shared resources (commons) through self-governance and cooperative norms, a… | Agentic |
| C | BUILD-AND-FINDBUILD-AND-FIND evaluates whether downstream agents can recover intended design choices from agent-generated… | Software Engineering |
| C | CADBenchCADBench measures how well models generate editable CAD programs (CadQuery code) from meshes or images,… | Multimodal |
| C | CAM-BenchCAM-Bench measures how well language models and formalization agents prove 1,000 Lean 4 theorems in… | Math |
| C | Can Large Language Models UnderstandCodeMixQA measures how well LLMs understand, reason over, and answer questions in code-switched… | Multilingual |
| C | CanItEditCanItEdit measures how well LLMs perform instructional code editing, updating Python programs given natural… | Coding |
| C | CapRiCorn-1KCapRiCorn-1K measures how accurately and comprehensively models caption long videos (15 s to 10 min) while… | Video |
| C | CAR-benchCAR-bench measures the consistency, uncertainty handling, and limit-awareness of multi-turn, tool-using LLM… | Agentic |
| C | CAREBenchCAREBench evaluates LLMs' emotion understanding through cognitive appraisal reasoning, appraisal ratings, and… | Behavioral |
| C | Causal Sensitivity ScoreCSS measures whether clinical LLMs and agents correctly update oncology tumor-board recommendations when… | Biomedicine Healthcare |
| C | CBT-AudioCBT-Audio measures how well audio language models estimate turn-level patient distress intensity from spoken… | Audio Healthcare |
| C | CFEvalCFEval is a coding benchmark that scores models on competitive programming problems, reported as an Elo-style… | Coding |
| C | ChaosBench-Logic v2ChaosBench-Logic v2 measures LLM logical reasoning about dynamical systems, with 40,886 binary questions over… | Reasoning |
| C | Chart data extraction benchmarkMeasures how accurately multimodal LLMs extract data tables from real-world chart images that lack visible… | Multimodal Understanding |
| C | ChatRAG BenchChatRAG Bench evaluates a model's conversational question answering over documents or retrieved context across… | Retrieval |
| C | ChiEngMixBenchChiEngMixBench evaluates how spontaneously and naturally large language models produce Chinese-English… | Multilingual |
| C | ChronosBenchChronosBench measures how well LLM agents maintain long-term user intents and proactively follow up on… | Agentic |
| C | Citation GroundingCitation Grounding measures how accurately LLMs cite Ukrainian legal provisions, scoring citation precision,… | Legal |
| C | CITYREPCityRep evaluates urban representation learning models across 8 cities and 8 tasks (regression,… | Geospatial |
| C | Claw-AnythingClaw-Anything measures how well LLM agents act as always-on personal assistants over long activity histories,… | Agentic |
| C | Claw-Eval-LiveClaw-Eval-Live measures AI agents on 105 real-world enterprise workflow tasks across 17 families, such as… | Agentic Business |
| C | Clinical Reasoning GraphsClinical Reasoning Graphs evaluates LLM diagnostic reasoning on NEJM clinicopathological cases by extracting… | Healthcare |
| C | ClinicalBenchClinicalBench is a 400-question test over 43 MIMIC-IV patients that measures assertion-aware retrieval and… | Biomedicine Healthcare |
| C | ClinPivotClinPivot measures whether clinical models correctly change treatment decisions when pivoted patient contexts… | Biomedicine Healthcare |
| C | CNMO 2024CNMO 2024 measures models' ability to solve challenging competition mathematics problems from the 2024 Chinese… | Math |
| C | CoCoVideoCoCoVideo measures how well detectors distinguish real videos from high-quality AI-generated videos produced… | Video |
| C | Code-Guided Reasoning for Small Language ModelsCGR is an evaluation protocol measuring when executable Python reasoning scaffolds generated by code improve… | Reasoning |
| C | CodeClinicCodeClinic measures whether LLM agents can synthesize and compose reusable code-based clinical skills over… | Healthcare |
| C | CodeGolf BenchCodeGolf Bench measures LLMs' ability to generate concise, minimal-length code across up to 60 programming… | Coding |
| C | CodeGuard: Improving LLM Guardrails in CS EducationCodeGuard is a benchmark dataset of 8,000 prompts for evaluating how well LLM guardrails classify irrelevant,… | Safety |
| C | CogCAPTCHA30CogCAPTCHA30 is a battery of 30 cognitive tasks that measures whether process-level behavioral features… | Behavioral |
| C | CogRailCogRail measures how well vision-language models perceive railway intrusion risks through spatial position… | Spatial Reasoning |
| C | COLLIECOLLIE measures how well language models generate text satisfying rich, compositional constraints at word,… | Writing |
| C | ColorConceptBenchColorConceptBench measures how well text-to-image models associate colors with implicit concepts such as… | Creative Media |
| C | CombEvalCombEval measures how well large language models solve generated natural-language combinatorial counting… | Reasoning |
| C | CommunityFactCommunityFact measures LLMs' ability to verify claims and detect misinformation in the wild across five… | Factuality |
| C | COMPASSCOMPASS evaluates whether LLMs comply with organization-specific allowlist and denylist policies across eight… | Alignment |
| C | Connecting the DotsRefMem-Bench (Connecting the Dots) measures reflective memory in long-horizon dialogue, requiring models to… | Reasoning |
| C | ConvFinQAConvFinQA measures models' ability to perform chains of numerical reasoning in conversational question… | Finance |
| C | Cookie-BenchCookie-Bench measures LLMs' ability to generate front-end web applications, scored reference-free by an agent… | Coding |
| C | COPACOPA measures open-domain commonsense causal reasoning by asking models to pick which of two alternatives more… | Knowledge |
| C | CorpusQACorpusQA measures how well LLMs and systems reason over entire document corpora up to 10 million tokens,… | Long Context |
| C | Correction Suppression BenchmarkMeasures how often LLMs fail to correct false premises when those claims are embedded in task-oriented… | Factuality |
| C | CounterCountCounterCount measures whether vision-language models count objects from visual evidence rather than prior… | Reasoning |
| C | CPCD-BenchCPCD-Bench evaluates models' long-horizon campus psychological counseling abilities in Chinese across… | Healthcare |
| C | CRAGCRAG measures how accurately LLMs and retrieval-augmented generation systems answer factual questions across… | Search |
| C | CSSBenchCSSBench measures the safety of lightweight LLMs against Chinese-specific adversarial patterns such as… | Safety |
| C | CSVFidelity-BenchCSVFidelity-Bench measures the fidelity of knowledge graphs that LLM pipelines build from statistical CSV… | Data |
| C | CUDAHerculesCUDAHercules measures whether LLMs can write expert-level, hardware-aware optimized CUDA code compared against… | Coding |
| C | CulMindCulMind measures multimodal LLMs' understanding and reasoning on Chinese Cultural Heritage tasks, scoring both… | Reasoning |
| C | CurveBenchCurveBench measures whether vision-language models can recover the exact rooted containment tree of nested,… | Spatial Reasoning |
| C | CUSPCUSP measures whether AI systems can forecast scientific progress across 4,760 events via feasibility… | Science |
| C | CVE-BenchCVE-Bench measures AI agents' ability to autonomously exploit 40 critical real-world web vulnerabilities… | Cybersecurity |
| C | CVerifBenchAn evaluation framework of 500 C program verification tasks built on SV-COMP 2025 that scores models on… | Coding |
| C | CyberChainBenchCyberChainBench measures how well LLM-based agents detect vulnerabilities, generate exploits, and synthesize… | Cybersecurity |
| C | CyBiasBenchCyBiasBench measures attack-selection bias of LLM agents in offensive cybersecurity scenarios across targets,… | Cybersecurity |
| C | DACompDAComp measures how well data agents handle enterprise data engineering (building and evolving multi-stage SQL… | Data |
| C | DataClawBenchDataClawBench measures how well autonomous agents perform exploratory analysis over noisy, real-world,… | Finance Business |
| C | DecisionBenchDecisionBench measures how well AI agents delegate subtasks to peer models in long-horizon agentic workflows,… | Agentic |
| C | DecodingTrust-Agent Platform (DTap)AI agents are increasingly deployed across diverse domains to automate complex workflows through long-horizon… | Safety |
| C | DeepResearch BenchDeepResearch Bench measures how well deep research agents produce citation-rich research reports on 100… | Agentic |
| C | DeepSynth-EvalDeepSynth-Eval measures how well LLMs and agents consolidate information from hundreds of reference papers… | Summarization |
| C | DeepWeb-BenchDeepWeb-Bench measures how well deep research agents search the open web, collect and reconcile cross-source… | Reasoning |
| C | DeFAbDeFAb measures whether foundation models can perform defeasible abduction, constructing hypotheses that… | Reasoning |
| C | DisasterBenchDisasterBench measures whether LLMs can plan executable multi-step disaster-response workflows by composing… | Agentic |
| C | DistractionBenchDistractionBench evaluates whether video large language models can robustly link subjects to events when… | Video |
| C | DMV-BenchDMV-Bench measures whether multimodal agents retain visual memory across chains of autonomous web shopping… | Agentic |
| C | Do Joint Audio-Video Generation Models Understand Physics?AV-Phys Bench measures whether joint audio-video generation models produce outputs consistent with… | Video |
| C | Doc-PPDoc-PP measures whether large vision-language models adhere to user-defined non-disclosure policies when… | Document Understanding |
| C | Don't Judge a Book by its CoverLogiQAte, introduced in the paper Don't Judge a Book by its Cover, measures whether LLMs keep reasoning… | Reasoning |
| C | DPBenchDPBench measures how well multi-agent LLM systems coordinate under simultaneous resource contention, using the… | Agentic |
| C | Dr. DocBenchDr. DocBench measures how well vision-language models and document parsers handle expert-level, difficult… | Document Understanding |
| C | DRBenchDRBench measures dense-scene visual reasoning in vision-language models, with 14,573 questions over 2,943… | Vision |
| C | DRInQDRInQ measures language models' pragmatic reasoning about conversational implicature in questions, varying… | Reasoning |
| C | DrugAuditDrugAudit is a 3,772-item authority-aware benchmark for drug-information question answering that scores source… | Biomedicine Healthcare |
| C | DSGymDSGym evaluates LLM data science agents on executing end-to-end data analyses, bioinformatics tasks and… | Data |
| C | E2V-BenchE2V-Bench measures whether text-to-image models can generate pedagogically meaningful visuals from arithmetic… | Education |
| C | ECLeKTicECLeKTic measures LLMs' cross-lingual knowledge transfer via closed-book fact-seeking questions across 12… | Knowledge |
| C | EComAgentBenchEComAgentBench measures how well LLM shopping agents uncover hidden user intent across query, profile and… | Agentic |
| C | EDIREDIR is a fine-grained composed image retrieval benchmark of 5,000 queries across five categories and fifteen… | Retrieval |
| C | EduBenchEduBench evaluates large language models on 9 student- and teacher-oriented educational scenarios, scoring… | Education |
| C | EduGuardBenchEduGuardBench evaluates LLMs acting as simulated teachers on pedagogical fidelity (teaching harm) and… | Education |
| C | EduResearchBenchEduResearchBench evaluates LLMs on educational academic writing across six research modules and 24… | Education |
| C | Ego2WorldEgo2World measures how well embodied agents plan under partial observation in executable symbolic household… | Embodied |
| C | EgoBenchEgoBench measures how well multimodal video-LLM agents combine egocentric visual perception, tool-augmented… | Agentic |
| C | EmoMMEmoMM measures how multimodal large language models recognize emotions from video, audio and text when… | Multimodal |
| C | EnactToMEnactToM measures functional Theory of Mind in embodied multi-agent agents, scoring whether models can act on… | Reasoning |
| C | EnterpriseClawBenchEnterpriseClawBench measures how well harness-model agent systems complete realistic enterprise workplace… | Agentic Business |
| C | EnvSimBenchEnvSimBench measures how accurately LLMs can simulate environment feedback and state transitions for agent… | Agentic |
| C | EnvTrustBenchEnvTrustBench measures whether LLM agents remain grounded in the true environment state or overtrust stale,… | Agentic |
| C | EO-GymEO-Gym evaluates multimodal, tool-using vision-language agents on interactive Earth Observation analysis tasks… | Geospatial |
| C | ERGeoBenchERGeoBench measures how well multimodal LLMs perform embodied geo-localization from street-view imagery across… | Embodied |
| C | ES-MemEvalES-MemEval measures conversational agents' long-term memory capabilities (information extraction, temporal… | Long Context |
| C | ETHICSETHICS measures language models' knowledge of basic moral concepts across justice, deontology, virtue ethics,… | Generalization |
| C | Every Act Has Its PriceMoral Trolley Arena, introduced in the paper Every Act Has Its Price, is a two-stage blind ELO benchmark… | Safety |
| C | EvoEnvEvoEnv is a dynamic evaluation environment that scores agents in workplace scenarios on context-aware… | Agentic |
| C | EvolMemEvolMem measures multi-session dialogue memory capabilities of LLMs and agent systems across declarative and… | Long Context |
| C | EvoMemBenchEvoMemBench measures how well memory mechanisms in LLM-based agents store, update, and retrieve information… | Agentic |
| C | Executable GamesMeasures interactive multi-turn reasoning of LLMs across 474 executable games where models query a hidden… | Game |
| C | Eye-QEye-Q measures vision-language models' ability to solve cue-implicit visual word puzzles by inferring a target… | Multimodal Understanding |
| C | Factorio Learning EnvironmentFactorio Learning Environment scores LLM agents on building automated factories in the game Factorio, measured… | Game |
| C | FastKernelsFastKernels measures how well LLM-based agents generate optimized GPU kernels that are correct and faster than… | Coding |
| C | FINAL Bench MetacognitiveFINAL Bench measures functional metacognition in language models: whether they recognize uncertainty and… | Knowledge |
| C | FinREDFinRED evaluates the safety of financial LLMs against finance-specific red-teaming prompts covering fraud,… | Finance |
| C | FinRetrievalFinRetrieval measures how accurately AI agents retrieve specific numeric financial values from structured… | Finance |
| C | FinSafetyBenchFinSafetyBench is a bilingual English-Chinese red-teaming benchmark that measures whether LLMs refuse requests… | Finance |
| C | FinSearchCompFinSearchComp measures how well search-enabled LLM agents retrieve time-sensitive, simple historical, and… | Finance |
| C | FlenQAFlenQA measures how increasing input length, via padding of different lengths, types and locations, degrades… | Long Context |
| C | FLEURSFLEURS is a 102-language parallel speech benchmark evaluating few-shot speech tasks such as ASR, speech… | Speech |
| C | FML-BenchFML-Bench measures how well AI research agents iteratively improve baseline codebases on 18 fundamental… | Machine Learning |
| C | FORCEBENCHFORCEBENCH is a contrastive stress test measuring whether citation evaluators and model judges score… | Retrieval |
| C | FormInvFormInv measures whether models give consistent answers to semantically equivalent paraphrases of… | Math |
| C | FounderBenchFounderBench measures whether LLM agents can make repeated, consequential startup-like business decisions… | Agentic |
| C | From 0-Order Selection to 2-Order JudgmentLogiHard measures LLM logical reasoning on multiple-choice exam questions combinatorially transformed from… | Reasoning |
| C | From Hallucination to GroundingCRISP Bench evaluates vision-language models' visual spatial intelligence by checking consistency between… | Spatial Reasoning |
| C | From Knowing to DoingKTD-Fin measures whether LLM trading agents can generate genuine stock-selection alpha in end-to-end… | Finance |
| C | From Prompt Injection to Persistent ControlClawTrojan measures whether LLM agents or defenses in local agentic harnesses can stop multi-step persistent… | Cybersecurity |
| C | From Text to VoiceEvaluates tool-calling ability of omni-modal voice agents on audio-converted versions of text tool-calling… | Speech |
| C | FrontierORFrontierOR measures whether LLMs can design scalable, efficient algorithms for realistic large-scale… | Math |
| C | FRTR-BenchFRTR-Bench measures how well LLMs reason over large multimodal enterprise Excel workbooks with linked sheets… | Spreadsheets |
| C | G-IdiomAlignG-IdiomAlign measures how well LLMs align and translate idioms across languages, using English Wiktionary… | Multilingual |
| C | Gaokao-BenchGAOKAO-Bench evaluates large language models' language understanding and logical reasoning using 2,811… | Knowledge |
| C | GateMemGateMem measures whether memory-augmented LLM agents in multi-principal shared-memory settings stay useful,… | Agentic |
| C | GAVEL: Grounded Caption Error Verification and LocalizationGAVEL measures whether vision-language models can verify image-caption misalignment, explain the discrepancy,… | Vision |
| C | Gaze4HRIGaze4HRI is a large-scale video dataset and benchmark that evaluates zero-shot appearance-based 3D gaze… | Robotics |
| C | Gen-ArenaGen-Arena is a human-annotated benchmark measuring how faithfully text-to-image models realize complex visual… | Vision |
| C | Generalization SpectrumGeneralization Spectrum is an evaluation framework that measures how far learning from individual competitive… | Generalization |
| C | Generalized Zero-Shot Benchmark for Facial Age EstimationMeasures how well facial age estimation models trained only on adult (18-59) face images generalize zero-shot… | Generalization |
| C | GENSTRATGENSTRAT measures LLM strategic reasoning by evaluating models head-to-head on procedurally generated… | Reasoning |
| C | Geo-EvalGeo-Eval is a domain-specific benchmark that scores language models on specialized geological reasoning, such… | Science |
| C | GeoRCGeoRC evaluates vision-language models on geolocation reasoning from images, scoring reasoning chains by… | Geospatial |
| C | German Statutory QA BenchmarkMeasures how well LLMs answer time-sensitive German statutory law questions, testing post-cutoff staleness and… | Legal |
| C | GlobalDentBenchGlobalDentBench measures LLM clinical reasoning and safety in dentistry with 8,978 expert-validated questions… | Biomedicine Healthcare |
| C | Gorilla Benchmark API BenchAPIBench measures how accurately language models generate API calls for HuggingFace, TorchHub, and TensorHub… | Tool Use |
| C | GovReportGovReport evaluates long document summarization of government agency reports with significantly longer… | Summarization |
| C | GRACEGRACE measures step-level faithfulness of chain-of-thought reasoning over input context, with human… | Reasoning |
| C | GraphARCGraphARC measures abstract few-shot reasoning by asking models to infer a graph transformation rule from… | Reasoning |
| C | GraphIP-BenchGraphIP-Bench measures how easily graph neural networks can be stolen through model-extraction attacks and how… | Cybersecurity |
| C | GroundUI-1KGroundUI evaluates GUI grounding: whether models can locate the right on-screen element or coordinates for an… | Multimodal |
| C | GroupMemBenchGroupMemBench measures how well LLM agent memory systems extract, retrieve and apply information in… | Agentic |
| C | GroupTravelBenchGroupTravelBench measures LLM agents on multi-user, multi-turn travel planning, testing preference… | Agentic |
| C | GrowLoopGrowLoop is a self-evolving evaluation system and benchmark that measures human-likeness of large language… | Human-preference |
| C | GRUFF: LLM Pronoun Fidelity, Reasoning, and Biases in GermanGRUFF measures LLMs' pronoun fidelity, referential reasoning, and gender biases in German across gender… | Reasoning |
| C | GSM-SEMGSM-SEM measures the robustness of LLM mathematical reasoning on semantically perturbed variants of… | Reasoning |
| C | GSO-BenchGSO-Bench is a frontier benchmark, described as a general science and observation benchmark, that scores… | Science |
| C | GTEvalGTEval is an evaluation pipeline that measures whether graph-tokenizing LLMs understand graph tokens, using… | Reasoning |
| C | GUI Agents for Continual Game GenerationPlaytestArena, introduced in this paper, measures whether models can generate playable browser-based games: a… | Game |
| C | GUI vs. CLIA matched benchmark of 440 desktop tasks across 18 applications comparing screen-only GUI agents and… | Agentic |
| C | gwBenchmarksgwBenchmarks measures whether LLM coding agents can perform end-to-end, high-precision gravitational wave… | Science |
| C | ScienceAgentBenchScienceAgentBench measures how well language agents can generate self-contained Python programs that complete… | Agentic |
| C | USACOUSACO measures whether language models and agents can solve 307 competitive programming problems from the USA… | Coding |
| C | HalluHard: A Hard Multi-Turn Hallucination BenchmarkHalluHard measures how often language models hallucinate ungrounded factual claims in multi-turn conversations… | Truthfulness |
| C | HallusionBenchHallusionBench measures how large vision-language models handle image-context reasoning, diagnosing language… | Vision |
| C | Halluverse-M3Halluverse-M3 measures how well language models detect fine-grained (entity, relation, sentence-level)… | Safety |
| C | HalluWorldHalluWorld measures language model hallucination against fully specified reference worlds across gridworlds,… | Factuality |
| C | Harness-BenchHarness-Bench measures how agent harness configurations, across multiple model backends, affect completion,… | Agentic |
| C | HarnessAudit-BenchHarnessAudit-Bench measures whether LLM agent harnesses respect user intent, permission boundaries and… | Safety |
| C | HealthAdminBenchHealthAdminBench measures how well AI agents complete administrative healthcare tasks (prior authorization,… | Healthcare |
| C | HearSay Benchmark: Do Audio LLMs Leak What They Hear?HearSay measures whether audio LLMs infer and leak private user attributes (age, gender, health, income, etc.)… | Audio |
| C | HerculeanHerculean measures whether AI agents can carry out end-to-end financial professional workflows (Trading,… | Finance |
| C | HIDBenchHIDBench measures how well large language models perform host-based intrusion detection over large, noisy,… | Cybersecurity |
| C | HOLMES: Evaluating Higher-Order Logical Reasoning in LLMsHOLMES measures LLMs' higher-order symbolic logical reasoning over rules, predicates and constraints using… | Reasoning |
| C | HoloFairHoloFair measures multidimensional demographic bias (gender, age, race) in text-to-image models under neutral… | Safety |
| C | Honeyquest for LLMsHoneyquest for LLMs evaluates how often LLM attackers fall for cyber deception traps across 174 reconnaissance… | Cybersecurity |
| C | HoneyvalHoneyval evaluates LLMs acting as HTTP honeypots against AI hacking agent attackers across 16 backend… | Cybersecurity |
| C | How Well Do Models Follow Their Constitutions?An audit pipeline that measures how often models violate their developers' published behavioral specifications… | Alignment |
| C | HPR-BenchHPR-Bench measures how well LLMs perform holistic persona reasoning from simulated user behavioral histories,… | Behavioral |
| C | HRBenchHRBench is a unified evaluation framework that compares thinking-mode switching strategies in hybrid-reasoning… | Reasoning |
| C | htkgh-polecathtkgh-polecat measures how well LLMs forecast geopolitical events, via relation prediction on a… | Geospatial |
| C | IB-benchIB-bench measures whether large language models can perform investment banking analyst tasks, scored across… | Domain Business |
| C | ICU-BenchICU-Bench measures how well unlearning methods let multimodal large language models continually forget… | Biomedicine Healthcare |
| C | IDEAFixIDEAFix evaluates LLMs' divergent thinking and originality in open-ended idea generation for short design… | Creative |
| C | IFStruct v1.0IFStruct measures whether a model can follow instructions to produce valid JSON or YAML output that matches a… | Instruction Following |
| C | IMAVBIMAVB measures whether omnimodal LLMs detect textual premises that contradict the video or audio of long-form… | Multimodal |
| C | IMCBenchIMCBench measures how well multimodal LLMs handle image-grounded, multi-turn medical conversations, scoring… | Healthcare |
| C | IMLogicIMLogic measures how well LLM memory systems retrieve implicit, logically relevant memories with limited… | Retrieval |
| C | Inductive Generalization for Robotic ManipulationMeasures inductive generalization of robotic manipulation (visuomotor) policies on progressively harder,… | Robotics |
| C | InfiniBenchInfiniBench measures how well large multi-modal models understand very long videos (movies and TV shows)… | Multimodal |
| C | InfiniteBenchInfiniteBench measures language models' ability to process, understand, and reason over contexts longer than… | Generalization |
| C | InfiniteBM Settlers of CatanA head-to-head arena that ranks LLMs by Bradley-Terry Elo ratings from matches of Settlers of Catan involving… | Game |
| C | InfoVQAtestInfographicVQA measures visual question answering over infographic images, requiring joint reasoning over… | Multimodal |
| C | InkBench OCRInkBench OCR ranks OCR models on InkBench document images using pairwise VLM-as-judge comparisons scored with… | Multimodal |
| C | InteractiveBench MathInteractiveBench Math evaluates LLMs' ability to solve HLE math problems through budgeted multi-turn… | Math |
| C | InteractiveBench Texas Hold'emInteractiveBench Texas Hold'em evaluates LLMs' strategic reasoning by having them play thousands of no-limit… | Game |
| C | InteractiveBench Trust GameInteractiveBench Trust Game evaluates LLMs' strategic reasoning in a round-robin Iterated Prisoner's Dilemma… | Game |
| C | InterLV-SearchInterLV-Search measures how well multimodal agents perform interleaved language-vision agentic search, where… | Agentic |
| C | INVESTORBENCHInvestorBench evaluates LLM-based agents on financial decision-making tasks such as trading stocks,… | Finance |
| C | IPO Finance AgentIPO Finance Agent measures how well LLM agents perform IPO due diligence by answering questions over long SEC… | Finance |
| C | IslamicLegalBenchIslamicLegalBench measures LLM knowledge and reasoning of Islamic law across seven schools of jurisprudence,… | Legal |
| C | iWorld-BenchiWorld-Bench evaluates interactive world models on interaction abilities such as visual generation, trajectory… | Video |
| C | JamSet/JamBenchJamBench measures models' ability to generate and complete project-level game code on the Godot engine, scored… | Game |
| C | JEEBenchJEEBench measures LLM problem-solving on 515 challenging mathematics, physics and chemistry problems from the… | Math |
| C | JMMMU-ProJMMMU-Pro measures how well large multimodal models understand Japanese multi-discipline questions presented… | Multimodal |
| C | JourneyBenchJourneyBench measures how well LLM customer-support agents adhere to multi-step business policies across 703… | Agentic Business |
| C | Judge Model for Large-scale Multimodality BenchmarksA multimodal benchmark spanning text, audio, image and video tasks in which a judge model scores MLLM outputs… | Multimodal |
| C | JudgmentBenchJudgmentBench measures how well rubric scoring versus pairwise preference judgments, from expert attorneys and… | Legal |
| C | JuICEJuICE measures how well LLM judges detect span-level cultural and linguistic errors in long-form LLM responses… | Alignment |
| C | K-FinHalluK-FinHallu measures how well LLMs detect hallucinations, including justified abstention, in multi-turn Korean… | Finance |
| C | K12-BenchK12-Bench measures LLMs' curriculum cognition in K-12 education via 23,640 multi-select questions derived from… | Education |
| C | KARMA-MVKARMA-MV measures models' causal audio-visual reasoning about how visual dynamics drive musical structure in… | Video |
| C | KernelBenchXKernelBenchX measures how well LLM-generated Triton GPU kernels compile, match reference outputs numerically,… | Coding |
| C | Kinship Data Benchmark for Multi-hop ReasoningKinshipQA measures LLMs' multi-hop reasoning over kinship relations in generated, culture-specific family… | Reasoning |
| C | Knowledge-Intensive Video GenerationKIVI-Bench measures the factuality and helpfulness of videos that text-to-video models generate from short… | Factuality |
| C | KOCO-BENCHKOCO-BENCH measures how well LLMs and coding agents acquire and apply domain knowledge from curated corpora to… | Coding |
| C | KramaBenchKramaBench measures how well AI systems and agents can design and execute end-to-end data-to-insight pipelines… | Data Agentic+1 |
| C | KSAFE-MMKSAFE-MM measures the safety of multimodal large language models against general and Korean culture-specific… | Safety |
| C | KVoiceBench, KOpenAudioBench, KMMAUKorean speech benchmarks that evaluate speech language models on Korean spoken question answering… | Speech |
| C | LaborBenchLaborBench is a question-and-answer benchmark that evaluates how accurately AI systems, including RAG… | Legal |
| C | LabOSBenchLabOSBench measures how well multimodal GUI agents operate web-based scientific-instrument simulators across… | GUI |
| C | LAMBADALAMBADA measures language modeling and broad context understanding by scoring models on predicting the final… | Language |
| C | LaViSALaViSA measures how well vision-language models resolve structural (syntactic) ambiguity in sentences by using… | Multimodal |
| C | LEAF: A Living Benchmark for Event-Augmented ForecastingLEAF is a living benchmark that evaluates LLMs on event-augmented forecasting tasks, including future event… | Reasoning |
| C | LegalWorldLegalWorld is an interactive environment modeling Chinese civil litigation across five causally connected… | Legal |
| C | LemonadeBenchLemonadeBench measures LLMs' economic intuition, long-term planning and decision-making under uncertainty by… | Business |
| C | LexGeniusLexGenius is an expert-level Chinese legal multiple-choice benchmark evaluating LLMs' legal understanding,… | Legal |
| C | LIBERTyLIBERTy benchmarks the faithfulness of concept-based explanation methods for LLMs using structural… | Machine Learning |
| C | LibEvoBenchLibEvoBench measures whether code generation models correctly use library APIs as they exist in specific… | Software Engineering |
| C | LifeAgentBenchLifeAgentBench measures how well LLM-based personal health assistants answer long-horizon, cross-dimensional,… | Healthcare |
| C | LinAlg-BenchLinAlg-Bench evaluates LLMs on structured linear algebra computation over 3x3, 4x4 and 5x5 matrices across 9… | Math |
| C | LingoQALingoQA measures how truthfully vision-language models answer questions about short autonomous-driving video… | Multimodal |
| C | LithoBenchLithoBench measures how well large multimodal models understand geological semantics in remote-sensing… | Domain |
| C | LITMUSLITMUS measures whether LLM agents operating in real OS environments can be induced via jailbreak, skill… | Safety |
| C | LiveMathematicianBenchLiveMathematicianBench measures LLMs' research-level mathematical reasoning through monthly multiple-choice… | Math |
| C | LiveBrowseCompLiveBrowseComp measures whether deep-search agents can find recently published facts on the web beyond their… | Search |
| C | LiveFMBenchLiveFMBench measures how well LLMs and agents generate correct formal ACSL specifications for C programs that… | Software Engineering |
| C | LiveK12BenchLiveK12Bench measures large multimodal models' reasoning on real, continuously updated high school exam… | Multimodal |
| C | LivePILivePI measures how often AI agents with tool access fall for indirect prompt-injection attacks embedded in… | Safety |
| C | LLM-as-a-ReviewerBenchmarks 12 LLMs as academic paper reviewers on 898 NeurIPS and ICLR papers, measuring rating calibration,… | Safety |
| C | LLM Game BenchmarkLLM Game Benchmark measures LLMs' strategic play and invalid-move behavior by having them compete against each… | Agentic |
| C | LLMEval-MedLLMEval-Med measures how well large language models handle real-world clinical medical questions across five… | Healthcare |
| C | LLMTabBenchLLMTabBench measures how well LLMs perform binary classification on tabular data in zero- and few-shot… | Tabular |
| C | LogDx-CILogDx-CI measures how well log context-reduction tools preserve evidence for LLM root-cause diagnosis of real… | Software Engineering |
| C | Long Code ArenaLong Code Arena is a suite of six benchmarks evaluating code models on tasks requiring project-wide context,… | Coding |
| C | LongActLongAct evaluates planning-level autonomy of embodied agents in long-horizon household tasks specified by… | Embodied |
| C | LongDALongDA measures how well LLM agents answer analytical queries over U.S. national survey data, where they must… | Data |
| C | LongDS-BenchLongDS-Bench measures how well LLM agents maintain, update, restore and compose evolving analytical states… | Data |
| C | LongWebBenchLongWebBench evaluates vision-language models on generating long, multi-screen webpages from visual inputs,… | Coding |
| C | LungCURELungCURE measures multimodal LLMs' clinical reasoning for lung cancer TNM staging, treatment recommendation,… | Healthcare |
| C | M3-BENCHM3-BENCH measures LLM agents' social behaviors in 24 mixed-motive games, evaluating behavioral trajectories,… | Behavioral |
| C | M3KangM3Kang measures vision-language models' multilingual, multimodal mathematical reasoning on 1,747 Kangaroo Math… | Math |
| C | MaCBenchMaCBench measures how well vision-language models handle real-world chemistry and materials science tasks… | Science |
| C | MacroLensMacroLens measures contextual financial reasoning over U.S. small- and micro-cap equities across seven tasks… | Finance |
| C | MalruleLibMalruleLib measures whether language models can infer a student's mathematical misconception from one worked… | Education |
| C | MammoExpertMammoExpert measures how accurately and interpretably models classify breast lesions in mammography images… | Biomedicine |
| C | Managing Procedural Memory in LLM AgentsAFTER measures how well procedural memory skills in LLM agents improve and transfer across 382 realistic… | Agentic Business |
| C | MANTRAMANTRA is a framework and benchmark suite of 285 tasks across 6 domains that measures whether tool-using LLM… | Tool Use |
| C | MASBENCHMASBENCH is a controlled benchmark that characterizes tasks along five axes (Depth, Horizon, Breadth,… | Agentic |
| C | MASLegalBenchMASLegalBench evaluates LLM-based multi-agent systems on deductive legal reasoning tasks grounded in GDPR… | Legal |
| C | Mat-PrefMat-Pref measures compositional chemical reasoning in inorganic materials via 10,837 DFT-grounded… | Materials |
| C | MathAtlasMathAtlas measures how well models autoformalize graduate-level mathematics (theorems, definitions, exercises,… | Math |
| C | MathConstraintMathConstraint measures LLMs' combinatorial reasoning on solver-verified constraint satisfaction problem… | Reasoning |
| C | MathlibPRMathlibPR measures whether LLMs and LLM agents can judge if Lean Mathlib4 pull requests are merge-ready,… | Software Engineering |
| C | MathTutorBenchMathTutorBench measures the open-ended pedagogical capabilities of LLM tutors in math, such as problem… | Education |
| C | MAVEN-BenchMAVEN-Bench is a stress-test benchmark scoring agents on multi-step mathematical and physical reasoning with… | Tool Use |
| C | MCGAMCGA measures how well multimodal LLMs handle Classical Chinese literary audio across six speech tasks: ASR,… | Audio |
| C | MCJudgeBenchMCJudgeBench measures how correctly and consistently LLM judges verify, constraint by constraint, whether… | Instruction Following |
| C | MCR-BenchMCR-Bench measures the reproducibility of LLM outputs across heterogeneous GPUs on mission-critical tasks. | Efficiency |
| C | MDPBenchMDPBench measures how well models parse digital and photographed document images into structured text across… | Multimodal |
| C | Med-HALTMed-HALT measures hallucination in large language models in the medical domain through reasoning-based and… | Healthcare |
| C | MedBench v5MedBench v5 evaluates clinical multimodal models (language, vision-language, agent systems) with dynamic,… | Biomedicine Healthcare |
| C | MedFrameQAMedFrameQA measures how well multimodal LLMs perform multi-image medical visual question answering requiring… | Biomedicine Healthcare |
| C | Medical Chronology LLM BenchmarkMeasures how well LLMs extract structured medical timelines (encounter dates, facilities, key fields) from… | Healthcare |
| C | Medical Information Response AuditMIRA measures whether LLMs provide comparable medical information across user phrasings varying in language,… | Healthcare |
| C | MedMCQAMedMCQA measures how accurately models answer multiple-choice questions from Indian medical entrance exams… | Healthcare |
| C | MedMemoryBenchMedMemoryBench measures how well agent memory systems store, retrieve and reason over long-horizon… | Healthcare |
| C | MedMetaMedMeta measures how well LLMs synthesize conclusions of medical meta-analyses from the abstracts of the cited… | Biomedicine Healthcare |
| C | MedMosaicMedMosaic measures how well audio and multimodal reasoning models answer medical audio questions, covering… | Biomedicine Healthcare |
| C | MedRedFlagMedRedFlag measures whether LLMs detect and redirect false premises embedded in 1100+ real-world patient… | Healthcare |
| C | MedRepBenchMedRepBench measures how well vision-language models and OCR-plus-LLM pipelines produce structured… | Healthcare |
| C | MedSafe-DxMedSafe-Dx measures whether language models safely escalate synthetic patients needing prompt attention from… | Healthcare |
| C | MedStruct-SMedStruct-S measures how well models perform key discovery, key-conditioned QA and end-to-end key-value… | Biomedicine Healthcare |
| C | MEGA MLQAMLQA measures cross-lingual extractive question answering across 7 languages using aligned Wikipedia-based QA… | Question Answering |
| C | MEGA TyDi QAMEGA TyDi QA measures information-seeking question answering across 11 typologically diverse languages. | Question Answering |
| C | MEGA UDPOSMEGA UDPOS measures how well generative language models perform part-of-speech tagging across many… | Language |
| C | MEGA XCOPAXCOPA measures causal commonsense reasoning across 11 typologically diverse languages, asking models to choose… | Knowledge |
| C | Mem-GalleryMem-Gallery measures how well multimodal LLM agents preserve, reason over, and manage long-term memory across… | Multimodal Understanding |
| C | MEME: Multi-entity & Evolving Memory EvaluationMEME evaluates how well LLM agent memory systems store, update, and reason over multi-entity, evolving… | Agentic |
| C | MemEyeMemEye evaluates whether multimodal agent memory systems preserve fine-grained visual evidence and reason… | Agentic |
| C | Memory-WorldMemory-World evaluates mobile GUI agents on long-horizon tasks that require memorizing and later retrieving… | Mobile |
| C | MEMPROBEMEMPROBE measures how well a memory-equipped LLM agent's long-term memory lets a hidden, structured user state… | Agentic |
| C | MemTraceMemTrace measures how LLM agent long-term memory systems handle individual user facts across memory age,… | Agentic |
| C | Metal-SciMetal-Sci measures how well LLMs, driving an evolutionary search loop, can write and optimize correct, fast… | Coding |
| C | MINDGAMESMINDGAMES is a live multi-game arena that rates LLM agents on social and strategic (theory-of-mind) reasoning… | Reasoning |
| C | MIRAGEMIRAGE measures anti-Muslim bias in frontier LLMs across direct completion, chain-of-thought reasoning, and… | Safety |
| C | Misaligned by RewardMeasures whether reward models prefer socially undesirable responses across bias, safety, morality, and… | Alignment |
| C | ML-BenchML-Bench measures how well guardrail models make multilingual safety judgments grounded in region-specific… | Safety |
| C | MLBMLB evaluates large language models on medical knowledge and scenario-based clinical reasoning across five… | Healthcare |
| C | MLReplicateMLReplicate measures how well autonomous research systems reproduce machine learning research from ICML 2025… | Science |
| C | MLX Benchmark V2MLX Benchmark V2 measures how well language models know Apple's MLX machine learning framework, scored by… | Coding |
| C | MM-MT-BenchMM-MT-Bench evaluates vision-language models on multi-turn, open-ended instruction following in practical… | Multimodal |
| C | MM-OptBenchMM-OptBench measures whether multimodal language models can build mathematical optimization formulations and… | Multimodal |
| C | MMCL-BenchMMCL-Bench measures whether multimodal models can learn task-local rules, procedures, and empirical patterns… | Multimodal |
| C | MMDG-BenchMMDG-Bench measures multimodal domain generalization methods across action recognition, mechanical fault… | Multimodal |
| C | MMed-Bench-IRMMed-Bench-IR measures multilingual medical information retrieval across six languages via cross-lingual… | Retrieval Healthcare |
| C | MMEVerseMMEVerse evaluates multimodal large language models on emotion recognition and reasoning from video, audio and… | Multimodal Understanding |
| C | MMVIADMMVIAD evaluates video multimodal LLMs on industrial inspection clips across anomaly detection, defect… | Domain |
| C | MobileGymMobileGym-Bench measures how well mobile GUI agents complete everyday tasks across 28 simulated mobile apps,… | Mobile |
| C | MolViBench: Evaluating LLMs on Molecular Vibe CodingMolViBench measures how well LLMs generate executable, chemically correct code for molecular tasks across 358… | Coding |
| C | MomentoMomento measures whether agents can complete persistent, tool-mediated tasks in multi-session service… | Agentic |
| C | MomentSeekerMomentSeeker measures how accurately MLLMs and video retrievers can locate key moments in long videos from… | Multimodal |
| C | MonitoringBenchMonitoringBench measures how well AI monitors detect sabotage attacks by tool-using coding agents, using 2,644… | Safety |
| C | MOSAIC-BenchMOSAIC-Bench measures whether coding agents can be induced to ship exploitable code when a malicious objective… | Agentic |
| C | MosaicLeaksMosaicLeaks measures whether deep research agents leak private enterprise information through their external… | Safety |
| C | MotionHallucMotionHalluc measures directional, attributional and temporal motion hallucinations of large multimodal models… | Video |
| C | MOV-BenchMOV-Bench measures Omni-LLMs' ability to perform multi-hop reasoning over temporally dispersed audio and… | Reasoning |
| C | MPCI-BenchMPCI-Bench measures whether multimodal language-model agents respect contextual integrity, distinguishing… | Safety |
| C | Mr DreMr Dre evaluates whether deep research agents can reliably revise long-form research reports across multiple… | Agentic |
| C | MRAGMRAG measures how well retrieval-augmented generation systems answer medical and biomedical questions in… | Biomedicine |
| C | MRI-EvalMRI-Eval measures LLM knowledge of MRI physics and GE scanner operations through tiered multiple-choice and… | Biomedicine Healthcare |
| C | MTAVG-Bench 2.0MTAVG-Bench 2.0 measures how well omni large language models diagnose high-level cinematic failure modes… | Creative Media |
| C | MTBBenchMTBBench measures multimodal LLMs on Molecular Tumor Board style oncology decision-making, requiring… | Healthcare |
| C | Multi-Legal-BenchMulti-Legal-Bench evaluates LLMs on identical legal tasks (court-type classification, judgment form, case… | Legal |
| C | MultiBreakMultiBreak measures LLM safety by scoring attack success rates of 10,389 multi-turn jailbreak prompts spanning… | Safety |
| C | MultiEmo-BenchMultiEmo-Bench measures how well multimodal large language models predict the distribution of emotions evoked… | Multimodal |
| C | Multimodal Claim Extraction for Fact-CheckingMeasures how well multimodal LLMs extract fact-checkable claims from social media posts combining text and… | Factuality |
| C | Multimodal UNcommonsenseMUN evaluates vision-language models' commonsense reasoning on visual scenes paired with surprising or… | Multimodal Understanding |
| C | MultiSessionCollabMultiSessionCollab measures how well conversational agents learn user preferences and use them to improve… | Agentic |
| C | MULTITEXTEDITMULTITEXTEDIT measures cross-lingual degradation in text-in-image editing models across 12 languages, 5 visual… | Multilingual |
| C | MuPHIMuPHI evaluates vision-language models on detecting and reasoning about implicit harm that emerges from… | Safety |
| C | MuPPETMuPPET measures whether LLM assistants leak private information inappropriately in multi-party conversations… | Safety |
| C | MusTBENCHMusTBENCH measures how well large audio-language models ground their answers about music in the correct… | Audio |
| C | MVBenchMVBench measures multi-modal large language models' video understanding across diverse temporal tasks. | Knowledge |
| C | MVRBMVRB measures how well multimodal retrieval models perform visualized information retrieval, where texts,… | Multimodal |
| C | MyPCBenchMyPCBench measures how well computer-use agents act as personal assistants on a Linux desktop with 17… | Agentic |
| C | NARRA-Gym for Evaluating Interactive Narrative AgentsNARRA-Gym measures how well LLMs act as interactive narrative agents, sustaining coherent evolving stories,… | Reasoning |
| C | NarrativeTrackNarrativeTrack measures how well multimodal LLMs understand narratives in videos by tracking entities'… | Video |
| C | NarrativeWorldBenchNarrativeWorldBench measures how well language models maintain narrative structure in long-form serialized… | Creative |
| C | Natural Language to MongoshMeasures how well language models translate natural-language questions into correct, executable mongosh… | Coding |
| C | NatureBenchNatureBench measures whether AI coding agents can match or surpass the published state of the art on 90 real… | Science |
| C | needle-1M-benchneedle-1M-bench measures long-context needle retrieval and faithfulness by having models find planted… | Long Context |
| C | "Newspaper Eat" Means "Not Tasty"Measures how well language models detect, classify, and understand coded language in real-world Chinese Google… | Language-understanding |
| C | NICENICE is a 137-item diagnostic benchmark measuring the social intelligence of LLMs across 4 categories and 11… | Behavioral |
| C | NL-PDDL-BenchNL-PDDL-Bench measures how well language models translate natural-language planning task descriptions into… | Reasoning |
| C | NL2ScratchNL2Scratch measures how well language models generate executable Scratch block-based programs from… | Coding |
| C | NLCONLCO measures LLMs' ability to solve natural-language combinatorial optimization problems end-to-end, scored… | Reasoning |
| C | NoisyBenchNoisyBench evaluates model robustness to noisy contextual distractors (random documents, irrelevant chat… | Reasoning |
| C | NoisyCausalNoisyCausal measures how well LLMs perform causal reasoning in natural language scenarios generated from… | Reasoning |
| C | Not All Proofs Are EqualProofRank measures the quality of LLM-generated mathematical proofs beyond correctness, including conciseness,… | Math |
| C | Not Truly MultilingualPuMVR measures whether vision-language models answer identical visual reasoning questions consistently across… | Multilingual |
| C | NPHardEvalNPHardEval measures LLM reasoning ability on 900 algorithmic questions spanning complexity classes up to… | Knowledge |
| C | NRITYAMNRITYAM measures language models' cultural knowledge and comprehension of global dance traditions through… | Domain |
| C | NSMQ RiddlesNSMQ Riddles measures how well LLMs answer multi-clue scientific and mathematical riddles (biology, chemistry,… | Reasoning |
| C | NuscenenuScenes evaluates 3D object detection and tracking for autonomous driving using a full sensor suite of… | Multimodal |
| C | OBLIQ-BenchOBLIQ-Bench measures how well retrieval systems surface documents for oblique queries that seek latent… | Retrieval |
| C | OCR Bench BritannicaRanks OCR models on Encyclopaedia Britannica document images using pairwise VLM-as-judge comparisons scored… | Multimodal |
| C | OctoBenchOctoBench measures how well LLM coding agents follow scaffold-specified, heterogeneous, persistent… | Instruction Following |
| C | OI-BenchOI-Bench measures how susceptible LLMs are to misleading directive options injected into multiple-choice… | Safety |
| C | OLA: Output Language Alignment in Code-Switched LLM InteractionsCode-switching, alternating between languages within a conversation, is natural for multilingual users, yet… | Multilingual |
| C | OlympiadBenchOlympiadBench measures models' olympiad-level reasoning on 8,476 bilingual (English/Chinese), partly… | Math |
| C | Omni-DeepSearchOmni-DeepSearch measures whether omni-modal models can start from audio clips, use text, image and video… | Agentic |
| C | OmniACTOmniACT measures an agent's ability to generate executable scripts that accomplish computer tasks across… | Agentic |
| C | OmniCleanOmniClean is a visually debiased evaluation set of 8,551 queries from nine omni-modal benchmarks that measures… | Multimodal |
| C | OmniCodeOmniCode evaluates LLM-powered software engineering agents on 1794 tasks across Python, Java and C++ covering… | Software Engineering |
| C | OmniEarth-BenchOmniEarth-Bench measures multimodal models on Earth-science tasks across six spheres and cross-sphere… | Geospatial |
| C | OmniMatBenchOmniMatBench measures multimodal language models' reasoning on 3,171 expert-curated QA and calculation… | Materials |
| C | OmniRef-BenchOmniRef-Bench measures how well image generation models handle complex multi-reference image generation with… | Multimodal |
| C | OmniToMOmniToM measures LLM theory of mind by requiring explicit extraction and labeling of actors' belief… | Reasoning |
| C | On Stable Long-Form GenerationVOLTBench measures the length volatility and length accuracy of LLMs' long-form text generation across… | Long Context |
| C | Oncology Report-Derived VQAMeasures zero-shot vision-language model accuracy on multiple-choice VQA questions derived from private… | Biomedicine Healthcare |
| C | OP-BenchOP-Bench measures over-personalization (irrelevance, repetition, sycophancy) in memory-augmented… | Behavioral |
| C | OpenAI-MRCR: 2 needle 128kOpenAI-MRCR 2 needle 128k measures a model's ability to distinguish between and retrieve one of two similar… | Long Context |
| C | OpenBioRQ: Unsolved Biomedical Research Questions for AgentsOpenBioRQ measures whether agentic models faithfully answer or abstain on 12,553 unsolved biomedical research… | Biomedicine |
| C | OpenClaw Arena Model LeaderboardOpenClaw Arena evaluates frontier models as personal AI agents on 100 real-world tasks, measuring task… | Agentic |
| C | OpenComputerOpenComputer measures how well computer-use agents complete verifiable desktop tasks across 33 real… | Agentic |
| C | OpenExemptOpenExempt is a diagnostic benchmark that measures language models' legal reasoning over U.S. Bankruptcy Code… | Legal |
| C | OpenLearnLM BenchmarkOpenLearnLM Benchmark evaluates LLMs as educational tools across Knowledge, Skills, and Attitude dimensions… | Education |
| C | OpenSkillEvalOpenSkillEval evaluates skill-augmented LLM agent systems and open-source skills on dynamically generated… | Agentic |
| C | OpenSTBenchOpenSTBench evaluates speech translation systems (S2TT and S2ST, offline and streaming) across translation… | Speech |
| C | Opus Magnum BenchMeasures how well AI models solve Opus Magnum campaign puzzles via a Python REPL harness, scored on sum scores… | Game |
| C | ORAgentBenchORAgentBench measures whether LLM agents can solve end-to-end operations research tasks by writing and running… | Reasoning |
| C | OrderProbeOrderProbe measures whether LLMs can reconstruct the canonical order of scrambled four-character expressions… | Language-understanding |
| C | OverEagerOverEager measures how often coding agents perform out-of-scope actions (e.g., leaking credentials, deleting… | Safety |
| C | OverEager-Gen / OverEager-BenchOverEager-Bench measures how often autonomous coding agents take out-of-scope (overeager) actions, such as… | Agentic |
| C | Overlooked Safety Vulnerability in LLMsMalOptBench measures how susceptible LLMs are to malicious intelligent optimization algorithm design requests… | Safety |
| C | P2PCLAW Innovative BenchmarkMeasures the quality of scientific research papers written by AI agents, scored by multiple LLM judges across… | Agentic |
| C | PAL-BenchPAL-Bench measures how well AI systems reconstruct owner profiles, social identities and relations, with… | Multimodal |
| C | PaliBenchPaliBench measures how well large language models translate classical Pali Buddhist texts into English, scored… | Translation |
| C | Parallel WebBenchParallel WebBench measures how well long-horizon web agents complete parallel web-exploration tasks, scoring… | Agentic |
| C | ParaPairAudioBenchParaPairAudioBench measures how reliably large audio-language models acting as judges compare speech pairs… | Audio |
| C | Paraphrase-Induced Output-Mode CollapseMeasures whether LLMs preserve the requested output format and answer consistency when prompts are rewritten… | Instruction Following |
| C | Partial Evidence BenchPartial Evidence Bench measures whether agentic systems recognize and report when material evidence lies… | Agentic |
| C | PashtoTTS-BenchPashtoTTS-Bench evaluates Pashto text-to-speech systems with automated screening: synthesis completion, ASR… | Speech |
| C | PBT-BenchPBT-Bench measures whether AI agents can write property-based tests (Hypothesis strategies) that reveal… | Coding |
| C | PCD BenchmarkMeasures how well LLMs detect causal relations in text and extract exact cause-effect phrase pairs across 12… | Reasoning |
| C | PDEAgent-BenchPDEAgent-Bench measures how well LLMs and code agents generate executable, accurate and efficient numerical… | Coding |
| C | pdfQApdfQA measures how well LLM-based QA pipelines answer human-annotated and synthetic questions over real PDF… | Document Understanding |
| C | PDP-BenchPDP-Bench measures how well LLMs predict prosecutorial decisions (prosecution or one of three non-prosecution… | Legal |
| C | PersLitEvalPersLitEval measures LLM knowledge of Persian literature through 4,514 multiple-choice questions across eight… | Multilingual |
| C | PHBenchPHBench measures how well models, including ML models and zero-shot LLMs, predict whether startups launched on… | Finance Business |
| C | PhiBenchPhiBench is an internal Microsoft benchmark that scores language models on diverse skills and reasoning… | Math |
| C | PhoneSafetyPhoneSafety measures whether phone-use agents take the safe action, the unsafe action, or fail to act at 700… | Safety |
| C | Physical AI Bench Conditional GenerationPhysical AI Bench's conditional generation track evaluates how well video generative models produce physically… | Multimodal |
| C | PinTracePinTrace measures the version-level security and compatibility risk of third-party library versions that LLMs… | Cybersecurity |
| C | PInVerifyPInVerify measures how well embodied agents actively choose viewpoints around a candidate object to verify… | Embodied |
| C | PiSARPiSAR scores how well models predict a user's action and rationale on an app screen, given a persona and… | Agentic |
| C | PitchBenchPitchBench measures how reliably audio-language models perceive absolute and relative musical pitch across… | Audio |
| C | PIVOTSPIVOTS evaluates multimodal LLMs' ability to predict bidirectional fine-grained interpersonal relationship… | Reasoning |
| C | PlaceboBenchPlaceboBench measures hallucinations in LLM answers to real pharmaceutical questions in a RAG setting grounded… | Healthcare |
| C | PlanBench-XLPlanBench-XL measures long-horizon planning of LLM tool-use agents that must retrieve and invoke tools across… | Agentic |
| C | PlotQAPlotQA measures visual question answering over scientific plots (bar, line and dot-line charts), including… | Knowledge |
| C | PluriHarmsPluriHarms measures how well AI safety models and alignment methods predict diverse, often disagreeing human… | Safety |
| C | PluRulePluRule measures whether AI models can moderate pluralistic social media communities by identifying which… | Safety |
| C | PMC-VQAPMC-VQA measures how well models answer questions about medical images, with a manually verified test set and… | Healthcare |
| C | Pocket-DentistPocket-Dentist is an efficiency-aware benchmark for dental multimodal question answering that scores… | Biomedicine Healthcare |
| C | Polaris-BenchPolaris-Bench measures whether multimodal LLMs can do visual reasoning robustly by re-formulating 53 visual… | Spatial Reasoning |
| C | POLECAT-FOR-MEPOLECAT-FOR-ME is a Middle East-focused geopolitical event forecasting benchmark where models predict the… | Geospatial |
| C | PolitNuggetsPolitNuggets measures how well agentic LLM systems discover and synthesize long-tail political facts from… | Agentic |
| C | PopQAPopQA is an open-domain QA dataset of 14k entity-centric questions that probes how well language models… | General Knowledge |
| C | PosIRPosIR measures position bias in embedding-based retrieval models, testing how retrieval quality varies with… | Retrieval |
| C | PostEDA-BenchPostEDA-Bench measures how well LLM agents handle post-tool-run EDA tasks: fixing sign-off DRC violations and… | Agentic |
| C | PrecisionMemBenchPrecisionMemBench measures the retrieval precision of LLM memory systems independently of generative models,… | Retrieval |
| C | PrepBenchPrepBench evaluates LLM agents on natural-language-driven data preparation, covering interactive… | Data |
| C | PreScamPreScam measures whether language models can predict how real-world conversational scams progress, via… | Safety |
| C | PrivacyPeekPrivacyPeek measures acquisition-stage privacy leakage of LLM-based tool-using agents, i.e. whether they… | Agentic |
| C | PrivacyQAPrivacyQA measures question answering over mobile app privacy policies, comparing model answers against… | Legal |
| C | PrivacySIMPrivacySIM measures how accurately LLMs conditioned on user personas simulate individual users' privacy… | Behavioral |
| C | ProcCtrlBenchProcCtrlBench evaluates LLM coding agent trajectories for process-level execution defects and control… | Coding |
| C | Proof of TimeProof of Time measures how well models and tool-using agents judge scientific ideas by forecasting post-cutoff… | Science |
| C | ProofNetProofNet measures autoformalization and formal theorem proving of undergraduate-level mathematics in Lean,… | Math |
| C | PseudoBenchPseudoBench measures whether agentic auto-research systems can identify and resist pseudoscientific narratives… | Safety |
| C | PsychEthicsBenchPsychEthicsBench measures LLMs' ethical knowledge and behavioral responses in mental health against Australian… | Healthcare |
| C | QMFOLBenchQMFOLBench measures LLM deductive reasoning on monadic first-order logic tasks with controllable complexity in… | Reasoning |
| C | QMSumQMSum measures query-based summarization of long multi-party meeting transcripts, where models must select and… | Summarization |
| C | QSTRBenchQSTRBench measures how well large language models reason with qualitative spatial and temporal calculi,… | Spatial Reasoning |
| C | QUACKQUACK evaluates multimodal VLM agents in a social deduction game at the level of game outcomes, behavioral… | Game |
| C | QuantEvalQuantEval measures LLM ability in quantitative finance across knowledge QA, quantitative mathematical… | Finance |
| C | QuantSightBenchQuantSightBench measures whether LLMs can produce calibrated 90% prediction intervals for 1,000 real-world… | Finance |
| C | QuestBenchQuestBench measures how well deep research systems answer 256 student-designed expert-level questions across… | Question Answering |
| C | QUIETQUIET measures LLM creative generation capability via multi-blank cascaded story cloze tasks scored… | Creative |
| C | Re$^2$MathRe2Math measures whether models can retrieve and verify applicable theorems from the mathematical literature… | Retrieval |
| C | RealDataAgentBenchRealDataAgentBench scores LLM agents on data-science tasks (EDA, feature engineering, modeling, statistical… | Agentic |
| C | ReasonAudioReasonAudio measures how well text-audio retrieval models handle reasoning-intensive queries involving… | Audio |
| C | RecoAtlasRecoAtlas measures how well LLM shopping recommendation agents produce item sets and justifications, using… | Agentic |
| C | REDACTREDACT measures how well detectors and LLMs identify personally identifiable information across 51 entity… | Data |
| C | RedVoxRedVox measures safety and fairness of speech-capable models on unsafe and stereotypical spoken requests from… | Safety |
| C | Reinforcing Human Behavior Simulation via Verbal FeedbackSOUL measures how well LLMs simulate human-like behavior across 10 tasks spanning theory of mind, role play,… | Behavioral |
| C | ReplaySCMReplaySCM measures whether language models can induce executable Boolean structural causal mechanisms from… | Reasoning |
| C | RepoBenchRepoBench measures repository-level code auto-completion, covering cross-file retrieval, next-line code… | Coding |
| C | RepoMirageRepoMirage measures whether code agents can reason over repository context, using semantics-preserving… | Software Engineering |
| C | Rethinking Continual Anomaly Detection on the EdgeA benchmark evaluating continual anomaly detection methods for industrial inspection under discrete-task and… | Vision |
| C | Rethinking Evaluation for LLM Hallucination DetectionTRIVIA+ is a RAG-based, long-context hallucination detection benchmark with human annotations and noisy label… | Factuality |
| C | Rethinking Reasoning-Intensive RetrievalBRIGHT-Pro, introduced in this paper, measures how well retrievers surface multi-aspect evidence that supports… | Retrieval |
| C | Reward Hacking BenchmarkReward Hacking Benchmark measures how often tool-using LLM agents exploit shortcuts such as skipping… | Safety |
| C | RFC BenchRFC Bench evaluates large language models on detecting financial misinformation in realistic paragraph-level… | Finance |
| C | RGBRGB measures how well large language models perform retrieval-augmented generation across noise robustness,… | Generalization |
| C | RMCBRMCB measures how well confidence estimation methods predict the correctness of long-form reasoning traces… | Reasoning |
| C | RoadmapBenchRoadmapBench measures coding agents' ability to implement long-horizon, multi-target software development… | Software Engineering |
| C | Roboflow Vision Evals - DetectionMeasures how well vision models localize objects by drawing bounding boxes from text queries, scored by… | Vision |
| C | Roboflow Vision Evals - Visual UnderstandingMeasures how well AI vision models answer real-world visual QA tasks such as reading text from photos,… | Vision |
| C | ROK-FORTRESSROK-FORTRESS measures LLM safety on national security and public safety adversarial prompts across… | Safety |
| C | RP-BenchRP-Bench measures how well LLMs perform in roleplay, scoring character consistency, user agency respect,… | Creative |
| C | RTV-BenchRTV-Bench measures multimodal LLMs' continuous perception, understanding and reasoning on real-time video… | Multimodal |
| C | RubricEvalRubricEval measures how accurately LLM judges make rubric-level judgments when evaluating… | Instruction Following |
| C | RubRIXRubRIX is a clinician-validated rubric framework that scores risks (inattention, bias, inaccuracy, uncritical… | Safety |
| C | RuleSafe-VLRuleSafe-VL measures whether vision-language models apply explicit platform moderation policy rules correctly,… | Safety |
| C | SafeClawBenchSafeClawBench measures the security of tool-using LLM agents against 600 adversarial tasks (prompt injection,… | Safety |
| C | SAKESAKE measures LLMs' software architectural knowledge with 2154 expert-curated multiple-choice questions across… | Software Engineering |
| C | Sand-GlassSand-Glass evaluates LLM translation under syllable-level duration constraints, such as for subtitling and… | Translation |
| C | SastBench: A Benchmark for Testing Agentic SAST TriageSastBench measures how well LLM agents triage SAST tool findings, distinguishing real vulnerabilities (CVEs)… | Cybersecurity |
| C | ScaleWoBScaleWoB measures how well GUI agents complete verifiable tasks in synthesized interactive app environments,… | GUI |
| C | scBench-LongscBench-Long measures whether AI agents can recover scientific conclusions from raw or near-raw single-cell… | Biology |
| C | ScholarQuestScholarQuest measures how well LLM-based search agents retrieve sets of relevant academic papers for… | Agentic |
| C | SciIntegrity-BenchSciIntegrity-Bench measures whether AI scientist systems maintain academic integrity, honestly acknowledging… | Safety |
| C | SciPaths: Forecasting Pathways to Scientific DiscoverySciPaths measures whether language models can forecast discovery pathways: identifying the enabling… | Reasoning |
| C | SearchGEOSearchGEO measures how often LLM-based web-search agents are manipulated by attacker-published web content… | Agentic |
| C | SECQUESECQUE measures LLM financial analysis capabilities with 565 expert-written questions on SEC filings covering… | Finance |
| C | Seeing Isn't KnowingSeeing Isn't Knowing (SpatialUncertain) measures whether vision-language models abstain on spatial questions… | Spatial Reasoning |
| C | Seeing vs. BelievingSeeing vs. Believing (CAIT) measures whether multimodal LLMs correctly understand 400 synthetic… | Multimodal Understanding |
| C | Seneca-TRBenchSeneca-TRBench measures LLM proficiency in Turkish using multiple-choice structural-linguistics questions and… | Language |
| C | SEQUORSEQUOR measures how well models follow user constraints throughout long multi-turn conversations, including… | Instruction Following |
| C | SGR-BenchSGR-Bench measures how well search agents find the right specialized data website and configure its… | Agentic |
| C | SkillEvolBenchSkillEvolBench measures whether LLM agents can distill episodic trajectories into reusable procedural skills,… | Agentic |
| C | SkillGenBenchSkillGenBench measures how well LLM agent skill-generation pipelines produce correct, reusable, executable… | Agentic |
| C | SkillSafetyBenchSkillSafetyBench measures whether LLM agents (CLI agents) behave unsafely when adversarial influence is placed… | Safety |
| C | SLDBenchSLDBench measures whether AI agents can discover accurate scaling law formulas from experimental LLM training… | Agentic |
| C | SmartHome-BenchSmartHome-Bench evaluates LLM agents on multi-turn smart home control tasks in a simulated home environment,… | Agentic |
| C | SMDD-BenchSMDD-Bench measures how well LLM agents solve multi-turn, long-horizon small molecule drug design tasks such… | Biomedicine |
| C | SmellBenchSmellBench measures how well LLM agents repair architectural code smells in a Python codebase (scikit-learn),… | Software Engineering |
| C | SNN Fairness BenchmarkA benchmark that evaluates fairness of spiking neural networks under biased training data, spurious features,… | Alignment |
| C | Social CaptionSocial Caption evaluates multimodal LLMs' social understanding of human interactions across social inference,… | Multimodal Understanding |
| C | Social IQaSocial IQa measures commonsense reasoning about social situations through 38,000 multiple-choice questions… | Knowledge |
| C | SocialMemBenchSocialMemBench measures how well AI memory systems recall, attribute and track facts across multi-party social… | Agentic |
| C | SocialPersonaSocialPersona measures whether multimodal LLMs can infer users' revealed preferences from longitudinal… | Agentic |
| C | SoohakSoohak measures LLMs' research-level mathematical problem solving on 439 mathematician-authored problems,… | Math |
| C | SoundnessBenchSoundnessBench measures whether LLMs can judge the methodological soundness of machine-learning research… | Science |
| C | SpatialActSpatialAct measures whether vision-language model agents can perform action-conditioned spatial reasoning in… | Spatial Reasoning |
| C | SpatialBabelSpatialBabel evaluates vision-language models on reconstructing 3D scenes from geometric primitives as code… | Spatial Reasoning |
| C | SpatialBench-LongSpatialBench-Long measures whether AI agents can recover biological claims from raw or near-raw spatial… | Biology |
| C | SpreadsheetBench V1SpreadsheetBench V1 measures LLM agents' ability to manipulate complex real-world spreadsheets using 912… | Spreadsheets |
| C | SpurAudioSpurAudio measures how much few-shot audio classification methods rely on spurious correlations between… | Audio |
| C | SQuIDSQuID measures quantitative spatial reasoning of vision-language models on 2,000 satellite image… | Geospatial |
| C | SREGymSREGym measures how well AI agents diagnose and mitigate injected failures in live cloud-native production… | Agentic |
| C | STAGE-EvalSTAGE-Eval measures how accurately models extract information from long unstructured reports into structured… | Structured Output |
| C | STaRKSTaRK measures how well LLM-driven retrieval systems retrieve correct items from semi-structured knowledge… | Retrieval |
| C | StatABenchStatABench measures LLMs' statistical analysis capabilities through closed-form questions across 18… | Data |
| C | STEBSTEB evaluates Chinese–English speech-to-speech translation systems on translation fidelity, speaker… | Translation |
| C | STELLASTELLA measures how well embedding models retrieve aerospace technical documents (from NASA NTRS), separately… | Aerospace |
| C | StereoSetStereoSet measures stereotypical bias in pretrained language models. | Generalization |
| C | StreamPro-BenchStreamPro-Bench evaluates streaming video models on perception understanding, temporal reasoning, and… | Video |
| C | StreamProfileBenchStreamProfileBench measures how well LLMs infer and continuously update fine-grained user profiles from… | Behavioral |
| C | StressEvalStressEval is a failure-driven dynamic benchmarking framework that synthesizes challenging, controllable test… | Reasoning |
| C | StrongREJECTStrongREJECT measures jailbreak effectiveness by scoring how much useful forbidden information a model gives… | Safety |
| C | StructBreakStructBreak measures how structural cognitive overload attacks trigger safety failures (toxic generation) in… | Safety |
| C | StylisticBiasStylisticBias measures attribute-level social bias in multimodal LLMs by testing how single visual attribute… | Behavioral |
| C | SummScreenFDSummScreenFD measures abstractive summarization of long TV series transcripts into human-written plot recaps. | Summarization |
| C | SVAMPSVAMP is a 1,000-problem challenge set of elementary math word problems that tests whether models solve them… | Math |
| C | SVHallucSVHalluc measures whether audio-visual LLMs hallucinate by failing to ground speech content in visual… | Multimodal |
| C | SWE-MutationSWE-Mutation measures whether LLM-generated test suites for software engineering tasks can detect… | Software Engineering |
| C | SWE-PRBenchSWE-PRBench measures whether LLMs can identify the same issues human reviewers flagged on 350 real merged pull… | Coding |
| C | SynBio-ReasonSynBio-Reason measures how well models generate pysbol3 Python code to repair and design genetic circuits in… | Biology |
| C | TAEFTAEF measures LLMs' semantic understanding and reasoning over large aggregated collections of social media… | Question Answering |
| C | TangramPuzzleTangramPuzzle measures multimodal LLMs' compositional spatial reasoning through tangram-based outline… | Spatial Reasoning |
| C | TeachObsTeachObs measures how well vision-capable LLMs code observable teaching practices in classroom video scenes… | Education |
| C | TeamBenchTeamBench measures how well multi-agent teams coordinate across Planner, Executor, and Verifier roles under… | Agentic |
| C | TECCITECCI measures how well text-guided image editing models handle challenging edits, judged on instruction… | Vision |
| C | TeleResilienceBenchTeleResilienceBench measures LLMs' reasoning resilience in telecommunications: their ability to continue and… | Reasoning |
| C | TempCompassTempCompass measures the temporal perception ability of Video LLMs across temporal aspects such as speed and… | Multimodal |
| C | TempGlitchTempGlitch measures how well vision-language models detect temporal glitches in gameplay videos, using… | Game |
| C | TERMS-BenchTERMS-Bench evaluates LLM agents in simulator-based bilateral price negotiation, scoring surplus extraction,… | Agentic |
| C | Text2Opt-BenchText2Opt-Bench measures how well models translate natural-language problem descriptions into correct,… | Reasoning |
| C | ThaiSafetyBenchThaiSafetyBench measures LLM safety against 1,954 malicious Thai-language prompts, including Thai culturally… | Safety |
| C | The Age of Curiosity Meets the Age of AIKIDBench measures how safely and age-appropriately LLMs respond to child queries (ages 7-11) across ten… | Safety |
| C | The Agent CompanyTheAgentCompany measures how well LLM agents autonomously complete realistic professional tasks in a simulated… | Agentic Business |
| C | The Complexity Ceiling BenchmarkCCB measures how LLM sequential reasoning accuracy decays as the number of required reasoning steps grows… | Reasoning |
| C | The Compliance GapThe Compliance Gap paper releases BS-Bench, which measures whether AI models actually follow process… | Alignment |
| C | The Compliance TrapMeasures whether frontier AI models maintain metacognitive stability (refusing unanswerable questions,… | Safety |
| C | The Harder Text Embedding Benchmark (HTEB)Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static,… | Language |
| C | The Metanym GameThe Metanym Game is a competitive word game benchmark in which LLMs create analogies and rate each other's… | Reasoning |
| C | ThreatCoreThreatCore measures how well models detect fine-grained threats in text, distinguishing explicit threats,… | Safety |
| C | TIDE-BenchTIDE-Bench evaluates tool-integrated reasoning methods in language models across math reasoning,… | Tool Use |
| C | Time PuzzlesTime Puzzles measures LLMs' iterative temporal reasoning with tools such as web search, via algorithmically… | Tool Use |
| C | Time to REFLECTREFLECT is a meta-evaluation benchmark measuring how reliably LLM judges detect fine-grained process- and… | Agentic |
| C | Timely-EvalTimely-Eval measures how well LLM agents adapt their test-time strategies to wall-clock time budgets across… | Agentic |
| C | TimeSage-MTTimeSage-MT measures whether LLM agents can conduct reliable multi-turn time series analysis, from basic… | Agentic |
| C | ToolAlpacaToolAlpaca evaluates language models' generalized tool-use ability, scoring procedure and response accuracy… | Agentic |
| C | ToolPrivacyBenchToolPrivacyBench measures whether tool-using LLM agents complete tasks while routing private information only… | Tool Use |
| C | ToolQAToolQA measures how well large language models answer questions that require using external tools over text,… | Agentic |
| C | TraceEvalTraceEval measures whether LLMs can recover a program's runtime call structure (call graph) from source code… | Reasoning |
| C | Translation en→Set1 COMET22Measures machine translation quality from English into a set of target languages (Set1), scored with the… | Translation |
| C | Translation en→Set1 spBleuMeasures machine translation quality from English into a set of target languages using spBLEU (SentencePiece… | Translation |
| C | Translation Set1→en COMET22Measures how well models translate text from a set of source languages into English, scored with the COMET-22… | Translation |
| C | Translation Set1→en spBleuMeasures machine translation quality into English from a set of source languages using spBLEU… | Translation |
| C | TriggerBenchTriggerBench measures LLMs' prospective memory: spontaneously recalling and acting on latent constraints in… | Reasoning |
| C | TRIP-BenchTRIP-Bench measures how well LLM agents handle long-horizon, multi-turn travel-planning dialogues with many… | Agentic |
| C | ts-benchts-bench measures how well AI coding agent and model combinations solve TypeScript tasks (Exercism exercises… | Coding |
| C | TS-SkillTS-Skill measures how well LLMs and time-series language models answer time-series questions requiring… | Reasoning |
| C | TSCognitionTSCognition measures LLMs' multimodal time series reasoning across five cognitive tasks (Decoding, Grounding,… | Reasoning |
| C | TSM-BenchTSM-Bench evaluates machine-generated text detectors on task-specific LLM-generated text from real-world… | Writing |
| C | TukaBenchTukaBench measures LLM safety against jailbreak prompts in seven African languages, including culturally… | Safety |
| C | TVWorld: Foundations for Remote-Control TV AgentsTVWorld measures how well vision-language agents navigate and ground UI elements on real-world TV interfaces… | Embodied |
| C | TW-LegalBench: Measuring Taiwanese Legal UnderstandingTW-LegalBench measures LLMs' understanding of Taiwanese law through official exam multiple-choice questions,… | Legal |
| C | TxBench-PPTxBench-PP measures whether AI agents can recover accurate small-molecule preclinical pharmacology conclusions… | Biomedicine |
| C | UA-Legal-BenchUA-Legal-Bench measures how well large language models perform Ukrainian legal reasoning tasks, using court… | Legal |
| C | UCSF-PDGM-VQAUCSF-PDGM-VQA measures how well vision-language models answer clinically relevant questions about brain tumor… | Biomedicine Healthcare |
| C | Ukrainian Legal Text BenchmarkIt measures tokenizer fertility and zero-shot performance of foundation models on three tasks over validated… | Legal |
| C | UltraEval-AudioUltraEval-Audio is a unified evaluation framework with public leaderboards that scores audio foundation models… | Audio |
| C | UniREditBenchUniREditBench measures how well image editing models handle reasoning-based edits across real-world and… | Multimodal |
| C | Urban-ImageNetUrban-ImageNet evaluates how well vision and vision-language models perceive urban spaces from social media… | Geospatial |
| C | USAMO25USAMO25 scores models on the six proof-based problems of the 2025 USA Mathematical Olympiad. | Math |
| C | UXBenchUXBench measures how reliable and actionable LLM-generated UX critiques of runnable web interfaces are, scored… | GUI |
| C | V4FinBenchV4FinBench measures how well tabular models, foundation models and LLMs predict corporate bankruptcy across… | Finance |
| C | Vaani-Benchmark-V1.0Vaani-Benchmark-V1.0 measures Hindi automatic speech recognition accuracy by multi-reference word error rate… | Speech |
| C | VBVR-BenchVBVR-Bench evaluates video generation models' reasoning capabilities across abstraction, knowledge,… | Multimodal |
| C | VCG-BenchVCG-Bench measures vision-language models' ability to generate and edit structured diagrams as mxGraph XML… | Structured Output |
| C | VELI5VELI5 measures how well factuality correction methods fix systematically injected factual errors in long-form… | Factuality |
| C | VERA-MHVERA-MH evaluates the safety of chatbots responding to users with suicidal ideation risk in mental health… | Safety Healthcare |
| C | VERHalluVERHallu measures event relation hallucination (causal, temporal, subevent) in video large language models via… | Video |
| C | VeriContestVeriContest measures how well models generate verifiable Rust/Verus code, including formal specifications,… | Software Engineering |
| C | Verified MisguidanceMeasures structural citation failures in search-augmented LLMs, scoring each citation on intent-purpose… | Factuality |
| C | VeriTripVeriTrip measures how well multimodal travel planning agents retrieve and reason over noisy, unstructured… | Agentic |
| C | Verus-SpecBenchVerus-SpecBench measures whether LLM agents can translate informal Codeforces programming problems into… | Software Engineering |
| C | Vibe-EvalVibe-Eval measures multimodal chat models on 269 expert-authored visual understanding prompts, including a… | Multimodal |
| C | VIBenchVIBench measures vertical integration bias, whether LLMs favor their provider's own ecosystem, in direct and… | Coding |
| C | VibeSearchBenchVibeSearchBench measures how well LLM search agents handle long-horizon, multi-turn proactive search with… | Search |
| C | VideoFDBVideoFDB measures how well full-duplex audio-visual conversational agents perceive and generate nonverbal cues… | Multimodal |
| C | VideoNetVideoNet measures vision-language models' ability to recognize domain-specific actions in videos, covering… | Video |
| C | ViDoRe V3 PipelineViDoRe V3 Pipeline evaluates full retrieval-augmented generation pipelines on visually rich multi-domain… | Retrieval |
| C | VietMed-MCQVietMed-MCQ measures LLM knowledge and diagnostic reasoning in Vietnamese Traditional Medicine through 3,190… | Healthcare |
| C | VisAnomBenchVisAnomBench measures how well vision-language models localize anomalies in time-series data and explain them… | Data |
| C | Vision-DeepResearch BenchmarkVDR-Bench measures how well multimodal LLM deep-research systems use visual and textual search to answer 2,000… | Multimodal Understanding |
| C | Vision2CodeVision2Code measures whether vision-language models can generate executable code that reconstructs source… | Coding |
| C | VisIT-Bench Single ImageVisIT-Bench measures how well vision-language models follow real-world instructions about images, scored… | Multimodal |
| C | VisualNeedleVisualNeedle measures whether multimodal LLMs can perform active, fine-grained visual search in… | Vision |
| C | VisualWebArenaVisualWebArena measures how well multimodal web agents complete realistic, visually grounded tasks on websites… | Agentic |
| C | VNU-BenchVNU-Bench measures how well multimodal large language models understand news videos across multiple sources,… | Video |
| C | VoiceAgentBenchVoiceAgentBench evaluates speech language models and ASR-LLM pipelines on spoken agentic tasks: tool… | Speech |
| C | VotIE: Information Extraction from Meeting MinutesVotIE measures how well models extract structured voting events from free-form Portuguese municipal meeting… | Document Understanding |
| C | VPCTVPCT scores models on visual perception and comprehension tasks, ranking multimodal models on a leaderboard. | Multimodal |
| C | V-STaRV-STaR measures how well Video-LLMs perform sequential spatio-temporal reasoning in videos, evaluating what… | Multimodal |
| C | VT-BenchVT-Bench evaluates models on visual-tabular multi-modal discriminative prediction and generative reasoning… | Multimodal |
| C | VULCA-BenchVULCA-Bench evaluates vision-language models on multicultural art critique, scoring cultural understanding… | Multimodal Understanding |
| C | VulnLLM-RVulnLLM-R measures how accurately LLMs detect function-level vulnerabilities in white-box C, Java and Python… | Cybersecurity |
| C | WatchActWatchAct measures whether vision-language planners and robot policies can perform manipulation tasks grounded… | Robotics |
| C | WeatherBenchWeatherBench measures how well data-driven models forecast global medium-range weather, scored by Z500 and… | Weather |
| C | WeatherBench 2WeatherBench 2 evaluates and compares data-driven and traditional numerical global weather forecasting models… | Weather |
| C | WebCoderBenchWebCoderBench measures how well LLMs generate web applications from 1,572 real user requirements, scored with… | Coding |
| C | WebGameBenchWebGameBench measures whether coding agents can turn a structured game specification into a working… | Software Engineering |
| C | WebGymWebGym is a large open-source environment of nearly 300,000 rubric-evaluated tasks on real websites for… | Agentic |
| C | WebIGBenchWebIGBench evaluates multimodal LLMs on generating code for complex interactive webpages from visual designs,… | Coding |
| C | WebMainBenchWebMainBench measures how accurately extractors and model-backed pipelines pull main content from web pages,… | Multimodal |
| C | WebTrap ParkWebTrap Park measures the security of web agents by observing their actions on live web pages across 1,226… | Cybersecurity |
| C | When (and How) to Trust the ExpertA benchmark comparing query-time expert-guided reinforcement learning methods on continuous-control tasks with… | Robotics |
| C | When Cases Get RareOGCaReBench evaluates LLMs on free-form, retrieval-focused answering of rare, off-guideline clinical questions… | Biomedicine Healthcare |
| C | When Context FlipsSafety benchmark scores provide incomplete evidence of deployment readiness: aligned language models often… | Safety |
| C | When LLMs Imagine PeopleAn auditing benchmark (Persona Brainstorm Audit) that scores LLMs for social bias across 16 identity… | Safety |
| C | When Lower Privileges SufficeToolPrivBench, introduced in the paper When Lower Privileges Suffice, measures whether LLM agents select or… | Safety |
| C | When Multiple Scripts MatterMultiClin measures automatic speech recognition accuracy on non-English clinical doctor–patient dialogues,… | Speech Healthcare |
| C | When Reviews DisagreeRevCI, introduced in the When Reviews Disagree paper, measures how well models identify contradiction evidence… | Language-understanding |
| C | When Safe Skills CollideMeasures compositional security risk in LLM agent skill ecosystems: whether individually safe skills combine… | Agentic |
| C | When Search Agents Should AskDiscoBench measures whether LLM search agents can detect ambiguity in vague or incorrect queries, ask… | Agentic |
| C | When the Manual LiesMeasures how vulnerable LLM agents are to Tool Description Poisoning attacks hidden in MCP tool metadata,… | Cybersecurity |
| C | WHOOPS Explanation of ViolationWHOOPS Explanation of Violation measures whether vision-language models can identify and explain why a… | Multimodal |
| C | WideSearchWideSearch measures how reliably LLM-powered search agents collect large-scale atomic information across many… | Search |
| C | Wiki Live ChallengeWiki Live Challenge measures how well deep research agents generate Wikipedia Good Article-quality reports,… | Agentic |
| C | WildAgtEvalWildAGTEval evaluates LLM agents' function-calling capabilities under realistic API complexity, including… | Agentic |
| C | WildGraphBenchWildGraphBench measures how well GraphRAG and RAG systems retrieve and aggregate evidence from long,… | Retrieval |
| C | WMW (World Models in Words)Vision-language models (VLMs) are increasingly used to answer questions about physical scenes, yet most… | Reasoning |
| C | WorldArena Track 2WorldArena evaluates embodied world models on perceptual video quality and functional utility in embodied… | Multimodal |
| C | WorldLinesWorldLines measures how well embodied household-assistant agents use long-term memory of routines, world… | Embodied |
| C | WorldMemArenaWorldMemArena measures how well multimodal agents write, maintain, retrieve and use memory across 400… | Agentic |
| C | WorldScoreWorldScore measures world generation models (3D, 4D scene and video generation) on controllability, quality… | Multimodal |
| C | Wow-wo-valWow-wo-val is an Embodied Turing Test benchmark that evaluates video foundation models as world models for… | Embodied |
| C | XCR-BenchXCR-Bench measures LLMs' cross-cultural reasoning by testing identification, prediction and adaptation of… | Reasoning |
| C | XDomainBenchXDomainBench measures LLMs' interactive interdisciplinary scientific reasoning across 20 domains, diagnosing… | Reasoning |
| C | Xent GamesXent Games are cross-entropy-based single- and multi-player games over LLM probability measures, assembled… | Game |
| C | XL-SafetyBenchXL-SafetyBench measures LLM safety across 10 country-language pairs, scoring jailbreak robustness against… | Safety |
| C | XSTestXSTest measures exaggerated safety behaviour in language models, i.e. whether they wrongly refuse clearly safe… | Safety |
| C | Agentic Commerce World (ACWorld) BenchmarkACWorld Benchmark evaluates Buyer and Merchant agents conducting commerce tasks (product discovery,… | |
| C | BFI-AdaptBFI-Adapt scores whether event-induced Big Five personality shifts in personality-conditioned LLM agents align… | |
| C | Canary ToolsCanary Tools measures how LLM agents reason about tool selection by planting diagnostic decoy tools in an MCP… | |
| C | CircuitReason-1kCircuitReason-1k measures whether multimodal models can solve 1,000 textbook electrical circuit problems from… | |
| C | FormalRx-TestFormalRx-Test is a fine-grained diagnostic benchmark scoring how well models judge, categorize, localize and… | |
| C | IABench-CAIABench-CA measures how deployment rules for allocating consequences causally shape collective safety behavior… | |
| C | LLMForge evaluation benchmarkLLMForge evaluates foundation models on generating parametric CAD designs of mechanical parts from… | |
| C | LongVQUBenchLongVQUBench measures how well vision-language models understand video quality and distortions in… | |
| C | MatPhaseBenchMatPhaseBench evaluates vision-language models on open-ended understanding and description of materials phase… | |
| C | MentalHospitalMentalHospital evaluates LLMs on complete simulated psychiatric clinical encounters (interviewing,… | |
| C | MIOHMIOH measures object hallucination in multimodal LLMs across multi-image scenarios, covering existence,… | |
| C | P-BenchP-Bench measures whether LLM agents can perform statistically valid hypothesis testing — selecting a method,… | |
| C | PolicyShiftBenchPolicyShiftBench measures whether vision-language models and image guardrails correctly judge if an image… | |
| C | RAVEN-EvalRAVEN-Eval evaluates AI video generation models on 150 text-to-video and 100 image-to-video tasks using… | |
| C | SCHEDBenchSCHEDBench measures whether LLMs produce constraint-feasible, optimal schedules for combinatorial scheduling… | |
| C | Sci-VBenchSci-VBench measures how well video generation models produce scientifically accurate, knowledge- and… | |
| C | SciMIFSciMIF measures how well multimodal large language models follow complex, constraint-laden instructions on… | |
| C | Spider 2.0-AIFuncSpider 2.0-AIFunc measures whether language models can generate AI-native SQL queries that use Snowflake… | |
| C | TeXFix-BenchTeXFix-Bench measures how well LLMs repair broken LaTeX, Typst, and Markdown document sources so they compile… | |
| C | VisEditBenchVisEditBench measures how well vision-language models edit visualization code from multimodal feedback,… | |
| C | Bias in the Ear of the ListenerBiasInEar measures speech bias and robustness of multilingual audio language models on 11,200 spoken… | Audio |
| C | Enhancing Agent Safety JudgmentIt measures how well LLMs judge the safety of tool-using agent trajectories rewritten into deceptive,… | Safety |
| C | ChartREG++ChartREG++ measures how well multimodal models ground referring expressions to chart elements across multiple… | Document Understanding |
| C | MedHal-LocMedHal-Loc measures whether medical hallucination detectors faithfully localize the erroneous span in clinical… | Healthcare |
| C | One Instruction Does Not Fit AllMeasures how well multilingual embedding models align personas and instructions across 12 Indian languages via… | Multilingual |
| C | RSPCRSPC measures how well models classify psychiatric disorders, relational stress triggers, and relationship… | Biomedicine Healthcare |
| D | BrowseComp-VLBrowseComp-VL evaluates multimodal agents on deep web-research tasks that combine visual perception with… | Agentic |
| D | AIME 2023AIME 2023 measures models' mathematical problem-solving on 15 competition problems from the AIME 2023 I exam,… | Math |
| D | OSWorld 2.1OSWorld 2.1 measures how well AI agents complete long-horizon computer-use tasks across desktop apps, mocked… | Agentic |
| D | OSWorld 2.1 OfflineMeasures computer-use agents on real desktop OS and app tasks run offline, scored by whether each task ends in… | Agentic |
| D | TAU-benchTAU-bench measures how reliably language agents use domain-specific API tools and follow policy guidelines in… | Agentic |
| D | Tax Agent BenchTax Agent Bench measures how well AI agents answer research-grade US corporate tax questions through… | Agentic |
| D | BrowseComp ExtendedBrowseComp Extended measures agents' ability to persistently navigate the web to locate hard-to-find,… | Agentic |
| D | Terminal-Bench 2.1 ExtendedTerminal-Bench 2.1 Extended measures how well models complete tasks in command-line terminal environments,… | Agentic |
| D | AA LiveCodeBenchAA LiveCodeBench is Artificial Analysis's independently run LiveCodeBench evaluation scoring models' code… | Coding |
| D | HumanEvalHumanEval measures the functional correctness of model-generated Python code by running completions against… | Coding |
| D | LiveCodeBench v6LiveCodeBench v6 scores models on 1055 code generation problems released between May 2023 and April 2025. | Coding |
| D | LiveCodeBench v5LiveCodeBench v5 evaluates LLMs on contest-style competitive programming problems collected August 2024 to… | Coding |
| D | LiveCodeBench ProLiveCodeBench Pro measures LLM performance on continuously updated competitive programming problems from… | Coding |
| D | SWE-Bench Pro V2 FullSWE-Bench Pro V2 Full measures AI agents' resolve rate on long-horizon software engineering tasks drawn from… | Coding |
| D | SWE-Bench Pro V2 HARDSWE-Bench Pro V2 HARD is the hard split of Scale's refreshed SWE-Bench Pro public benchmark, measuring the… | Coding |
| D | SWE-bench Verified ExtendedSWE-bench Verified Extended measures whether models can edit real Python repositories to resolve GitHub… | Coding |
| D | ARC-AGI-3 Provider AdapterARC-AGI-3 Provider Adapter scores models on ARC-AGI-3 interactive reasoning game environments using a harness… | Reasoning |
| D | HLE-DiamondHLE-Diamond measures model accuracy on 1,000 refined Humanity's Last Exam questions testing expert knowledge… | Knowledge |
| D | AA-HLEAA-HLE is Artificial Analysis's run of Humanity's Last Exam, scoring models on 2,158 text-only questions… | Knowledge |
| D | Chinese-SimpleQAChinese SimpleQA measures how factually accurate language models are when answering short Chinese-language… | Knowledge |
| D | HLE w/o toolsHLE w/o tools scores models on Humanity's Last Exam questions answered without tool access, measuring raw… | Knowledge |
| D | SimpleQA VerifiedSimpleQA Verified measures LLM short-form factuality and parametric knowledge on 1,000 human-crafted questions… | Knowledge |
| D | HLE ExtendedHLE Extended tests models on frontier-level, closed-ended academic questions across many disciplines that… | Knowledge |
| D | AA Global-MMLU-LiteAA Global-MMLU-Lite measures models' multilingual MMLU-style knowledge and reasoning across STEM, humanities… | Multilingual |
| D | AA AIME 2025AA AIME 2025 is Artificial Analysis's independent evaluation of models on the 30 integer-answer AIME I and II… | Math |
| D | FrontierMath v2FrontierMath v2 is Epoch AI's corrected benchmark of private, advanced research-level mathematics problems… | Math |
| D | HLE-MultiChoiceHLE-MultiChoice measures model accuracy on 513 text-only multiple-choice questions from Humanity's Last Exam,… | Decision Models |
| D | FHIR-AgentBenchFHIR-AgentBench measures how well LLM agents answer real-world clinical questions by retrieving and reasoning… | Healthcare |
| D | SWE-bench DockerSWE-bench Docker is a Dockerized evaluation tool for SWE-bench that scores whether model-generated patches… | Coding |
| D | SWE-bench JavaScriptSWE-bench JavaScript evaluates software-engineering agents on realistic issue resolution, repository… | Coding |
| B | SWE Atlas: Codebase QnA, Test Writing, Refactoring | Coding Computer Use+2 |
| D | Adversarial Customer ServiceMeasures whether a model acting as a bank support agent can distinguish genuine customers from identity… | Agentic |
| D | ExtractBenchExtractBench measures how accurately models and agents perform schema-guided extraction from enterprise… | Multimodal |
| D | FACTS Benchmark SuiteFACTS Benchmark Suite measures the factual accuracy of LLMs across grounding in documents, image-based… | Knowledge |
| D | Artificial Analysis Decision IndexThe Artificial Analysis Decision Index scores decision models (and LLMs) on structured single-call decisions… | Decision Models |
| D | Beyond AIMEBeyond AIME measures difficult mathematical reasoning requiring deeper reasoning chains and harder problem… | Math |
| C | Tuesday Work Index leaderboard | Enterprise Workflows Long-Horizon |
| D | HANDBOOK.mdHANDBOOK.md measures long-context agentic instruction following, where agents must apply company handbook… | Agentic |
| D | Hermes IndexHermes Index averages model scores across four agentic benchmarks (Hermes Bench, Terminal-Bench 4.0,… | Agentic |
| D | Vals Science IndexVals Science Index averages three agentic science benchmarks (BioMysteryBench, MysteryMechanism,… | Agentic |
| D | FACTS Search V2FACTS Search V2 measures how well models use a shared web search tool (Brave Search API) to retrieve and… | Agentic |
| D | STEMBenchSTEMBench is a private benchmark from Ulam that scores models on STEM tasks such as engineering and astronomy. | Reasoning |
| D | Human Behavior BenchHuman Behavior Bench measures how accurately AI models predict response distributions of specific U.S. adult… | Reasoning |
| D | Game ArenaGame Arena ranks AI models by their play in strategic games such as chess, Go, poker and Werewolf, using a… | Reasoning |
| D | ExtractBench ProExtractBench Pro measures schema-guided document extraction on 100 synthetic long-tail enterprise documents… | Multimodal |
| D | MGSM BengaliMGSM Bengali measures mathematical reasoning on grade-school math word problems presented in Bengali. | Math |
| D | ClaimBenchClaimBench measures whether a model can tell if a 10-K financial document excerpt supports, contradicts, or… | Decision Models |
| D | JudgeBenchJudgeBench measures how well LLM-based judges and reward models identify the objectively correct response in… | Decision Models |
| D | When2CallWhen2Call measures LLM tool-calling decision-making: whether to call a tool, ask a follow-up question, answer… | Decision Models |
| D | WildGuardTestWildGuardTest measures how well LLM safety moderation models detect harmful prompts, harmful responses, and… | Decision Models |
| B | Vals IndexVals Index is a GDP-weighted composite benchmark scoring AI models on real-world finance, coding, legal, and… | Cross-domain |
| C | Andon FMAndon FM tests whether AI agents can autonomously run live radio stations around the clock, covering music… | Computer Use Long-Horizon |
| C | ShelfLife (measured 2026-08-13): | Coding Enterprise Workflows+1 |
| C | ShelfLife E-Sim (measured 2026-08-26): | Coding Enterprise Workflows+1 |
| D | ALUEALUE benchmarks LLMs on aviation and aerospace-specific language tasks such as hazard classification,… | Aerospace |
| D | TEAChTEACh evaluates embodied agents on completing household tasks in simulation using dialogue history, language… | Robotics |
| D | XTREMEXTREME measures cross-lingual generalization of multilingual representations across 40 languages and 9 tasks… | Multilingual |
| D | APPSAPPS measures a model's ability to generate Python code from natural language problem specifications, checked… | Coding |
| D | FLUEFLUE (Financial Language Understanding Evaluation) is a suite of benchmarks scoring language models on five… | Finance |
| D | FormalizeLarge Language Models (LLMs) struggle to solve complex combinatorial problems through direct reasoning, so… | Coding |
| D | GIMGIM measures how well language models integrate multiple cognitive operations (constraint satisfaction, state… | Reasoning |
| B | HARPHARP measures how perturbations or attacks on individual agents in multi-agent LLM systems amplify into… | Safety |
| B | MUSEMUSE measures how well LLMs generate manufacturable, functional, and assemblable multi-part CAD assemblies… | Alignment |
| D | SURESURE is a unified evaluation framework that scores speech understanding systems, from conventional pipelines… | Speech |
| D | TRIAGETRIAGE measures LLMs' prospective metacognitive control: selecting, ordering, and allocating a token budget… | Agentic |
| D | EQ-Bench 4EQ-Bench 4 measures active emotional and social intelligence of models in 16-turn multi-turn roleplay chats… | Agentic |
| D | AA-AnalystAgentAA-AnalystAgent measures how reliably AI agents answer quantitative analyst questions from real-world… | Agentic |
| D | CTI-REALMCTI-REALM measures AI agents' ability to interpret cyber threat intelligence reports and develop working… | Agentic |
| D | AI4AI-BenchAI4AI-Bench measures whether LLM agents can redesign the training algorithm inside frozen AI research… | Agentic |
| D | ApprenticeBenchApprenticeBench measures whether a computer-use agent can learn a real accounts-payable job end to end,… | Agentic |
| D | DiG-benchDiG-bench measures AI agents' ability to discover hidden rules and unknown win conditions through interaction… | Agentic |
| D | Bug Hunt BenchBug Hunt Bench measures how many of 105 planted real bugs in two production codebases frontier coding models… | Coding |
| D | DSBench-HardDSBench-Hard is DeepSeek's internal hard benchmark measuring complex coding-agent capabilities. | Coding |
| D | APEX-SWE (Terminus-2)APEX-SWE (Terminus-2) measures models on software integration and production-failure diagnosis tasks run with… | Coding |
| D | APEX-SWE (Inspect)APEX-SWE (Inspect) measures models on software integration and production-failure diagnosis tasks run with the… | Coding |
| D | CVDPCVDP evaluates LLMs and agents on RTL hardware design, verification, and comprehension problems in Verilog. | Coding |
| D | FACTS Grounding V1FACTS Grounding V1 measures whether model long-form responses are factually grounded in a supplied context… | Reasoning |
| D | Flame-VLM-CodeFlame-VLM-Code scores vision-language models on generating correct code, reported as a percentage, with GLM 5V… | Multimodal |
| D | A-OKVQAA-OKVQA measures visual question answering on about 25K questions that require commonsense and world knowledge… | Multimodal |
| S | AA Openness IndexA composite index scoring how open AI models are, based on weights availability, licensing, and transparency… | Knowledge |
| S | Artificial Analysis Intelligence IndexA composite index from Artificial Analysis that aggregates ten evaluations into a single 0-100 score measuring… | Knowledge |
| D | DrugDiscoveryBenchDrugDiscoveryBench measures AI agents on 82 real drug-discovery and life-sciences research tasks run in a… | Knowledge |
| C | FORTRESSFORTRESS measures the robustness of LLM safeguards against national-security and public-safety risks using 500… | Knowledge |
| D | AdvancedIFAdvancedIF measures LLMs' ability to follow complex, multi-turn, and system-level instructions, graded with… | Instruction Following |
| D | BRUMO 2025BRUMO 2025 measures LLMs' ability to solve 30 olympiad-style competition math problems with verifiable final… | Math |
| D | AIME26AIME26 scores models on the 30 answer-checked problems of the 2026 American Invitational Mathematics… | Math |
| D | APEX-1APEX-1 measures whether frontier models can perform economically valuable single-turn professional text tasks… | Cross-domain |
| D | AA-PersonalityBenchAA-PersonalityBench measures language models' default personality across 30 traits and their communication… | Cross-domain |
| D | LiveClawClaw-Eval-Live (LiveClaw) scores AI agents on completing realistic multi-step enterprise workflows across… | Agentic |
| D | When2Call Multi-TurnWhen2Call Multi-Turn measures whether an assistant selects the correct response policy as a conversation… | Agentic |
| D | RelyToolBenchRelyToolBench measures tool hallucination in LLM agents, scoring pass rate, tool hallucination rate and… | Agentic |
| D | ExplorationBench AlienLogicTests whether a system can discover altered logic rules through exploration and use them to prove or correctly… | Agentic |
| D | MCP-Atlas claim coverageMCP-Atlas claim coverage measures the mean coverage of required claims in model answers when discovering and… | Agentic |
| D | LAB all-passLAB all-pass measures the share of long-horizon legal agent tasks where every expert-curated rubric criterion… | Agentic |
| D | LAB criterion-passLAB criterion-pass measures the mean fraction of expert-written rubric criteria that AI agents pass across… | Agentic |
| D | Toolathlon Verified avg. turnsMeasures the average number of assistant turns models take per trajectory on Toolathlon Verified, a set of 108… | Agentic |
| D | Design Arena Agentic Web DevDesign Arena Agentic Web Dev ranks models by blinded head-to-head human preference on multi-file web… | Agentic |
| D | Harvey LAB-AA v1.1Harvey LAB-AA v1.1 measures whether AI legal agents complete realistic legal tasks satisfying every rubric… | Agentic |
| D | Harvey LAB-AA v1.1 Criterion PassMeasures the share of legal-deliverable rubric criteria a model passes on Harvey LAB-AA v1.1 tasks across 24… | Agentic |
| D | AA Tau3 BankingAA Tau3 Banking measures conversational banking agents that retrieve policy from a large document corpus,… | Agentic |
| D | AA Cyber IndexAA Cyber Index scores AI models on security-relevant tasks such as vulnerability analysis, exploit reasoning… | Agentic |
| D | CWE-Bench-AACWE-Bench-AA is Artificial Analysis' independent run of Collinear AI's CWE-bench, which measures whether AI… | Agentic |
| D | DeepsecBench-AADeepsecBench-AA measures how well AI agents find security vulnerabilities in a given codebase, scored against… | Agentic |
| D | CyberGym-E2E-AACyberGym-E2E-AA measures AI agents' end-to-end cyber defense capabilities on memory-safety tasks, from… | Agentic |
| D | b3 Agent Securityb3 (backbone breaker benchmark) measures how well backbone LLMs inside AI agents resist crowdsourced… | Agentic |
| D | Cyber Misuse RefusalCyber Misuse Refusal measures how often a model refuses malicious cybersecurity-related requests. | Agentic |
| D | Toolathlon-VerifiedToolathlon-Verified measures language agents' general tool use on long-horizon tasks across 600+ tools in… | Agentic |
| D | Harvey Legal AgentHarvey Legal Agent scores AI agents on legal-practice tasks, grading their deliverables against per-task… | Agentic |
| D | Kimi Hallu Bench HardAn internal Moonshot AI stress test of factual accuracy that measures models' claim-level hallucination rate… | Agentic |
| D | τ²-bench resultsτ²-bench measures how well conversational AI agents resolve customer-service tasks in airline, retail and… | Agentic |
| D | τ²-bench Airlineτ²-bench Airline measures whether a model can act as an airline customer-support agent, following a policy… | Agentic |
| B | OpenHands IndexOpenHands Index is a continually updated leaderboard evaluating language models in the OpenHands agent on… | Agentic |
| D | SWE Refactor BenchSWE Refactor Bench measures whether coding agents can complete whole-repository stack migrations, checking… | Agentic |
| D | MM-ClawBenchMM-ClawBench scores multimodal agents on everyday daily tasks. | Agentic |
| D | QwenWebBenchQwenWebBench is an internal Qwen benchmark scoring front-end code generation (web design, apps, games, SVG,… | Agentic |
| D | CoWorkBenchCoWorkBench measures how well models complete long-running office tasks across fields such as computer… | Agentic |
| D | ReviewBenchReviewBench measures how well AI code review agents find true, relevant issues in real GitHub pull requests,… | Agentic |
| D | Ramp Accounting BenchRamp Accounting Bench measures frontier models on 137 long-horizon accounting tasks across simulated… | Agentic |
| D | Visual Tool BenchVisualToolBench (VTB) is a leaderboard benchmark that scores AI models, out of 100, on visual tasks that… | Agentic |
| D | SciUniverse Level 1SciUniverse Level 1 measures whether AI models can carry out real-world scientific research across biology,… | Agentic |
| D | Parallel Search CapabilityA leaderboard by Parallel Web Systems ranking how well models use Parallel's search and extract tools to… | Agentic |
| D | Vals Router IndexVals Router Index compares model routers against single models on long, tool-using agentic tasks in finance,… | Agentic |
| D | Vals Legal IndexVals Legal Index scores models on legal research, legal drafting/review (HLAB-Vals-1.1) and tax agent tasks,… | Agentic |
| D | Vals Inference Provider IndexCompares the same open-weight models served by different inference providers against the creator's own API on… | Agentic |
| D | Vals Finance IndexThe Vals Finance Index scores models on agentic finance work by combining Finance Agent v2.1, EMB v1.1 and Tax… | Agentic |
| D | xdev-bench-hardxdev-bench-hard measures a model agent's ability to discover and exploit real, recently disclosed software… | Agentic |
| D | The Last Ones completionMeasures whether AI models can complete a 32-step cyber range simulation (The Last Ones) end to end, reported… | Agentic |
| D | SEC-Bench ProSEC-bench Pro measures how well frontier models and coding agents find real vulnerabilities by reproducing… | Agentic |
| D | CyScenarioBench solvedCyScenarioBench solved counts the number of cybersecurity scenarios a model completes successfully in at least… | Agentic |
| D | Atomic network attacksAtomic network attacks scores models on simulated network attack tasks as part of a cybersecurity capability… | Agentic |
| D | SCONE simulated revenueMeasures the simulated value (revenue) captured by models exploiting smart-contract vulnerabilities in the… | Agentic |
| D | Anthropic OSS-Fuzz crashAnthropic's internal OSS-Fuzz-derived evaluation measuring the fraction of roughly 830 entry points where the… | Agentic |
| D | Anthropic OSS-Fuzz write primitiveAnthropic's internal OSS-Fuzz-derived evaluation measuring the share of entry points where a model achieves a… | Agentic |
| D | CVE-Bench zero-dayCVE-Bench zero-day mode measures whether AI models can discover and then exploit unknown vulnerabilities in… | Agentic |
| D | RuneScape-BenchRuneScape-Bench measures how well AI coding agents play RuneScape gameplay tasks by writing TypeScript against… | Agentic |
| D | Agent Poker BenchAgent Poker Bench measures how well AI models, acting as agents, play poker against each other, scored by… | Agentic |
| D | Public Benefits Bench v1.1Measures whether general-purpose models can correctly answer SNAP food-assistance benefits questions across… | Agentic |
| D | MysteryMechanismMysteryMechanism measures whether models can rediscover hidden mathematical mechanisms through mathematical… | Agentic |
| D | GDPval ExtendedGDPval Extended evaluates models on producing real-world, economically valuable knowledge-work deliverables… | Agentic |
| B | Harvey LABHarvey's Legal Agent Benchmark (LAB) measures how well AI agents complete long-horizon legal tasks from… | Agentic |
| D | PostTrainBench v1.1PostTrainBench measures how well CLI AI agents can post-train small base language models within ten hours on… | Coding |
| D | VulcanBench v3VulcanBench v3 measures model accuracy, cost, tokens, and time on real software engineering tasks graded by… | Coding |
| D | OpenHarmony BenchOpenHarmony Bench measures whether LLM-based coding agents can implement app-level changes in buildable… | Coding |
| D | VulcanBench CII v1VulcanBench CII v1 evaluates coding agents on post-cutoff open-source software engineering tasks using hidden… | Coding |
| D | FrontierCode 1.1 ExtendedFrontierCode 1.1 Extended measures models on real-world software engineering tasks, grading mergeability and… | Coding |
| D | FrontierCode 2.0FrontierCode 2.0 measures whether coding agents produce mergeable code changes on real open-source repository… | Coding |
| D | ScarfBenchScarfBench measures whether coding agents can perform behavior-preserving migrations of enterprise Java… | Coding |
| D | SWE MultimodalA multimodal variant of SWE-bench that adds visual context such as screenshots and design mockups to software… | Coding |
| D | Kimi Code Bench v2Kimi Code Bench v2 is Moonshot AI's coding-agent benchmark used to score models on coding and agent… | Coding |
| D | Kimi Code Bench 3.0 Long-horizonKimi Code Bench 3.0 Long-horizon is Moonshot AI's internal benchmark measuring coding agents' multi-step… | Coding |
| D | PostTrainBench v1.2PostTrainBench v1.2 measures how well AI agents can post-train small base language models on one H100 GPU in… | Coding |
| D | QwenReactBenchQwenReactBench is Qwen's internal benchmark measuring React application generation, reported as a BT/Elo… | Coding |
| D | DSBench-FullStackDSBench-FullStack is DeepSeek's internal agent benchmark that scores models on complete full-stack software… | Coding |
| D | Next.js EvalsNext.js Evals measures how well AI coding agents build and migrate Next.js apps, scored by withheld assertions… | Coding |
| D | Spider 2.0-LiteSpider 2.0-Lite measures how well language models translate natural-language questions into SQL over… | Coding |
| D | AA Coding IndexThe Artificial Analysis Coding Index scores models on coding as a weighted average of coding benchmarks such… | Coding |
| D | SVG-BenchSVG-Bench (svgbench.ai) is a blind arena benchmark where people vote between pairs of model-generated SVG… | Coding |
| D | BridgeBenchBridgeBench rates frontier AI models on vibe-coding and software-building axes such as reasoning, front end,… | Coding |
| D | IOI v1IOI v1 measures how well AI agents solve International Olympiad in Informatics programming problems in C++,… | Coding |
| D | Vibe Code Bench 1-100Measures whether models can extend a working web application through a sequence of up to ten dependent… | Coding |
| D | SWE Atlas - Codebase QnA ExtendedMeasures coding agents on codebase question answering, test writing, and refactoring in software repositories,… | Coding |
| D | Web Clone BenchWeb Clone Bench measures how well coding agents can build functional, visually faithful replicas of nine… | Coding |
| D | SciCode Main StandardMeasures language models' ability to solve 80 core expert-level scientific research coding problems from… | Coding |
| D | SciCode Main With BackgroundMeasures models' ability to write code solving 80 core scientific research problems when scientist-curated… | Coding |
| D | SciCode Subproblem StandardMeasures models' ability to solve decomposed scientific research coding subproblems curated by scientists,… | Coding |
| D | SciCode Subproblem With BackgroundMeasures models' ability to write code solving 338 decomposed scientific research subproblems when relevant… | Coding |
| D | LongMemEvalLongMemEval measures chat assistants' long-term memory, testing information extraction, multi-session… | Reasoning |
| D | GraphWalks BFS 256K–1MGraphWalks BFS 256K-1M measures whether a model can perform breadth-first search over graphs encoded as edge… | Reasoning |
| D | Conceptual ReasoningThe Conceptual Reasoning Index aggregates three benchmarks (LMCA, ACCoRD, DTBench) measuring how well models… | Reasoning |
| D | Pencil Puzzle BenchPencil Puzzle Bench measures large language models' multi-step reasoning by having them solve pencil puzzles… | Reasoning |
| D | MRCR v2 64K-128KMRCR v2 64K-128K measures a model's ability to retrieve information from long contexts in the 64K-128K token… | Reasoning |
| D | MRCR v2 128K-256KMRCR v2 128K-256K measures long-context retrieval, scoring models on multi-round coreference resolution over… | Reasoning |
| D | MRCR v2 256K-512KMRCR v2 256K-512K measures a model's ability to retrieve multiple relevant items (eight needles) placed within… | Reasoning |
| D | MRCR v2 512K-1MMRCR v2 512K-1M measures a model's ability to retrieve information across very long contexts of 512K to 1M… | Reasoning |
| D | AI-NeedleNEEDLE is a suite of search and retrieval benchmarks from Keenable.ai that evaluates search engines on query… | Reasoning |
| D | MLCR-AAMLCR-AA measures how well models reason over and synthesize long, fragmented medical case files (25k–64k… | Reasoning |
| D | BullshitBench v2BullshitBench v2 measures whether AI models detect and reject nonsensical prompts instead of confidently… | Reasoning |
| D | PropensityBenchPropensityBench measures LLMs' propensity to choose misaligned, high-risk tools under operational pressure… | Reasoning |
| D | NoLiMaNoLiMa measures long-context retrieval in LLMs using needle-in-a-haystack tests where questions and needles… | Reasoning |
| D | NoLiMa-HardNoLiMa-Hard measures long-context retrieval accuracy of (mainly reasoning) models on the 10 most challenging… | Reasoning |
| D | Long Context Reasoning (AA-LCR) ExtendedMeasures models' ability to answer hard questions requiring synthesis and reasoning across multiple long… | Reasoning |
| D | Long Context Reasoning (AA-LCR v1.1)AA-LCR measures models' ability to answer hard questions requiring synthesis and reasoning across multiple… | Reasoning |
| D | FACTS Grounding v2FACTS Grounding v2 measures whether a model's long-form responses to user requests about a supplied context… | Reasoning |
| D | OfficeQA Pro v2OfficeQA Pro v2 measures model and agent performance on grounded, multi-document retrieval and numerical… | Multimodal |
| D | GDP.pdf w/ PythonGDP.pdf w/ Python measures professional document understanding, where the model receives each original PDF in… | Multimodal |
| D | MMMUMMMU measures multimodal models on 11.5K college-level exam questions requiring subject knowledge and… | Multimodal |
| D | olmOCRolmOCR-bench measures how accurately OCR systems convert PDF documents to markdown, preserving text, tables,… | Multimodal |
| D | VoxPopuli WERVoxPopuli WER measures speech-recognition accuracy as word error rate on European Parliament speech clips from… | Multimodal |
| D | Design Arena WebsiteDesign Arena Website is a crowdsourced, vote-based Elo leaderboard ranking AI models on generating websites… | Multimodal |
| D | MathVision w/ PythonMathVision w/ Python measures visual mathematical reasoning where the model may use Python as a tool. | Multimodal |
| D | WorldVQA ForceAnswerWorldVQA ForceAnswer measures a model's accuracy on visual question answering grounded in real-world knowledge… | Multimodal |
| D | We-MathWe-Math measures how large multimodal models solve visual math problems, decomposing composite problems into… | Multimodal |
| D | RefCOCORefCOCO measures referring expression comprehension: whether a model can localize the image object described… | Multimodal |
| D | ImageMiningImageMining evaluates multimodal agents on combining high-density visual understanding with multi-step… | Multimodal |
| D | MMSearchMMSearch evaluates how well large multimodal models perform as multimodal AI search engines, via requery,… | Multimodal |
| D | Facts-VLMFacts-VLM is a public leaderboard benchmark scoring vision-language models on grounded multimodal factuality… | Multimodal |
| D | CharXiv w/o toolsCharXiv w/o tools scores models on visual reasoning over scientific charts without tool use, as tracked on a… | Multimodal |
| D | OmniDocBench 1.6OmniDocBench 1.6 measures how accurately models parse diverse documents (text, formulas, tables, reading… | Multimodal |
| B | AudioMCAudioMultiChallenge measures whether spoken dialogue systems and audio-language models can follow… | Multimodal |
| D | Humanity's Sixth SenseHSS measures intuitive visual reasoning in multimodal models: inferring implicit temporal, spatial, social,… | Multimodal |
| D | ViDoRe V3 ImageViDoRe V3 Image measures retrieval quality (nDCG@10) of embedding models over visual document images given… | Multimodal |
| D | ViDoRe V3 MarkdownViDoRe V3 Markdown measures retrieval quality (nDCG@10) over specialized industry document corpora in English… | Multimodal |
| B | Vals Multimodal IndexA GDP-weighted composite index scoring models on image-inclusive economic tasks across finance, coding, and… | Multimodal |
| D | VoxelBench TextA human-preference leaderboard where language models build voxel structures from text prompts and voters… | Multimodal |
| D | VoxelBench ImageVoxelBench is a human-preference Elo leaderboard that rates how well models build 3D Minecraft-like voxel… | Multimodal |
| D | CharXiv ExtendedMeasures multimodal models' understanding of real arXiv research-paper charts through descriptive and… | Multimodal |
| D | MedXpertQA MM ExtendedMeasures advanced medical reasoning via multimodal multiple-choice questions over clinical images and patient… | Multimodal |
| D | MMMU-Pro ExtendedMMMU-Pro Extended measures joint visual and textual reasoning on multimodal questions, with text-only-solvable… | Multimodal |
| D | GaRAGeGaRAGe measures whether LLMs ground long-form RAG answers in the relevant supplied passages, attribute sources… | Knowledge |
| D | Protein DesignEvaluates protein foundation models on protein design tasks such as generating sequences and structures,… | Knowledge |
| D | Morphology-to-molecule matchingMeasures how well models match multi-channel cell imaging data from a human liver in-vitro assay to the… | Knowledge |
| D | Protein Design library rankingMeasures how well models rank a library of protein designs to prioritize them for experimental (wet-lab)… | Knowledge |
| D | FrontierScienceFrontierScience measures expert-level scientific reasoning of frontier language models on olympiad-level and… | Knowledge |
| D | AA-Omniscience AccuracyAA-Omniscience Accuracy measures the percentage of knowledge questions, spanning science, software… | Knowledge |
| D | Kimi Hallu BenchKimi Hallu Bench is Moonshot AI's internal benchmark that stress-tests factual accuracy, scoring models by… | Knowledge |
| D | FrontierScience ResearchFrontierScience Research evaluates models on open-ended PhD-level scientific research subtasks in physics,… | Knowledge |
| D | MultiLoKoMultiLoKo measures LLMs' multilingual local knowledge across 31 languages, with locally sourced questions plus… | Knowledge |
| D | FACTS ParametricFACTS Parametric measures language models' world knowledge and factuality by having them answer closed-book… | Knowledge |
| D | Immigration BenchImmigration Bench measures how well AI models answer realistic, attorney-written U.S. immigration law… | Knowledge |
| D | SUPERChemSUPERChem measures expert-level, multi-step multimodal chemical reasoning of LLMs on 500 expert-curated… | Knowledge |
| D | VWT2k-liteVWT2k-lite, now cataloged as a Qwen provider comparison on WMT24++, measures machine translation quality… | Multilingual |
| D | NanoBEIR MultilingualNanoBEIR Multilingual measures text retrieval quality of embedding/retriever models across multiple languages… | Multilingual |
| D | MKQA-11MKQA-11 measures cross-lingual open-domain QA retrieval quality (Recall@20) of retrieval models across 11… | Multilingual |
| D | WMT26WMT26 General MT task evaluates machine translation systems' translation quality across many language pairs,… | Multilingual |
| D | Brevity BenchmarkBrevity Benchmark scores conversational responses on both answer quality and conciseness relative to… | Instruction Following |
| D | RoleBenchRoleBench measures large language models' character-level role-playing abilities, such as imitating a given… | Instruction Following |
| D | RoleMRCRoleMRC measures how well LLMs follow diverse, nested and prioritized instructions while maintaining a defined… | Instruction Following |
| D | IHEvalIHEval measures whether language models follow the instruction hierarchy (system messages, user messages,… | Instruction Following |
| D | InverseIFEvalInverseIFEval measures whether language models can follow counter-intuitive instructions that conflict with… | Instruction Following |
| D | SOB Value AccSOB Value Acc measures structured-output value accuracy: whether a model's extracted JSON values exactly match… | Instruction Following |
| D | MASKMASK measures whether large language models lie under pressure, disentangling honesty from accuracy using a… | Instruction Following |
| D | ACCR Daybreak BlueMeasures the share of approved advanced-cyber requests that GPT-5.6 Sol completes rather than refuses under… | Instruction Following |
| D | ArXivMath Jun. 2026ArXivMath Jun. 2026 measures models on final-answer research mathematics problems drawn from June 2026 arXiv… | Math |
| D | ArXivMath Aug. 2026ArXivMath measures models on research-level mathematics problems, reported as a percentage score. | Math |
| D | CMathCMATH measures large language models' ability to solve 1.7k Chinese elementary school math word problems,… | Math |
| D | AIME25AIME25 measures mathematical reasoning using the 30 problems from the 2025 American Invitational Mathematics… | Math |
| D | HMMT Feb 2023HMMT Feb 2023 scores models on 30 competition math problems from the February 2023 Harvard-MIT Mathematics… | Math |
| D | HMMT Feb 2024HMMT Feb 2024 scores models on competition-level math problems from the February 2024 Harvard-MIT Mathematics… | Math |
| D | HMMT Feb 2025HMMT Feb 2025 scores models on short-answer problems from the February 2025 Harvard–MIT Mathematics… | Math |
| D | IPhO 2025IPhO 2025 evaluates models on the three official theory problems of the 2025 International Physics Olympiad,… | Math |
| D | HMMT Nov 2025Measures advanced mathematical reasoning on 30 competition problems from the HMMT November 2025 high school… | Math |
| D | HMMT Feb 2026HMMT Feb 2026 scores models on problems from the February 2026 Harvard-MIT Mathematics Tournament, measuring… | Math |
| D | MMAnswerBenchMMAnswerBench is a math benchmark on which language models are scored by answer accuracy, with results shown… | Math |
| D | Cloudflare Decision Index 0.2.1Cloudflare's Decision Index is an evaluation suite scoring decision models on typed classification and… | Decision Models |
| D | FinancialPhraseBankFinancialPhraseBank measures whether models can classify the sentiment of English financial news sentences as… | Decision Models |
| D | TruthfulQA binaryTruthfulQA binary measures whether language models pick the truthful answer over an imitative falsehood when… | Decision Models |
| D | Decision Index 0.2.1Decision Index 0.2.1 is a leaderboard that scores models on decision tasks such as classification and routing… | Decision Models |
| D | CLadderCLadder measures whether language models can perform formal causal inference (associational, interventional,… | Decision Models |
| D | Decision Index Full ScoreDecision Index 0.4 publishes full scores for 209 evaluated systems. The full score weights private tests of… | Decision Models |
| D | Decision Index KnowledgeDecision Index 0.4 publishes chance-corrected knowledge scores for 209 evaluated systems. | Decision Models |
| D | Decision Index LanguageDecision Index 0.4 publishes chance-corrected language scores for 209 evaluated systems. | Decision Models |
| D | Decision Index RetrievalDecision Index 0.4 publishes chance-corrected retrieval scores for 209 evaluated systems. | Decision Models |
| D | KMMLUKMMLU measures Korean-language expert-level knowledge with 35,030 multiple-choice questions from original… | Korean |
| D | KMMLU-HardKMMLU-Hard is a 4,104-question subset of the Korean expert-level multiple-choice KMMLU benchmark containing… | Korean |
| D | KMMLU-ReduxKMMLU-Redux measures LLMs' Korean expert-level and industrial knowledge using cleaned questions from the… | Korean |
| D | KoBALTKoBALT measures advanced Korean linguistic competence of LLMs via 700 expert-curated multiple-choice questions… | Korean |
| D | HRM8KHRM8K measures mathematical reasoning in Korean with 8,011 problems drawn from Korean exams and competitions… | Korean |
| D | Mental Health BenchMental Health Bench scores how AI models respond in 1,215 synthetic multi-turn mental-health conversations,… | Cross-domain |
| D | AA-SciCodeAA-SciCode is Artificial Analysis's run of SciCode, scoring models' pass@1 on 288 scientific coding subtasks… | Coding |
| D | Korean CSATMeasures LLM performance on ten years of Korean CSAT Korean-language section exams, scored with official… | Korean |
| D | Cohere Embed 5 launch: FinanceBench text retrievalMeasures embedding models' retrieval quality (RCP-nDCG@10) on FinanceBench financial document retrieval as… | Cross-domain |
| D | Cohere Embed 5 launch: FinQA text retrievalFinQA text retrieval measures how well embedding models retrieve relevant financial document passages for… | Cross-domain |
| D | Cohere Embed 5 launch: Chinese retrieval compositeA Cohere-reported composite measuring embedding models' document retrieval quality on Chinese-language queries… | Cross-domain |
| D | A Human-Centric Pipeline for Aligning Large Language Models with Chinese Medical EthicsMedES is a scenario-centric benchmark that scores LLMs on Chinese medical ethics through objective and… | Alignment |
| D | ActivityNet-QAActivityNet-QA measures how well video question answering models answer human-written questions about long web… | Knowledge |
| D | Adaptive Hierarchical Evaluation of LLMs and SAST tools for CWE Prediction in PythonALPHA is a function-level Python benchmark scoring LLMs and SAST tools on CWE-specific vulnerability… | Cybersecurity |
| D | ADR: An Agentic Detection System for Enterprise Agentic AI SecurityWe present the Agentic AI Detection and Response (ADR) system, the first large-scale, production-proven… | Cybersecurity |
| D | AgentCARDAgentCARD is a role-aware benchmark suite that evaluates the cost and accuracy of LLM agent teams across role… | Agentic |
| D | AgentKernelArenaAgentKernelArena measures AI coding agents on GPU kernel optimization across 196 HIP, Triton and… | Agentic |
| D | AgentOdysseyAgentOdyssey evaluates test-time continual learning agents in procedurally generated open-ended long-horizon… | Agentic |
| D | AgentQuestAgentQuest is a modular benchmark framework that measures LLM agent progress on multi-step reasoning tasks… | Agentic |
| A | AIIQ Composite IQAIIQ Composite IQ estimates an IQ-like score for AI models by mapping public benchmark results into calibrated… | Knowledge |
| D | AIRS-BenchAIRS-Bench measures the autonomous machine learning research abilities of LLM agents across 20 tasks sourced… | Agentic |
| D | AMC_2022_23AMC_2022_23 scores models on American Mathematics Competition problems from the 2022-23 year, testing… | Math |
| D | AmharicStoryQAAmharicStoryQA measures LLMs' long-sequence narrative understanding in Amharic through question answering over… | Multilingual |
| D | An Empirical Study of Proactive Coding Assistants in Real-World Software DevelopmentProCodeBench, introduced in this study, measures how well models predict latent developer intent from real IDE… | Coding |
| D | An explainable hypothesis-driven approach to Drug-Induced Liver Injury with HADESThe DILER Benchmark evaluates models on predicting drug-induced liver injury and generating mechanistic… | Biomedicine |
| D | Android in the WildAndroid in the Wild measures how well device-control agents follow natural language instructions by operating… | Mobile |
| D | APEX-v1-extendedAPEX-v1-extended measures whether frontier models can perform economically valuable expert tasks in four jobs:… | Agentic Business |
| D | AppWorldAppWorld measures how well autonomous agents complete day-to-day digital tasks by writing interactive code… | Agentic |
| D | ArchBench ADR GenerationArchBench ADR Generation measures how well LLMs generate Architecture Decision Records from given decision… | Coding |
| D | ArchBench Dynamic Service GenerationArchBench's Dynamic Service Generation task scores how well LLMs and coding agents generate services… | Coding |
| D | ArchBench Microservice GenerationArchBench's Microservice Generation task measures how well LLMs and coding agents generate microservice… | Coding |
| D | Are Agents Ready to Teach? A Multi-Stage Benchmark for Real-World Teaching WorkflowsEduAgentBench measures how well tutor agents handle the full scope of teaching work: pedagogical judgment,… | Education |
| D | Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR AssistancePhysAssistBench measures how well LLMs assist physicians in interactive, multi-turn doctor-patient-EHR… | Healthcare |
| A | Arena AI WebDev ArenaWebDev Arena is a crowdsourced human-preference leaderboard ranking AI models on building front-end web… | Coding |
| B | Arena AI DocumentDocument Arena is a crowdsourced pairwise human-preference leaderboard ranking AI models on document analysis… | Document-ai |
| A | Arena AI Image EditA crowdsourced human-preference leaderboard ranking AI models on single- and multi-image editing tasks. | Multimodal |
| A | Arena AI Text-to-ImageA crowdsourced pairwise human-preference leaderboard ranking text-to-image generation models by Arena score… | Multimodal |
| B | Arena AI Text-to-VideoA crowdsourced pairwise human-preference leaderboard ranking text-to-video generation models by votes on… | Multimodal |
| C | Arena AI Video EditCrowdsourced pairwise human-preference leaderboard ranking AI video editing models by Arena score from user… | Multimodal |
| D | ART (Action-based Reasoning Task)Reliable clinical decision support requires medical AI agents capable of safe, multi-step reasoning over… | Healthcare |
| A | Artificial Analysis Coding Agent IndexAn Artificial Analysis aggregate index scoring coding agents on end-to-end software engineering tasks,… | Coding |
| A | Artificial Analysis Image Editing ArenaRanks image models on editing an image from a text instruction using Elo scores derived from pairwise human… | Multimodal |
| B | Artificial Analysis Image-to-Video ArenaRanks image-to-video generation models (with audio) by Elo from blind pairwise human preference votes on… | Multimodal |
| B | Artificial Analysis Instrumental Music ArenaRanks instrumental music-generation models by Elo from blind pairwise human-panel preference votes on tracks… | Audio |
| C | Artificial Analysis Vocals Music ArenaRanks music-generation models that produce vocals using Elo ratings from blind pairwise human-panel preference… | Audio |
| A | Artificial Analysis Speech Arena — Provider VoicesRanks text-to-speech models by Elo from blind human preference comparisons using each provider's own native… | Audio |
| A | Artificial Analysis Text-to-Image ArenaRanks image generation models on producing an image from a text prompt, using Elo scores from pairwise human… | Multimodal |
| B | Artificial Analysis Text-to-Video ArenaRanks text-to-video models (with and without audio) by Elo from blind pairwise human preference votes on… | Multimodal |
| C | Artificial Analysis Video Editing ArenaRanks video models on editing a video from a text instruction (with audio kept) by Elo from blind pairwise… | Multimodal |
| D | AssistantBenchAssistantBench measures whether web agents can solve 214 realistic, time-consuming tasks by navigating the… | Agentic |
| D | ASTA BenchAstaBench measures AI agents' ability to assist scientific research across literature understanding, code and… | Agentic |
| C | AutoLabAutoLab measures AI agents' auto-research capability on hours-long iterative performance-engineering tasks… | Agentic |
| B | BALROG Official LLM LeaderboardBALROG measures the agentic reasoning, planning and exploration abilities of LLMs and VLMs across a set of… | Game |
| C | BALROG Official VLM LeaderboardBALROG's VLM leaderboard measures the agentic, long-horizon reasoning of vision-language models across game… | Game |
| D | BEHAVIOR 2025 ChallengeBEHAVIOR 2025 Challenge measures robot policies on 50 long-horizon household mobile manipulation tasks in… | Embodied |
| D | BeliefTrackBeliefTrack measures whether language models correctly update, preserve, or isolate their belief state over… | Reasoning |
| A | BenchBenchBenchBench is a meta-benchmark leaderboard that evaluates LLM benchmarks against their peers via standardized… | Meta |
| C | Beyond BinaryBBBench (Beyond-Binary Bench) evaluates GUI critic models on fine-grained ranking of candidate GUI agent… | GUI |
| D | Beyond ContentBeyond Content (ToxiAlert-Bench) measures toxic speech detection in audio, including toxicity arising from… | Speech |
| D | Beyond Fixed Benchmarks and Worst-Case AttacksDynamic Boundary Evaluation (DBE) adaptively locates each LLM's ability boundary on a calibrated item bank… | Safety |
| D | BioASQBioASQ is a challenge that evaluates systems on biomedical semantic indexing, question answering,… | Healthcare |
| D | BRIGHTBRIGHT measures how well text retrieval models find relevant documents for real-world queries that require… | Retrieval |
| D | BugsInPyBugsInPy is a benchmark of real, reproducible bugs in Python projects for controlled testing, debugging and… | Coding |
| D | BugsJSBugsJS is a benchmark of real JavaScript bugs from 10 Node.js projects, with buggy and fixed revisions and… | Coding |
| D | CAIA BenchmarkCAIA measures how well LLM agents answer high-stakes crypto and financial-market analysis tasks, scoring… | Agentic |
| A | CAIS Risk IndexA composite risk index on the CAIS AI Dashboard that scores frontier models on safety risks such as prompt… | Safety |
| B | CAIS Text Capabilities IndexA composite index on the CAIS AI Dashboard that averages frontier model scores on text benchmarks such as… | Knowledge |
| B | CAIS Vision Capabilities IndexA composite index on the CAIS AI Dashboard that averages model scores on vision benchmarks covering embodied… | Vision |
| D | Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost ReasoningChemCost measures whether LLM agents can estimate chemical procurement costs for reactions by grounding… | Chemistry |
| D | Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability?AgenticInterpBench measures whether LM agents can explain the function of components in already-localized… | Reasoning |
| D | Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency OperationsDORA measures how well LLM agents carry out end-to-end disaster response workflows, such as perception,… | Agentic |
| D | Cangjie-benchCangjie-bench measures how well LLMs perform code generation, translation, and program repair in the… | Coding |
| D | ChainWorldChainWorld measures how well computer use agents complete long-horizon desktop workloads built by chaining… | Agentic |
| B | LMArena Vision ArenaLMArena Vision Arena ranks multimodal AI models on reasoning over visual inputs using crowdsourced pairwise… | Multimodal |
| D | ChehreChehre measures how well vision-language models recognize dynamic facial expressions in videos, both the… | Video |
| D | CodeContestsCodeContests is a competitive programming dataset with train, validation and test splits used to evaluate… | Coding |
| D | ContextBenchContextBench measures how well coding agents retrieve and use code context (recall, precision, efficiency)… | Coding |
| D | CrackMeBenchCrackMeBench measures how well language-model agents can reverse engineer symbol-poor binaries in a sandbox to… | Cybersecurity |
| D | CrowS-PairsCrowS-Pairs measures the degree to which masked language models exhibit U.S. stereotypical social biases… | Generalization |
| D | DALPHINDALPHIN measures how well vision-language pathology AI copilots answer diagnostic visual questions on… | Biomedicine Healthcare |
| D | DAWNDAWN measures how well model agents fit quantitative statistical models to data, covering distribution fitting… | Data |
| C | Decomposing and Measuring Evaluation AwarenessMeasures evaluation awareness in frontier language models, i.e., whether models recognize they are being… | Safety |
| D | DeepSurvey-BenchDeepSurvey-Bench evaluates the academic value of automatically generated scientific surveys across… | Summarization |
| D | DigiWorldDigiWorld evaluates computer use agents on 15 realistic sandboxed mobile applications across over 3.2 million… | Agentic |
| D | Disentangling Language Roles in Multilingual LLM Task ExecutionMTM-Bench measures how LLMs execute tasks when instruction, source content, and required response languages… | Multilingual |
| D | Do Agents Know What They Can't Do? Evaluating Feasibility Awareness in Tool-Using AgentsMeasures whether tool-using agents can recognize infeasible tasks (with critical tools masked) and stop… | Agentic |
| D | Do Enterprise Systems Need Learned World Models? The Importance of Context to Infer DynamicsCascadeBench evaluates how well models predict field-level state changes (business-rule cascades) caused by… | Agentic Business |
| D | Do LLMs Build World Models From Text? A Multilingual Diagnostic of Spatial ReasoningMentalMap measures whether LLMs build spatial world models from pure-text descriptions of household scenes… | Spatial Reasoning |
| D | Do Vision-Language-Models show human-like logical problem-solving capability in point and click puzzle games?VLATIM evaluates whether vision-language models can solve physics puzzles in The Incredible Machine 2, from… | Game |
| D | DocILEDocILE measures key information localization and extraction and line item recognition from semi-structured… | Document Understanding Business |
| D | Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action ModelsAct2Answer measures how much commonsense and world knowledge Vision-Language-Action models retain by requiring… | Embodied |
| D | DS-1000DS-1000 measures code generation on a thousand realistic data science problems drawn from StackOverflow across… | Coding |
| D | DUE BenchmarkDUE Benchmark measures how well models understand layout-rich documents across document QA, key information… | Document Understanding |
| D | Efficient Multimodal Clinical Question Answering for Pulmonary Embolism Risk AssessmentEvaluates efficient multimodal LLMs on eight diagnostic and prognostic pulmonary embolism clinical question… | Healthcare |
| D | EgoBabyVLMEgoBabyVLM measures how well vision-language models trained on naturalistic egocentric video (e.g., infant… | Multimodal Understanding |
| D | Embodied Multi-Agent Coordination by Aligning World Models Through DialogueMeasures whether LLM-based embodied agents with partial observability align their world models through… | Embodied |
| D | Emotional intelligence in large language models is fragmented across perceptionFACET evaluates the emotional intelligence of LLMs with 480 expert-crafted items covering perception,… | Human-preference |
| D | EngiAIEngiAI is a benchmark suite that evaluates LLM agents on engineering design workflows, covering tool-use… | Agentic |
| D | ENPMR-BenchENPMR-Bench measures whether memory-augmented emotional support agents can infer users' latent emotional needs… | Agentic |
| D | EnterpriseRAG-BenchEnterpriseRAG-Bench measures how well RAG systems retrieve company-internal documents and generate correct,… | Agentic Business |
| D | EvalCrafterEvalCrafter benchmarks text-to-video generation models on 700 prompts across visual quality, content quality,… | Video |
| D | Evaluating and Achieving Controllable Code Completion in Code LLMC3-Bench measures LLMs' ability to follow user instructions during code completion, beyond functional… | Coding |
| D | Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMsRS-Neg evaluates how well multimodal large language models understand negation in remote sensing imagery… | Vision |
| D | Evaluating Cross-lingual Knowledge Consistency in Code-Mixed vis-a-vis Indian Languages using IndicKLARIndiKLAR measures cross-lingual knowledge recall consistency of LLMs across English, 18 Indian languages, and… | Multilingual |
| D | Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI ReportsMeasures how well open-source LLMs classify sentences from unstructured cyber threat intelligence reports into… | Cybersecurity |
| D | Evaluating the Architectural Reasoning Capabilities of LLM Provers via the Obfuscated Natural Number GameMeasures whether LLM provers can synthesize Lean 4 formal proofs using only local axioms and definitions in an… | Math |
| D | Ex Ante Evaluation of AI-Induced Idea Diversity CollapseMeasures, ex ante from model-only generations versus matched unaided human baselines, how much creative LLM… | Creative |
| D | Executable Benchmarking Suite for Tool-Using AgentsAn executable benchmarking suite that evaluates closed-loop tool-using agents across web (WebArena Verified,… | Agentic |
| D | FAB-BenchFAB-Bench evaluates retrieval-augmented generation systems on semiconductor manufacturing question answering… | Retrieval |
| D | FactoryBenchFactoryBench measures how well time-series models and LLMs understand industrial robot machines from… | Robotics |
| D | FEVERFEVER measures claim verification against Wikipedia, requiring systems to label claims as Supported, Refuted… | Knowledge |
| D | FEVEROUSFEVEROUS measures a model's ability to retrieve evidence from Wikipedia sentences and table cells and verify… | Retrieval |
| D | FieldWorkArenaFieldWorkArena measures how well multimodal agentic AI performs real-world field work tasks, such as detecting… | Domain Business |
| A | Focusable Monocular Depth EstimationFDE-Bench measures target-centric monocular relative depth estimation, scoring how well models prioritize… | Vision |
| D | FoundationalASSISTFoundationalASSIST evaluates whether LLMs can predict K-12 math student performance and exact responses… | Education |
| D | From RAG to Agentic RAG for Faithful Islamic Question AnsweringIslamicFaithQA is a bilingual Arabic/English generative benchmark of 3,810 Islamic questions measuring answer… | Question Answering |
| D | Habitat ChallengeHabitat Challenge is an annual embodied AI challenge that evaluates uploaded navigation agents in novel,… | Embodied |
| B | Harvey Legal Agent BenchmarkOpen-source long-horizon legal agent benchmark where agents work from client-matter instructions,… | Legal |
| D | Health Memory ArenaHealth Memory Arena is an open, live leaderboard for healthcare AI agents that scores models on benchmarks… | Agentic Healthcare |
| D | HELM InstructHELM Instruct scores instruction-following language models with absolute ratings from human and LM evaluators… | Knowledge |
| D | HELM Long ContextHELM Long Context is a leaderboard that evaluates LLMs' long context capabilities on curated tasks like… | Generalization |
| D | HiddenMathHiddenMath measures model performance on an internal holdout set of competition math problems. | Math |
| D | HourVideoHourVideo measures multimodal models' understanding of hour-long egocentric videos through multiple-choice… | Multimodal |
| B | IDP LeaderboardIDP Leaderboard evaluates AI models on document understanding tasks such as OCR, table extraction, key… | Multimodal |
| A | JARVIS-Leaderboard AIJARVIS-Leaderboard AI scores AI models on materials science tasks such as property prediction, force fields… | Materials |
| D | LAB-BenchLAB-Bench measures language models and AI systems on practical biology research tasks such as literature… | Science |
| D | Language-TableLanguage-Table is a multi-task continuous control benchmark measuring language-conditioned robotic tabletop… | Robotics |
| D | LeanDojo BenchmarkLeanDojo Benchmark measures how well language models can prove formal theorems in Lean, using 98,734 theorems… | Math |
| D | LegalBench-RAGLegalBench-RAG evaluates retrieval systems on complex legal contract understanding questions, measuring… | Legal |
| D | LLM Agents Already Know When to Call Tools -- Even Without ReasoningWhen2Tool measures whether tool-augmented LLM agents call tools only when necessary, across 18 environments… | Tool Use |
| B | LMArena Search ArenaSearch Arena ranks LLMs with integrated web search using crowdsourced pairwise human-preference votes on… | Agentic |
| B | LMArena WebDev ArenaLMArena WebDev Arena (Code Arena) ranks models on web development tasks using human preference votes in… | Coding |
| B | LMArena Text ArenaLMArena Text Arena ranks text chat models by crowdsourced anonymous pairwise human-preference votes aggregated… | Generalization |
| D | localmaxxingLocalMaxxing is a community leaderboard that measures local LLM inference speed (tokens/sec,… | Inference |
| D | LocalScoreLocalScore measures how well local hardware runs LLMs, scoring prompt processing speed, generation speed, and… | Inference |
| D | ManiSkill2ManiSkill2 is a unified benchmark of 20 simulated robotic manipulation task families (rigid/soft-body,… | Robotics |
| D | MASSIVEMASSIVE measures multilingual natural language understanding of virtual assistant utterances, scoring intent… | Multilingual |
| D | MatSci MLMatSci ML measures how well machine learning models predict properties of solid-state materials (energies,… | Materials |
| D | MedVidBenchMedVidBench measures how well vision-language models understand medical and surgical videos across video,… | Healthcare |
| D | MineDojoMineDojo is a Minecraft-based simulation benchmark of 3,142 open-ended tasks measuring how well embodied… | Agentic |
| D | MiniF2FMiniF2F measures automated theorem proving systems on formalized olympiad and high-school/undergraduate math… | Math |
| D | MLPerf ClientMLPerf Client measures ML inference performance (e.g., LLM chatbots, image classification) on client devices… | Inference |
| D | MLPerf Inference v6.0MLPerf Inference v6.0 measures AI system inference performance (throughput, latency, energy efficiency) across… | Inference |
| D | Mobile-Agent-EMobile-Eval-E, introduced with the Mobile-Agent-E framework, measures how well mobile agents complete complex,… | Mobile |
| D | NYU LLM CTFNYU CTF Bench evaluates LLM agents on solving real-world capture-the-flag cybersecurity challenges. | Cybersecurity |
| D | OASB Skills Security BenchmarkOASB is a labeled corpus and methodology for evaluating AI agent skill security scanners on detecting… | Cybersecurity |
| D | OmniAI OCR LeaderboardCompares OCR and data extraction performance (JSON accuracy, text similarity, latency, cost) of traditional… | Multimodal |
| B | Open FinLLM LeaderboardOpen FinLLM Leaderboard evaluates LLMs across about 50 financial tasks in seven financial categories, such as… | Finance |
| D | OpenAI EvalsOpenAI Evals is a framework and open registry of evals that score language models on varied tasks, such as… | Knowledge |
| D | PatentBenchPatentBench measures AI systems on patent prosecution tasks such as USPTO Office Action docketing, deadline… | Legal |
| D | PAWS-XPAWS-X measures cross-lingual paraphrase identification using human-translated adversarial PAWS pairs in six… | Multilingual |
| D | PDEBenchPDEBench measures how well machine learning models act as surrogates for time-dependent PDE simulations (fluid… | Science |
| B | PEFT-ArenaPEFT-Arena measures how parameter-efficient finetuning methods trade off target-task gains (math and medical… | Machine Learning |
| D | Poor Paul's BenchmarkPoor Paul's Benchmark measures local LLM inference performance on consumer hardware, ranking model and GPU… | Inference |
| D | ProjDevBenchProjDevBench measures how well AI coding agents build complete, executable software repositories from… | Coding |
| D | PubTables-1MPubTables-1M evaluates models on table detection, table structure recognition and functional analysis in… | Document Understanding |
| C | PulseMindThe PulseMind Benchmark evaluates multi-modal medical models on multi-turn diagnostic consultations, scoring… | Healthcare |
| D | Quantitative Video World Model Evaluation for Geometric-ConsistencyPDI-Bench scores generative video models on geometric coherence (scale-depth alignment, 3D motion consistency,… | Video |
| D | RAGBenchRAGBench measures how well methods evaluate retrieval-augmented generation outputs, using 100k labeled RAG… | Retrieval |
| D | Realm WarrenRealm Warren (Realm legal-reasoning) evaluates models on long-horizon legal work products in litigation,… | Legal |
| D | ResearchGymResearchGym measures how well AI agents perform end-to-end AI research, proposing hypotheses, running… | Agentic |
| D | RoboCasa365RoboCasa365 measures generalist robot policies' task success on kitchen manipulation tasks, including atomic… | Embodied |
| D | RoboDojoRoboDojo evaluates generalist robot manipulation policies on 42 simulation and 18 real-world tasks covering… | Robotics |
| D | RSIBench-DataRSIBench-Data measures whether frontier researcher agents can improve a fixed Qwen3.5 target model by… | Agentic |
| D | Room-Across-RoomRoom-Across-Room (RxR) measures how well agents follow multilingual (English, Hindi, Telugu) natural-language… | Robotics |
| D | SafeArenaSafeArena measures how often autonomous LLM web agents complete or refuse deliberately harmful versus safe… | Safety |
| D | SEATauBenchSEATauBench measures how well conversational tool-use agents perform customer-service tasks (airline, retail,… | Agentic |
| D | SEC-benchSEC-bench evaluates LLM agents on authentic security engineering tasks, namely generating proof-of-concept… | Cybersecurity |
| D | SheetCopilot BenchmarkMeasures how well LLM-based agents complete natural-language spreadsheet control tasks (formatting, charts,… | Spreadsheets |
| B | Speech ArenaSpeech Arena ranks text-to-speech models by Elo ratings from crowd-sourced blind pairwise listening tests. | Audio |
| D | Speech-MASSIVESpeech-MASSIVE is a multilingual spoken language understanding dataset and benchmark that scores models on… | Speech |
| D | SUNRGBDSUN RGB-D is a benchmark suite of 10,000 densely annotated RGB-D indoor images for evaluating scene… | Spatial Reasoning |
| D | SurgVLA-BenchSurgVLA-Bench evaluates vision-language-action models on simulated laparoscopic surgical robotics tasks, from… | Robotics |
| D | SWE-ArenaSWE-Arena is an arena-style platform that compares AI models' coding capabilities in real time with direct… | Coding |
| D | Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective ResponsesCEDAR measures whether multilingual LLMs understand culturally grounded affective responses, scoring emotion… | Multilingual |
| D | TravelPlannerTravelPlanner measures whether language agents can use search tools to gather information and produce… | Agentic |
| D | TUA-BenchTUA-Bench measures how well terminal-use agents complete 120 real-world, execution-scored tasks spanning… | Computer Use |
| D | TurnBenchTurnBench measures how accurately and quickly spoken-dialogue systems detect end-of-turn and interruption… | Audio Speech |
| D | VAKRAVAKRA measures LLM agents' ability to complete multi-step tool-calling tasks over executable APIs and document… | Agentic Business |
| D | WANDRWANDR measures how well research agents perform wide-and-deep web research, discovering many qualifying… | Agentic Business |
| D | WebChoreArenaWebChoreArena measures how well web browsing agents complete 532 tedious, labor-intensive web chores requiring… | Agentic |
| D | WebShopWebShop measures how well language agents can navigate a simulated e-commerce website to search, customize,… | Agentic |
| D | WikiTableQuestionsWikiTableQuestions measures answering complex, multi-step questions over semi-structured Wikipedia tables,… | Knowledge |
| D | WorkArena-L3WorkArena-L3 measures how well web agents complete harder, compositional knowledge-work workflows on the… | Agentic Business |
| D | XNLIXNLI evaluates cross-lingual natural language inference, testing whether models can classify entailment in 15… | Multilingual |
| D | XTREME-RXTREME-R evaluates multilingual models on ten natural language understanding tasks, including… | Multilingual |
| D | Design Arena — 3D DesignDesign Arena's 3D Design leaderboard ranks models by blind crowd votes on generated 3D design outputs. | Multimodal |
| D | Design Arena — ASCII ArtDesign Arena ASCII Art measures which models produce the best ASCII art via blind human pairwise voting… | Multimodal |
| D | Design Arena — Android AppsDesign Arena's Android/Mobile Arena measures how well coding agents build native Android (Kotlin/Jetpack… | Multimodal |
| D | Design Arena — Data VisualizationDesign Arena's Data Visualization category scores models by Elo rating on generating data visualizations. | Multimodal |
| D | Design Arena — Frontend Web AppsDesign Arena's Web App Arena ranks AI models on building functional React web applications (now including… | Multimodal |
| D | Design Arena — Full-Stack Web AppsDesign Arena's full-stack web app evaluation measures whether AI models can build complete, production-ready… | Multimodal |
| D | Design Arena — Godot Game DevelopmentDesign Arena's Godot Game Dev leaderboard ranks models by Elo from head-to-head human preference battles on… | Multimodal |
| D | Design Arena — Graphic DesignDesign Arena — Graphic Design measures model quality at generating graphic design via blind human pairwise… | Multimodal |
| D | Design Arena — HTML Game DevelopmentDesign Arena's Game Dev leaderboard ranks models by Elo rating from head-to-head comparisons of the HTML games… | Multimodal |
| D | Design Arena — Image EditingDesign Arena's Image Editing arena ranks image-editing models by Elo from comparative evaluations, also… | Multimodal |
| D | Design Arena — Image GenerationDesign Arena — Image Generation is an arena leaderboard that ranks models by Elo on image generation tasks. | Multimodal |
| D | Design Arena — Image to VideoDesign Arena's Image to Video leaderboard ranks AI models on generating videos from images using blind crowd… | Multimodal |
| D | Design Arena — Image to Web AppDesign Arena's Image-to-Website leaderboard ranks AI models by user-preference Elo on building websites/web… | Multimodal |
| D | Design Arena — Image to WebsiteDesign Arena's Image-to-Website leaderboard ranks AI models by user-preference Elo on generating websites from… | Multimodal |
| D | Design Arena — Logo DesignDesign Arena — Logo Design is a crowdsourced head-to-head voting benchmark that ranks AI models on logo design… | Multimodal |
| D | Design Arena — Multi-Image to VideoDesign Arena's Multi-Image to Video leaderboard ranks video generation models by Elo on generating videos from… | Multimodal |
| D | Design Arena — Multimodal to VideoDesign Arena's video leaderboard ranks video generation models by Elo ratings from head-to-head human… | Multimodal |
| D | Design Arena — Proprietary Harness SlidesDesign Arena's Slides leaderboard ranks AI presentation-generation models and tools by Elo from blind,… | Multimodal |
| D | Design Arena — Python-PPTX SlidesDesign Arena's Python-PPTX Slides leaderboard ranks models by Elo on automated presentation creation using… | Multimodal |
| D | Design Arena — React Native AppsDesign Arena's Mobile Arena (React Native track) ranks coding agents by blind pairwise user votes on mobile… | Multimodal |
| D | Design Arena — SVG DesignDesign Arena SVG is an Elo-rated, preference-based leaderboard scoring how well models generate SVG designs. | Multimodal |
| D | Design Arena — Text to SpeechA crowdsourced arena that ranks text-to-speech models by Elo from human votes on generated speech. | Multimodal |
| D | Design Arena — UI ComponentsDesign Arena UI Components is a crowdsourced arena where models generate UI components from design prompts and… | Multimodal |
| D | Design Arena — Video EditingDesign Arena's Video Editing leaderboard ranks AI video editing models by blind head-to-head human preference… | Multimodal |
| D | Design Arena — Video GenerationDesign Arena's video generation leaderboard ranks text-to-video models by score on how well they turn written… | Multimodal |
| D | Design Arena — Video to WebsiteDesign Arena's Video-to-Website benchmark measures how well an AI model can watch a reference video of a… | Multimodal |
| D | Design Arena — Web Design OverallDesign Arena is a crowd-voted arena where users compare AI models' outputs on creative tasks such as websites,… | Multimodal |
| D | Design Arena — Website BuildersDesign Arena's Website leaderboard ranks AI models by Elo rating from head-to-head comparisons of the websites… | Multimodal |
| D | Codegolf v2.2Codegolf v2.2 is a code-generation benchmark scored by pass@1, reported in the Gemma 3n model card under STEM… | Coding |
| D | MMBench-LiveMMBench-Live is a continuously evolving multimodal benchmark that evaluates vision-language models with… | |
| D | 3,034-sample human-curated spatial understanding benchmarkMeasures whether vision-language models understand 3D spatial layout via depth-ordered occlusion,… | Spatial Reasoning |
| D | A Closer Look at Failure Modes in Temporal Understanding of Large Audio-Language ModelsA benchmark of 1,657 questions across three foundational tasks measuring temporal reasoning in large… | Audio |
| D | A Visually Impaired Assistance Benchmark for VLM-as-a-Judge EvaluationVIABLE measures how reliably vision-language models act as judges of AI assistance responses for blind and… | Multimodal Understanding |
| D | Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision SupportMeasures how well LLMs actively gather evidence through multi-turn inquiry with a simulated standardized… | Biomedicine Healthcare |
| D | Adversarial Empathy BenchmarkAEB evaluates the robustness of empathetic language models in adversarial emotional dialogues where users… | Behavioral |
| D | AgentCollabBenchAgentCollabBench is a diagnostic benchmark of 900 human-validated multi-agent tasks that measures process… | Agentic |
| D | AgentEscapeBenchAgentEscapeBench measures whether LLM agents can infer, execute, and revise novel tool-use procedures in… | Agentic |
| D | AI Spreadsheet BenchmarkMeasures how spreadsheet AI assistants complete 53 realistic spreadsheet tasks, scoring success… | Spreadsheets |
| A | ALL Bench MultimodalALL Bench Multimodal is a composite leaderboard scoring models on aggregated vision-language, image… | Multimodal |
| D | ALME BenchmarkALME measures how often audio-language models follow conflicting text instead of the audio signal (Text… | Audio |
| D | Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examinationVS-Bench (VisualSwap) measures whether vision-language models genuinely re-examine images during reasoning by… | Multimodal Understanding |
| D | Aristotelian Virtue Profiling of LLMs through Ethical DilemmasVirtueMap profiles LLMs on five Aristotelian virtues by having them rank five responses to each of seven… | Alignment |
| D | Bench4HLSBench4HLS evaluates LLM-generated high-level synthesis (HLS) hardware designs on compilation success,… | Coding |
| D | Benchmarking Large Language Models for Grapheme-to-Phoneme ConversionMeasures how accurately large language models convert Japanese text into kana readings (grapheme-to-phoneme… | Speech |
| D | Benchmarking LLM Agents on Meta-Analysis Articles from Nature PortfolioMetaSyn evaluates LLM agents and RAG pipelines on reproducing meta-analyses from Nature Portfolio journals… | Biomedicine |
| D | Benchmarking Local Language Models for Social Robots using Edge DevicesBenchmarks 25 open-source language models on edge hardware for social-educational robots across inference… | Robotics |
| D | Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity ClassificationMeasures how accurately and efficiently small language models and small reasoning models classify severity… | Classification |
| D | Benchmarking von ASR-Modellen im deutschen medizinischen KontextEvaluates 29 ASR models on a curated dataset of simulated German doctor-patient anamnesis conversations,… | Speech |
| D | Beyond AccuracyBeyond Accuracy evaluates how many distinct valid solution strategies LLMs can produce on 80 AMC 10/12 and… | Reasoning |
| D | Beyond Known FactsA contamination-free synthetic evaluation dataset of predicted future temporal facts with text descriptions,… | Retrieval |
| C | BioToolBioTool measures how accurately LLMs make biomedical tool calls to NCBI, Ensembl and UniProt APIs, scored by… | Biomedicine |
| B | Block-R1Block-R1 is a benchmark for multi-domain reinforcement learning post-training of block-based diffusion large… | Reasoning |
| D | BountyBenchBountyBench evaluates AI agents on detecting, exploiting, and patching real-world bug bounty vulnerabilities… | Cybersecurity |
| D | BPL OCR BenchBPL OCR Bench ranks OCR models on document images using pairwise VLM-as-judge comparisons scored with… | Multimodal |
| D | Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe ExtractionVG-GUI-Bench measures whether MLLM-based GUI agents can follow video tutorials to complete corresponding… | GUI |
| D | CALVINCALVIN is an open-source simulated benchmark measuring how well agents learn long-horizon,… | Embodied |
| D | Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial MalayMalayPrag evaluates LLMs' ability to connect discourse particles in colloquial Malay with their pragmatic… | Multilingual |
| D | Can LLMs Compress (and Decompress)? Evaluating Code Understanding and Execution via InvertibilityRoundTripCodeEval measures whether LLMs can consistently reason about code execution forward and backward by… | Coding |
| D | Can OCR-VLMs Read Devanagari? A Stress-Test Benchmark and Post-Correction StudyMeasures how well ten OCR systems, from classical engines to OCR-VLMs and frontier multimodal LLMs, read… | Ocr |
| D | Can Segmentation Models Understand the World? Towards Proactive Affordance Reasoning via Visual Chain-of-ThoughtAn intent-to-part benchmark that evaluates whether segmentation models can segment the affordance-bearing… | Embodied |
| D | Can You Trust What You See? Human and AI Detection of Synthetic Legal EvidenceMeasures how well humans and multimodal LLMs distinguish authentic legal evidence photographs from… | Legal |
| D | CarCrashNetCarCrashNet benchmarks machine-learning neural solvers on predicting structural vehicle crash simulation… | Aerospace |
| D | CARLA Autonomous Driving LeaderboardEvaluates the driving proficiency of autonomous vehicle agents navigating predefined routes through realistic… | Embodied |
| D | CausaLabCausaLab measures whether LLM agents can perform interactive causal discovery in a synthetic laboratory, both… | Agentic |
| D | CFAgentBenchCFAgentBench measures autonomous construction-finance agents operating across a mock software stack (ERP,… | Agentic |
| D | ChaosBench-LogicChaosBench-Logic measures LLM logical and symbolic reasoning about chaotic dynamical systems via TRUE/FALSE… | Reasoning |
| D | CharadesSTACharades-STA measures how well models temporally localize activities in untrimmed videos given natural… | Multimodal |
| D | ChemSafetyBenchChemSafetyBench measures the accuracy and safety of LLM responses in chemistry, covering chemical property… | Safety |
| D | ChildEvalChildEval measures how well LLMs infer and follow child-centered preferences of children aged 3-6 in… | Education |
| D | ClaimRAG-LAWClaimRAG-LAW evaluates legal retrieval-augmented generation systems in French and English at the claim level,… | Legal |
| D | CLIMATE-FEVERCLIMATE-FEVER measures how well models retrieve evidence for and verify real-world climate change claims,… | Climate |
| D | ClimateBenchClimateBench measures how well emulators predict key climate variables (temperature, diurnal temperature… | Climate |
| D | Code-QA-BenchCode-QA-Bench measures repository-level code understanding by agents, separating genuine code comprehension… | Coding |
| D | CodeXGLUECodeXGLUE measures code intelligence across 10 programming language tasks such as clone detection, defect… | Coding |
| D | Collider-BenchCollider-Bench measures whether LLM agents can reproduce Large Hadron Collider particle physics analyses from… | Science |
| D | CounselBenchCounselBench evaluates large language models on open-ended mental health question answering using expert… | Healthcare |
| D | CoVoST2CoVoST 2 measures speech-to-text translation quality across many languages, covering translation from 21… | Speech |
| D | CSCBenchCSCBench measures LLM reasoning in commodity supply chains through 2.3K+ single-choice questions spanning… | Business |
| D | Decoupled Web Penetration Testing BenchmarkMeasures LLM-based web penetration testing agents' exploitation and reconnaissance capabilities separately… | Cybersecurity |
| D | DedeuceBenchDedeuceBench scores LLM agents on budgeted active-learning episodes, measuring success within a query budget,… | Agentic |
| D | Deep Research Agent BenchmarkMeasures how well frontier deep research agents produce decision-grade management consulting deliverables… | Agentic Business |
| D | DeepResearch Bench MultilingualMultilingual prompt translations of the 100 DeepResearch Bench research tasks across eight languages, for… | Agentic |
| D | DFIR-MetricDFIR-Metric evaluates large language models on digital forensics and incident response through expert-reviewed… | Cybersecurity |
| D | Diagnosing and Mitigating Sycophancy and Skepticism in LLM Causal JudgmentCAUSALT3 is a 454-instance expert-curated benchmark that scores LLM causal reasoning across Pearl's three… | Reasoning |
| D | Diagnosing LLM Arbitration Behavior over Pre-evidence Epistemic States in RAG-based Fact-CheckingPAVE measures how LLM verifiers in RAG-based fact-checking arbitrate between their pre-evidence parametric… | Factuality |
| B | DiagnosticIQDiagnosticIQ measures whether LLMs can recommend correct maintenance actions for industrial assets from… | Reasoning |
| D | Do Open-Vocabulary Detectors Transfer to Aerial Imagery? A Comparative EvaluationMeasures zero-shot transfer of open-vocabulary object detection models to aerial imagery on the LAE-80C… | Geospatial |
| D | Do Text Edits Generalize to Visual Generation? Benchmarking Cross-Modal Knowledge Editing in UMMsUniKE measures whether knowledge edits applied to unified multimodal models' text outputs transfer to their… | Multimodal |
| D | DomainCodeBenchDomainCodeBench measures code generation models on 20 domain-specific tasks in healthcare, finance, molecular… | Coding |
| D | DRIP-RDRIP-R measures how LLM agents make decisions and reason in customer-service return scenarios governed by… | Domain Business |
| C | DrivingGenDrivingGen evaluates generative video world models for autonomous driving on visual realism, trajectory… | Video |
| B | Edge LLM Leaderboard: Raspberry Pi 5Benchmarks throughput, memory/model size, and MMLU accuracy of quantized LLMs running locally on a Raspberry… | Inference |
| D | EditPropBenchEditPropBench measures whether LLM editors propagate factual edits through dependent claims in synthetic… | Factuality |
| D | EditRisk-BenchEditRisk-Bench evaluates how maliciously injected knowledge edits (misinformation, bias, safety violations)… | Safety |
| D | EDU-CIRCUIT-HWEDU-CIRCUIT-HW evaluates multimodal LLMs on recognizing and auto-grading authentic university-level STEM… | Education |
| D | EHRStructEHRStruct measures how well large language models perform 11 clinical tasks over structured electronic health… | Healthcare |
| D | ElecBenchElecBench measures how well large language models handle electric power domain knowledge and four business… | Energy |
| D | EmbSpatialBenchEmbSpatialBench scores vision-language models on embodied spatial understanding, reported under Spatial… | Embodied |
| D | EngiBenchEngiBench measures how well large language models solve real-world engineering problems across three levels:… | Science |
| D | Equation-suffix prediction benchmarkMeasures whether a model's forecast of the hidden remainder of a displayed equation in physics and mathematics… | Math |
| D | ESI-BENCHESI-Bench measures embodied spatial intelligence of multimodal LLM agents that must perceive, navigate and… | Embodied |
| D | Evaluating Long-Horizon Memory for Multi-Party Collaborative DialoguesEverMemBench measures long-horizon conversational memory of LLMs and memory systems over multi-party,… | Long Context |
| D | Evaluating Reward Model Generalization via Pairwise Maximum Discrepancy CompetitionsPMDC evaluates reward model generalization by selecting open-domain prompt-response pairs where two reward… | Alignment |
| D | Evaluating the Pre-Consultation Ability of LLMs using Diagnostic GuidelinesEPAG measures how well LLMs conduct medical pre-consultation, scoring collected History of Present Illness… | Healthcare |
| D | EvasionBenchEvasionBench measures how well models classify the directness of corporate management answers to analyst… | Safety |
| D | Exploring Autonomous Agentic Data Engineering for Model SpecializationEvaluates LLMs as autonomous data engineers that plan, generate and iteratively optimize training data to… | Agentic |
| D | extended needle-in-a-haystack benchmarkAn extended needle-in-a-haystack benchmark measuring how long-context LLMs perform literal extraction, logical… | Long Context |
| D | Extending Beacon to HindiMeasures sycophancy of language models in Hindi versus English using a forced-choice diagnostic with English,… | Alignment |
| D | FActScoreFActScore measures the factual precision of long-form text generated by language models by computing the… | Behavioral |
| D | FATE-HFATE-H measures LLM provers' ability to formally prove, in Lean, 100 hard abstract and commutative algebra… | Math |
| D | FATE-XFATE-X measures LLM provers' ability to formally prove, in Lean, 100 extra-hard abstract and commutative… | Math |
| D | FigQAFigQA is a LAB-Bench subtask of multiple-choice questions testing whether AI systems can reason about… | Multimodal |
| D | Fine-Grained Benchmark Generation for Comprehensive Evaluation of Foundation ModelsA framework that automatically generates textbook-grounded benchmarks with fine-grained metadata, producing… | Domain |
| D | FinToolBenchFinToolBench measures how well LLM agents use 760 real executable financial tools to answer 295 tool-required… | Finance |
| D | FiQA 2018FiQA 2018 is an open challenge that evaluates systems on aspect-based financial sentiment analysis and… | Finance |
| D | FIRE-BenchFIRE-Bench measures whether LLM agents can autonomously rediscover established findings from recent machine… | Agentic |
| D | first bias evaluation dataset for multimodal speech recognitionMeasures bias in multimodal (audio-visual) speech recognition by pairing different faces with the same audio… | Multimodal |
| D | ForeSciForeSci measures whether LLM agents can make forward-looking AI research judgements (which bottleneck,… | Reasoning |
| D | FPEval / FPBenchFPEval/FPBench measures LLM code generation in functional programming languages (Haskell, OCaml, Scala),… | Coding |
| D | From Task Solving to Robust Real-World Adaptation in LLM AgentsMeasures robustness of agentic LLMs in a long-horizon grid-based game under partial observability, dynamic… | Agentic |
| D | FunctionalMATHFunctionalMATH (MATH()) is a functional variant of the MATH benchmark that measures the reasoning gap between… | Math |
| D | GCSE Double-Marked BenchmarkMeasures how closely LLMs agree with human examiners when marking 32,534 double-marked real GCSE student… | Education |
| B | Generating synthetic computed tomography for radiotherapySynthRAD2025 is a challenge benchmark scoring methods that generate synthetic CT from MRI or CBCT for… | Biomedicine Healthcare |
| D | GeoBuildBenchGeoBuildBench measures whether LLMs and multimodal agents can turn natural-language plane geometry problems… | Reasoning |
| D | GeoQAGeoQA measures multimodal numerical reasoning by having models answer geometric questions involving diagrams,… | Knowledge |
| D | GISAGISA measures how well search agents and LLMs answer 373 human-crafted, real-world information-seeking queries… | Agentic |
| B | GR-BenGR-Ben measures how well process reward models and LLMs detect process-level reasoning errors across science… | Reasoning |
| C | Greenland Sovereignty GameMeasures LLM geopolitical behavior by having frontier models play six roles in game-theoretic simulations of… | Game |
| D | GUI-OdysseyGUI-Odyssey measures how well multimodal agents navigate mobile GUIs across multiple apps to complete long,… | GUI |
| C | GuideWebGuideWeb measures how well models automatically generate in-app guides on real-world web UIs by selecting… | GUI |
| D | HANSHANS is an evaluation set that tests whether natural language inference models rely on syntactic heuristics… | Knowledge |
| D | HAO / TBFactHAO/TBFact evaluates LLM-generated patient summaries for Molecular Tumor Boards, using a model-as-a-judge… | Healthcare |
| D | Hidden State Poisoning Attacks against Mamba-based Language ModelsRoBench-25 measures how well language models, especially Mamba-based SSMs, retain information retrieval… | Cybersecurity |
| D | HoudiniVexBenchHoudiniVexBench measures how well models generate, complete and explain VEX shader code for SideFX Houdini,… | Coding |
| D | How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?Measures how well tool-augmented LLM agents solve 243 expert-curated real-world energy market analytics tasks… | Energy |
| D | How Does Personalized Memory Shape LLM Behavior? Benchmarking Rational Preference Utilization in Personalized AssistantsRPEval measures whether LLM-powered personalized assistants rationally use (ignore, support, or let dominate)… | Reasoning |
| D | How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical StudyImmersedPrivacy measures how well vision-language models used as embodied assistants recognize and respect… | Safety |
| D | How Implicit Bias Accumulates and Propagates in LLM Long-term MemoryThe DIB Benchmark measures how implicit bias in LLMs with long-term memory accumulates over time and… | Alignment |
| D | IDEA-Bench ArenaIDEA-Bench Arena ranks image generation models on professional-level design and image processing tasks using… | Multimodal |
| D | (implicit — evaluation of prompting-based defenses against domain-camouflaged injection)Measures how well prompting-based defenses reduce the success of domain-camouflaged prompt injection attacks… | Safety |
| C | InteractiveBench OverallInteractiveBench Overall is a composite score of LLM reasoning through budgeted multi-turn interaction across… | Agentic |
| D | Is He Extroverted? Identifying Missing Relevant Personas for Faithful User SimulationPICQ-drama measures whether LLMs can identify relevant but missing persona dimensions needed to simulate a… | Behavioral |
| D | ISL FinePDFs OCRRanks OCR models on ISL FinePDFs document images using pairwise VLM-as-judge comparisons scored with… | Multimodal |
| D | KamonBenchKamonBench measures how well vision-language models recover compositional factors (container, modifier, motif)… | Vision |
| D | KnotBenchKnotBench measures whether vision-language models can reason about and operate on knot diagrams across 14… | Spatial Reasoning |
| D | L-EvalL-Eval measures long-context language models on 20 sub-tasks over long documents (3k–200k tokens), including… | Generalization |
| A | La LeaderboardLa Leaderboard evaluates generative LLMs on 66 datasets in Basque, Catalan, Galician and Spanish varieties of… | Language |
| D | Large Language Models Lack Temporal Awareness of Medical KnowledgeTempoMed-Bench measures whether LLMs have temporal awareness of medical knowledge, recalling which guideline… | Biomedicine Healthcare |
| B | LatamBoardLatamBoard is a leaderboard and evaluation hub scoring AI models on Latin American and Iberian language tasks… | Multilingual |
| D | Legal RAG BenchLegal RAG Bench measures end-to-end performance of legal retrieval-augmented generation systems on expert… | Legal |
| D | LegalBench-BR v2LegalBench-BR v2 measures how well models classify Brazilian court decisions in Portuguese into five legal… | Legal |
| D | LegalRikai: Open BenchmarkLegalRikai: Open Benchmark evaluates LLMs on four long-form Japanese corporate legal tasks: amendment… | Legal |
| D | LexNeo-BenchLexNeo-Bench measures whether LLMs can classify Luxembourgish tokens as native or as French, German, or… | Multilingual |
| D | LLaVA-BenchLLaVA-Bench evaluates large multimodal models on open-ended visual chat, i.e. answering free-form questions… | Knowledge |
| D | LLMs Infer Cultural Context but Fail to Apply It When RespondingCAPRI evaluates whether LLMs infer a user's cultural background from conversational cues and adapt responses… | Multilingual |
| D | LongDocURLLongDocURL measures how well large vision-language models understand, numerically reason over, and locate… | Document Understanding |
| D | LongMemEval-V2LongMemEval-V2 measures whether memory systems can turn long histories of multimodal web-agent trajectories… | Agentic |
| D | LOVEU-TGVELOVEU-TGVE is a CVPR 2023 competition benchmark scoring text-guided video editing models on 76 videos with… | Multimodal |
| D | MACE-THORMACE-THOR measures how well LLM-based planners enable heterogeneous multi-robot teams to complete long-horizon… | Embodied |
| D | ManiSoftManiSoft measures how well vision-language-action policy models perform manipulation tasks with simulated soft… | Robotics |
| D | MAUDMAUD measures models' reading comprehension of merger agreements, scoring answers to expert-annotated legal… | Legal |
| D | MAWPSMAWPS is a repository of math word problems used to evaluate models on solving arithmetic word problems by… | Math |
| D | MDGYMMDGYM measures whether AI agents can autonomously set up, run and interpret molecular dynamics simulations in… | Agentic |
| D | Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law CourtsTF-RefusalBench measures over-alignment (refusals and disclaimers) of LLMs on multilingual criminal-law… | Legal |
| D | Measuring Safety Alignment Effects in Autonomous Security AgentsA trace-based benchmark of 30 local vulnerability-analysis tasks measuring how safety-aligned versus… | Safety |
| D | MedBrowseCompMedBrowseComp measures how well agentic systems retrieve and synthesize multi-hop medical facts from live,… | Healthcare |
| D | MedCase-StructuredMedCase-Structured measures LLM diagnostic accuracy on clinician-authored diagnostic cases presented as… | Healthcare |
| D | MEGA XStoryClozeMEGA XStoryCloze, part of the MEGA multilingual suite, measures whether generative language models can pick… | Knowledge |
| C | Memes-as-RepliesMeasures whether models can select contextually humorous Japanese manga panel replies to social media posts,… | Multimodal Understanding |
| D | MemLeakMemLeak measures whether facts deleted from multimodal agent memory remain recoverable through retained… | Safety |
| C | Memory-Induced Tool-Drift in LLM AgentsMEMDRIFT measures how personality-driven biases stored in an LLM agent's long-term memory inappropriately… | Agentic |
| D | Memory Retention Is Not Enough to Master Memory Tasks in Reinforcement LearningMeasures RL agents' ability to rewrite (not just retain) memory under partial observability, via Endless… | Agentic |
| D | MemoryQuestMemoryQuest measures how well dialogue assistants retrieve user-specific facts from long multi-session… | Retrieval |
| D | MentisOculiMentisOculi measures whether multimodal models can form, maintain, and manipulate visual representations… | Multimodal Understanding |
| D | Meta-WorldMeta-World measures how well multi-task and meta-reinforcement learning algorithms learn and generalize across… | Robotics |
| C | Metacognitive ProbeThe Metacognitive Probe measures LLM confidence behaviour across five calibration dimensions: confidence… | Behavioral |
| D | MHRC-BenchMHRC-Bench measures how well language models complete repository-level hardware description language code… | Coding |
| D | MIMeBenchMIMeBench is an open-ended benchmark that evaluates LLMs on semantic abstraction and contrastive… | Reasoning |
| D | Mind2WebMind2Web measures how well generalist web agents follow language instructions to complete complex tasks on… | Agentic |
| D | MM-Mind2WebMM-Mind2Web measures how well multimodal web agents plan and execute multi-step actions (click, type, select)… | Agentic |
| D | MMVetMM-Vet evaluates large multimodal models on complicated multimodal tasks that require integrating six core… | Multimodal |
| D | MobileMiniWob++_SRMobileMiniWob++_SR measures the success rate of agents completing 92 MiniWob++ web interaction tasks adapted… | Agentic |
| D | MoleculeNetMoleculeNet benchmarks machine learning methods on predicting molecular properties across curated public… | Science |
| D | Moroccan Darija ASR LeaderboardA leaderboard ranking automatic speech recognition models on Moroccan Darija transcription accuracy. | Speech |
| D | MortarBenchMortarBench measures how accurately LLM agents perform mortgage loan origination tasks such as evaluating… | Finance Business |
| D | MTG-Causal-RLMTG-Causal-RL is a Gymnasium benchmark built on Magic: The Gathering that evaluates RL agents on win rate,… | Game |
| D | Multi-Domain Benchmark for Detecting AI-Generated Text-Rich ImagesMeasures how well AI-generated image detectors identify text-rich images (posters, infographics, receipts,… | Document Understanding |
| D | Multi-domain Multi-modal Document Classification Benchmark with a Multi-level TaxonomyMMM-Bench evaluates how well large multimodal models classify real-world business documents into a five-level… | Document-ai Business |
| D | Multi-genre Semantic Segmentation Benchmark for Conversational ArabicMeasures how well models perform semantic (discourse) segmentation of conversational dialectal Arabic across… | Language |
| D | Multi-Source Cybersecurity LogsEvaluates language models on classifying multi-source (system, network, browser) log chunks as normal or… | Cybersecurity |
| D | multi-target cross-lingual element-awareMEA measures how well LLMs summarize a source document into multiple target languages, covering 24 target… | Summarization |
| D | Multi-View Decompilation BenchmarkMeasures how well LLMs classify decompiled pseudo-C (from Ghidra and RetDec) of compiled binaries as benign or… | Cybersecurity |
| D | Multicultural SpyfallMulticultural Spyfall measures LLMs' multilingual and multicultural capabilities by having them play the… | Game |
| D | MultiHop-RAGMultiHop-RAG measures retrieval-augmented generation on multi-hop queries whose evidence spans 2 to 4… | Generalization |
| D | Multilingual MGSMMGSM measures language models' chain-of-thought math reasoning on 250 grade-school math problems manually… | Math |
| D | MultiPL-EMultiPL-E measures multi-language code generation by translating HumanEval and MBPP unit-test-driven… | Coding |
| D | MultiTurnPSBMultiTurnPSB measures how patient-facing medical chatbots degrade into unsafe responses under four-turn… | Safety Healthcare |
| D | MUTATEMUTATE is an interactive benchmark that evaluates agentic divergent thinking in LLM agents at path level… | Agentic |
| D | NanoGPT-BenchNanoGPT-Bench measures how much historical human progress on the NanoGPT Speedrun autonomous coding agents can… | Agentic |
| D | Needle In A HaystackNeedle In A Haystack pressure-tests LLM long-context retrieval by placing facts (needles) at varying depths in… | Generalization |
| D | Nexus Function CallingNexus Function Calling Benchmark measures how accurately models generate single, parallel and nested function… | Agentic |
| D | NL4COPNL4COP measures how well LLM-based methods translate natural-language combinatorial optimization problem… | Math |
| D | NoReGeo: Non-Reasoning Geometry BenchmarkNoReGeo measures whether LLMs and multimodal models natively understand geometric and spatial relationships,… | Spatial Reasoning |
| D | NRT-BenchNRT-Bench measures the robustness of LLM operator agents running a simulated nuclear power plant control room… | Safety |
| D | NsankuNsanku measures zero-shot machine translation performance of LLMs between English and 43 Ghanaian languages… | Translation |
| D | OCR Bench UFOOCR Bench UFO ranks OCR models on UFO document images using VLM-as-judge pairwise comparisons scored with… | Multimodal |
| D | OfficeBenchOfficeBench measures LLM agents' ability to complete realistic office workflows that span multiple… | Agentic Business |
| C | OMLAB Open Agent Math LeaderboardA leaderboard that compares prompting and agent algorithms (IO, CoT, SC-CoT, PoT, ReAct, ToT) across LLMs on… | Agentic |
| D | OmniInteractOmniInteract measures how well real-time omnimodal models handle streaming audio-visual interaction, scoring… | Multimodal |
| D | On the Cultural Anachronism and Temporal Reasoning in Vision Language ModelsTAB-VLM measures vision-language models' temporal reasoning and susceptibility to cultural anachronism on 600… | Translation |
| A | Open Arabic LLM LeaderboardA leaderboard that evaluates and compares Arabic large language models on Arabic-native benchmarks covering… | Language |
| B | Open ASR LeaderboardOpen ASR Leaderboard benchmarks speech recognition systems on word error rate and inverse real-time factor… | Audio |
| A | Open Chinese LLM LeaderboardA leaderboard from BAAI that evaluates LLMs on Chinese-language versions of key benchmarks covering reasoning,… | Language |
| B | Open Italian LLM LeaderboardA leaderboard that scores open LLMs on Italian-language benchmarks translated into Italian, such as MMLU, ARC… | Language |
| A | Open Japanese LLM LeaderboardA leaderboard evaluating Japanese LLMs across 16 tasks from more than 20 datasets, including NLI, QA,… | Language |
| A | Open Ko-LLM LeaderboardThe Open Ko-LLM Leaderboard evaluates Korean large language models on Korean-language benchmarks covering… | Language |
| A | Open LLM Leaderboard v2Open LLM Leaderboard v2 compares open large language models in a reproducible way, scoring them on IFEval,… | General Knowledge |
| D | Open Multilingual LLM Evaluation LeaderboardA leaderboard ranking LLMs across 29 mostly non-English languages on machine-translated ARC, HellaSwag, MMLU… | Multilingual |
| A | Open Portuguese LLM LeaderboardA public leaderboard that evaluates and ranks large language models on Portuguese-language tasks. | Language |
| B | Open Universal Arabic ASR LeaderboardA continuous leaderboard benchmarking open-source Arabic automatic speech recognition models across… | Speech |
| B | OpenVLM Video LeaderboardOpenVLM Video Leaderboard ranks vision-language models on general video understanding by aggregating scores… | Multimodal Understanding Video |
| D | OpenADMET ExpansionRxA blind community challenge scoring models on predicting nine ADMET properties of late-stage drug-discovery… | Science |
| D | OpenAI-MRCR: 2 needle 1MOpenAI-MRCR 2 needle 1M measures a model's ability to distinguish and retrieve one of two similar needles… | Long Context |
| D | OpenClaw Arena Config LeaderboardMeasures how different SOUL.md personal-agent configurations affect GPT-4.1's score on real-world personal… | Agentic |
| D | OpenScore String Quartets OMREvaluates optical music recognition systems on converting string quartet score page images (scanned and… | Multimodal |
| C | OracleProtoOracleProto measures LLMs' native forecasting ability on resolved real-world events reconstructed as… | Reasoning |
| D | OSUniverseOSUniverse measures how well multimodal GUI-navigation AI agents complete complex desktop tasks, from… | Agentic |
| D | PaperAudit-BenchPaperAudit-Bench measures how well language models detect subtle, within-section and cross-section errors in… | Long Context |
| D | PaperFit-BenchPaperFit-Bench measures how well systems transform compilable LaTeX papers into visually polished,… | Document-ai |
| D | PAREDAPAREDA measures automatic speech recognition accuracy (word error rate) on multi-accent, spontaneous speech… | Speech |
| D | PAS (Police Action Scenarios)The use of Large Language Models (LLMs) in police operations is growing, yet an evaluation framework tailored… | Legal |
| D | PCoAPCoA measures how well language models generate medical aspect-based summaries with phrase-level attribution… | Biomedicine |
| D | PerceptionTestPerception Test measures perception and reasoning skills of pre-trained multimodal models over real-world… | Multimodal |
| D | PerMemBenchPerMemBench evaluates personalized memory systems for LLM agents, testing whether they retain user-relevant… | Agentic |
| D | Personalized Turn-Level User Conversation Satisfaction BenchmarkPersTurnBench measures how well generation models satisfy individual users at specific conversation turns,… | Human-preference |
| D | PetroBenchPetroBench measures LLM knowledge in petroleum engineering (production, reservoir, drilling) with 1,200… | Domain |
| D | Phoenix-benchPhoenix-bench measures whether AI agents can resolve real hardware engineering bugs in Verilog repositories,… | Coding |
| D | Physics Question Scene GraphPQSG evaluates the physical plausibility of text-to-video model outputs using a graph of VLM-generated yes/no… | Reasoning |
| B | PII Masking BenchmarkPIIMB measures zero-shot PII masking: a model's ability to detect and mask personally identifiable information… | Safety |
| C | PoisonForgePoisonForge measures how vulnerable instruction-tuned open-weight LLMs are to task-level targeted data… | Safety |
| D | PolyMath-enPolyMath measures mathematical reasoning of LLMs across 18 languages and 4 difficulty levels; PolyMath-en is… | Math |
| D | Power Systems Agent BenchmarkMeasures how well tool-using AI agents solve executable electric power engineering tasks (power flow,… | Energy |
| D | PR ArenaPR Arena is a public leaderboard that ranks AI coding agents by how many of the GitHub pull requests they… | Coding |
| C | Prediction Bottlenecks Don't Discover Causal StructureA reusable falsification benchmark that tests claims that causal-discovery methods (e.g., prediction… | Science |
| D | ProjectionBenchProjectionBench measures LLMs' ability to generate scientific hypotheses as paper details are progressively… | Reasoning |
| D | ProtStructQAProtStructQA measures whether language models can answer natural-language protein structural questions by… | Biomedicine |
| D | Pseudo-Deliberation in Language ModelsMeasures alignment between LLMs' stated values and their reasoning and generated dialogue actions (the… | Alignment |
| D | PTCBENCHPTCBENCH measures how consistent LLM personalities (assessed via the NEO Five-Factor Inventory) remain across… | Behavioral |
| D | PulseBench-TabPulseBench-Tab measures how accurately table extraction systems convert document-image tables across 9… | Document Understanding |
| D | Push Your AgentPushBench measures quantitative goal persistence: whether long-horizon LLM agents keep working with tool calls… | Agentic |
| D | QasperQasper measures how well models answer information-seeking questions grounded in the full text of NLP research… | Long Context |
| D | Quantifying the Gap between Understanding and Generation within Unified Multimodal ModelsGapEval measures the gap between understanding and generation in unified multimodal models via bidirectional… | Multimodal Understanding |
| D | Query Symbolically or Retrieve Semantically? A Dataset and Method for Semi-Structured Question AnsweringSpecsQA measures question answering over semi-structured product specification documents from a commercial… | Question Answering |
| D | QuixBugsQuixBugs measures automated program repair on 40 classic algorithm programs in Python and Java, each… | Coding |
| D | RealICURealICU measures how well LLM agents assess patient status, acute problems, recommended actions and red-flag… | Healthcare |
| D | Redundant or Necessary? A Benchmark for Detecting Redundant Steps in Agent TrajectoriesRedundancyBench measures how well methods can detect redundant versus necessary steps in annotated LLM agent… | Efficiency |
| B | RefCOCO-avgRefCOCO-avg measures object grounding accuracy in images, averaged across the RefCOCO, RefCOCO+, and RefCOCOg… | Spatial Reasoning |
| D | RefSpatialBenchRefSpatialBench measures spatial reference understanding and grounding in vision-language models. | Spatial Reasoning |
| D | ReproRepoReproRepo evaluates whether LLM agents can identify real-world reproducibility blockers in ML paper-repository… | Uncategorized |
| D | RigorBenchRigorBench measures engineering process discipline (planning, verification, recovery, abstention, atomic… | Software Engineering |
| D | RLBenchRLBench measures vision-guided robot manipulation learning across 100 hand-designed simulated tasks, from… | Robotics |
| D | RMGAPRMGAP measures how well reward models generalize by correctly ranking responses to match diverse user… | Alignment |
| D | Roboflow Vision Evals - OCRMeasures how accurately vision models read text from images, combining OCRBench, a license-plate recognition… | Vision |
| D | Roboflow Vision Evals - SegmentationMeasures open-vocabulary instance segmentation: models output pixel masks from text queries, scored by… | Vision |
| D | Robust Asynchronous Planning via Auto-FormalizationMeasures how well LLMs plan asynchronous tasks with non-uniform durations, concurrency and execution-time… | Reasoning |
| C | Robusto-2Robusto-2 benchmarks humans and VLMs on visual question answering over dashcam footage from Lima and New York… | Vision |
| D | RubricsTreeRubricsTree evaluates open-ended responses of LLM personal health agents using a hierarchical taxonomy of over… | Healthcare |
| D | RW-PostRW-Post measures how well large vision-language models fact-check real-world social-media text-image posts,… | Factuality |
| D | SAFE Audio Challenge Task 1SAFE Audio Challenge Task 1 evaluates how well detection models distinguish raw synthetic speech generated by… | Audio |
| D | SAFE Audio Challenge Task 2SAFE Challenge Task 2 benchmarks synthetic speech detection models on processed audio (e.g., compression,… | Audio |
| D | SAFE Audio Challenge Task 3SAFE Audio Challenge Task 3 measures how well detection models identify synthetic speech in laundered audio… | Audio |
| D | SALTSALT measures LLM uncertainty estimation (correctness, calibration, and confidence ranking) in long-form… | Truthfulness |
| D | Scammer4UScammer4U measures how often autonomous web agents leak users' personally identifiable information to… | Safety |
| D | SCDBenchSCDBench measures how well LLMs decompile smart contract bytecode back into Solidity source, scoring format… | Software Engineering |
| D | SciBenchSciBench measures large language models' ability to solve college-level scientific problems in mathematics,… | Science |
| D | SCRuBEvalSCRuBEval measures how well language models reason about social concepts (norms, culture, institutions) with… | Reasoning |
| D | SEAGymSEAGym is an evaluation environment that measures how self-evolving LLM agents' harness updates affect… | Agentic |
| D | SEC-QASEC-QA evaluates how well models answer multi-document, long-context financial questions over SEC filings that… | Finance |
| D | self-evolving instrument for subjective behavior evaluationA self-evolving evaluation instrument that scores LLMs on subjective, human-facing behaviors such as emotional… | Behavioral |
| D | SlakeVQASLAKE (SlakeVQA) evaluates medical visual question answering systems on bilingual English-Chinese questions… | Healthcare |
| D | SLVMEvalSLVMEval meta-evaluates text-to-video evaluation systems on long videos (up to ~3 hours), testing whether they… | Multimodal |
| D | SocialBias-BenchSocialBias-Bench measures social bias in LLM-generated code across 343 real-world coding tasks spanning seven… | Coding |
| D | SpaceRejectSpaceReject measures whether embodied vision-language agents correctly refuse deliberately unanswerable… | Embodied |
| C | SpaRRTaSpaRRTa measures the ability of visual foundation models to recognize relative spatial positions of objects in… | Spatial Reasoning |
| D | SpecXSpecX measures how well specialized spectral models and multimodal language models perform molecular… | Chemistry |
| B | SphinxSphinx is a checklist-based benchmark that evaluates LLM pull request review quality by coverage of actionable… | Software Engineering |
| D | SQuALITYSQuALITY measures how well models write overview and question-focused summaries of long public-domain short… | Long Context |
| D | ST-VQAST-VQA measures visual question answering that requires reading and understanding scene text in images. | Knowledge |
| D | StableToolBenchStableToolBench measures LLMs' ability to use external tools/APIs, scoring solvable pass and win rates using a… | Agentic |
| B | SuiChat-CNSuiChat-CN measures how well language models assess users' suicide risk levels from contextual multi-party… | Healthcare |
| D | SUPERBSUPERB measures how well a shared (self-supervised) speech model performs across a wide range of speech… | Speech |
| D | SurgGoalSurgGoal is a multicentric meta-evaluation benchmark assessing whether vision-language models produce valid… | Healthcare |
| B | SWE AtlasSWE Atlas benchmarks coding agents on software engineering tasks beyond issue resolution. | Software Engineering |
| D | SWE-ChainSWE-Chain measures how well coding agents implement chained release-level upgrades to real Python packages,… | Software Engineering |
| D | SWE-PolyBenchSWE-PolyBench measures how well AI coding agents resolve repository-level software engineering tasks (bug… | Coding |
| D | Synthesis and Evaluation of Long-term History-aware Medical DialogueEvaluates healthcare agents' memory and reasoning over synthetic long-term patient medical dialogue histories… | Healthcare |
| D | T2QBenchT2QBench measures whether LLM agents acquire a grounded understanding of their environment, using 1,967 QA… | Agentic |
| A | Tau2 AverageTau2 measures how well conversational AI agents solve customer-support-style tasks in dual-control… | Agentic |
| D | The Abstraction Gap in Vision-Language Causal ReasoningCAGE measures whether vision-language models perform faithful causal reasoning over images versus merely… | Reasoning |
| D | The Curse of HelpfulnessDistractionIF measures LLM robustness to benign instruction-like distractor text embedded in reference… | Instruction Following |
| D | The Register GapThe Register Gap evaluates language models on the Meaning Intelligence Framework, scoring nine dimensions of… | Behavioral |
| B | The Wrong Kind of RightVETO, introduced in The Wrong Kind of Right, measures misfired alignment: how often LLMs reject… | Alignment |
| D | Theory-Scale Auto-Formalization of Logics for Computer ScienceLCS-Bench measures theory-scale auto-formalization and theorem-proving in Lean, translating hundreds of… | Math |
| D | TOFU LLaMA 10%TOFU measures how well unlearning methods make LLMs forget fictitious author profile QA data (here the LLaMA,… | Safety |
| D | TOFU LLaMA 1%TOFU measures how well unlearning methods make LLMs forget fictitious author profiles (here the 1% forget-set… | Safety |
| D | TOFU LLaMA 5%TOFU measures how well unlearning methods make LLMs forget fictitious author profile QA data (here LLaMA… | Safety |
| D | TOFU Phi 10%TOFU measures how well unlearning methods make LLMs forget a forget set of synthetic fictitious author QA… | Safety |
| D | TOFU Phi 1%TOFU measures how well unlearning methods make LLMs forget fictitious author profiles (here the Phi model with… | Safety |
| D | TOFU Phi 5%TOFU measures how well unlearning methods make LLMs forget a subset (here 5%) of fictitious author profile QA… | Safety |
| D | Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-CheckingFactArena benchmarks LLMs across the full fact-checking pipeline—claim extraction, tool-augmented evidence… | Factuality |
| D | Towards Direct Evaluation of Harness Optimizers via Priority RankingDirectly evaluates harness optimizer agents by having them rank harness components (prompt, tool, workflow,… | Agentic |
| D | Towards Evaluation of Implicit Software World Models in Coding LLMsMeasures whether coding LLMs can predict runtime behaviour of real software (test outcome, exception class,… | Software Engineering |
| D | Towards Understanding and Measuring COGNITIVE ATROPHY in LLM BehaviourMeasures cognitive atrophy in LLM mental-health support conversations: whether model responses help users keep… | Safety |
| B | TraitFactoryTraitFactory measures how well methods steer language models on multiple behavioral attributes (behavioral… | Behavioral |
| D | Truthful AI AdvisorsMeasures whether LLM advisors stay truthful when their objective is misaligned with the user's, scoring… | Alignment |
| D | UAV-VLN-FOVUAV-VLN-FOV measures whether a UAV agent can ground a visible target from vision-language instructions and… | Uncategorized |
| D | Uncovering Entity Identity Confusion in Multimodal Knowledge EditingEC-Bench is a diagnostic benchmark that measures Entity Identity Confusion in multimodal knowledge-edited… | Multimodal Understanding |
| D | Uncovering Political Bias in Large Language Models using Parliamentary Voting RecordsMeasures political bias in LLMs by comparing model-generated voting predictions on parliamentary motions with… | Alignment |
| D | Understanding Mental States to Guide Social Influence in Multi-Person Group DialogueSocialMindChange measures whether language models can use Theory of Mind to generate dialogue in multi-person,… | Behavioral |
| D | Unintended Long-Term State Poisoning BenchULSPB measures how routine multi-turn user interactions poison a personalized LLM agent's persistent long-term… | Safety |
| D | VAIRVAIR measures whether large reasoning models can correctly evaluate math solutions that reach valid answers… | Reasoning |
| D | VATEXVATEX is a multilingual (English-Chinese) video-and-language benchmark evaluating multilingual video… | Multimodal |
| D | Vera-BenchVera-Bench measures the safety of tool-using LLM agents against attacks, using 1,600 executable safety cases… | Safety |
| D | Veterinary Clinical Summarization EvalEvaluates commercial veterinary-focused LLM tools on summarizing veterinary oncology records, scored by a… | Healthcare |
| D | VIDAVIDA evaluates whether multimodal models use visual evidence to resolve visually dependent ambiguous… | Translation |
| D | VIMA-BenchVIMA-Bench is a simulation benchmark of procedurally generated tabletop robot manipulation tasks specified by… | Robotics |
| D | Vision Language Models for Optimization-Driven Intent Processing in Autonomous NetworksIntentOpt measures whether vision-language models can turn annotated network diagrams into correct… | Coding |
| D | VisIT-Bench Multiple ImagesVisIT-Bench measures how well vision-language models follow real-world instructions over images, scored… | Multimodal |
| D | VisualLeakBenchVisualLeakBench measures whether vision-language agents propagate sensitive or unsafe text visible in images… | Safety |
| D | VLABenchVLABench evaluates vision-language-action models and VLM workflows on 100 categories of language-conditioned… | Multimodal |
| D | VLegal-BenchVLegal-Bench measures how well large language models understand, reason about, and apply Vietnamese law across… | Legal |
| D | VLMsAreBlindVLMsAreBlind (BlindTest) measures whether vision-language models can perform simple low-level visual tasks… | Multimodal |
| D | Voice BenchmarkVoice Benchmark ranks voice AI platforms by measured real-world end-to-end response latency from recurring… | Audio |
| D | VSD FashionVSD Fashion scores models on zero-shot in-catalog visual similarity retrieval of fashion images, reporting ROC… | Vision |
| D | WBCMor VQAWBCMor VQA measures how well vision-language models answer bilingual English–Urdu morphology-aware questions… | Healthcare |
| D | WebTailBenchWebTailBench scores web agents on realistic, long-tail browser tasks such as compositional searches, bookings… | Agentic |
| D | What Does the AI Doctor Value? Auditing Pluralism in the Clinical Ethics of Language ModelsAudits the clinical ethical value priorities of language models using a benchmark of clinician-verified… | Healthcare |
| D | What Makes LVLMs Hallucinate Less? Unveiling the Architectural Factors Behind Hallucination RobustnessCoSimUE measures hallucination robustness of large vision-language models across co-occurrence, similarity,… | Factuality |
| D | When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech RecognitionMeasures whether audio-language models can use clinical context (diagnosis labels, speech ratings, clinical… | Audio Healthcare |
| C | When English Rewrites Local KnowledgeMeasures whether LLMs answering culturally grounded Bengali questions in Bangla versus English substitute… | General Knowledge |
| C | When Symptoms Are Not EnoughMeasures how accurately LLMs screen for psychiatric disorders (anxiety, depression, PTSD, any disorder) from… | Healthcare |
| D | Who Generated This 3D Asset? Learning Source Attribution for Generative 3D ModelsA benchmark for passive source attribution of generated 3D assets, testing whether and which of 22 3D… | Vision |
| D | Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM AgentsPrincipalBench measures whether multi-party LLM agents stay loyal to the principal they represent against a… | Agentic |
| D | WMT23WMT23 General Machine Translation shared task evaluates machine translation systems across multiple language… | Translation |
| D | Workspace-Bench 1.0Workspace-Bench measures how well AI agents complete workspace tasks that require retrieving, reasoning over… | Agentic Business |
| D | World-Ego Modeling for Long-Horizon Evolution in Hybrid Embodied TasksHTEWorld, introduced in this paper, measures long-horizon world modeling (future video prediction) for robots… | Embodied |
| B | WorldArena Track 1WorldArena Track 1 scores embodied world models on video perception quality and embodied task functionality,… | Multimodal |
| D | XLGoBenchXLGoBench uses synthetic algorithmic tasks posed in different languages to measure cross-lingual gaps in large… | Reasoning |
| C | Constraint Saturation EvaluationConstraint Saturation Evaluation measures how well large language models follow many simultaneous explicit… | |
| D | ELBenchELBench evaluates education-facing large language models across general capability, safety and… | |
| D | EvoPolicyGymEvoPolicyGym measures how well harness-model agents iteratively edit and improve executable policies in… | |
| D | JavaVulBenchJavaVulBench measures how well models detect security vulnerabilities in Java code at method and line level,… | |
| D | MathAdvMathAdv measures advanced undergraduate- and graduate-level mathematical reasoning across 13 domains through… | |
| D | MedStreamBenchMedStreamBench measures whether vision-language models give correct and well-timed answers and proactive… | |
| D | RustMizanRustMizan measures how well LLM agents detect, classify (CWE), and localize vulnerabilities in compilable Rust… | |
| D | SocietyBenchSocietyBench measures whether language models can forecast how real social events unfold over time, using… | |
| D | SPEARBenchSPEARBench evaluates the conversational naturalness of streaming speech-to-speech language models across… | |
| D | VGA-BenchV2VGA-BenchV2 evaluates video generation models on aesthetic value and generation quality using human-aligned… | |
| D | YOMI-BenchYOMI-Bench measures how well LLMs read kanji and understand Japanese phonology, through four tasks on reading… | |
| D | Are Video Models Zero-Shot Learners and Reasoners in Education? EduVideoBenchEduVideoBench evaluates whether video generation models produce educationally valid videos across knowledge,… | Education |
| D | Can Large Language Models Reliably Code Qualitative Humanitarian Data? A Benchmark Study Against Human Expert AdjudicationMeasures how reliably LLMs perform deductive coding of qualitative humanitarian interview transcripts compared… | Coding |
| D | FinBoardBenchFinBoardBench evaluates LLMs' dynamic wealth management and strategic financial decision-making through… | Finance |
| D | Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions from Japan's National Assessment of Academic AbilityMeasures multimodal LLMs on officially released Japanese middle-school exam items in Science, Mathematics, and… | Education |
| D | InvestPhilBenchInvestPhilBench measures whether large language models can accurately reconstruct and apply the procedural… | Finance |
| D | KGCEKGCE measures how well multimodal LLM agents execute educational tasks across Windows, Android, and… | Education |
| D | Legal Hallucination Detection BenchmarkAttorneys, judges, and pro se filers increasingly use AI to draft legal documents, yet these tools frequently… | Legal |
| D | Towards Error-Free EHRsEHR-ReasonCon measures how well models verify consistency between clinical notes and structured tables in… | Healthcare |
| S | HLEAn expert-authored benchmark designed to probe frontier knowledge and reasoning. | Knowledge |
| S | HLE (Mercor)Answering difficult academic questions across many subjects. This table shows Mercor-run configurations for… | Knowledge |
| A | HLE w/ tools | Agentic |
| A | WebArena-VerifiedWebArena-Verified is an audited release of the WebArena browser-agent benchmark. It rechecks task… | Agentic |
| A | Codeforces ELO | Coding |
| A | Hy-BrowseComp-Pro2 | Search |
| A | SimpleQA Net Score | Knowledge |
| A | LifeSciBench | Science Long-Horizon |
| A | BioMysteryBench | Agentic |
| A | EQ-Bench 3 | Instruction Following |
| A | LOCA-Bench 256KTests long-context agents as tool environments grow to a 256K-token environment description. | Agentic |
| A | GDPval rubricsA display-only provider-table GDPval rubric score for economically valuable work tasks. | Agentic |
| A | GMMLUMMLU-style knowledge evaluation across 42 high- and low-resource languages. | Multilingual |
| A | HELM LegalBenchHELM LegalBench: Measures legal reasoning, contract review, statute interpretation, or legal-domain QA. | Legal |
| A | SQASQA: Measures structured-data reasoning over tables, spreadsheets, charts, databases, or data analysis tasks. | Knowledge |
| A | Hy-CompanyBench V2Tencent internal evaluation listed in the agentic-coding section of the HY4 preview benchmark appendix. | Coding |
| A | Hy-FinAgentBench | Finance |
| A | Hy-LifeSearch | Search |
| A | timm ImageNet Robustness | Vision |
| A | 100Q-Hard Net Score | Knowledge |
| A | AraGen v3Arabic generative-task benchmark for chat and instruct models, evaluated with the 3C3H rubric over culturally… | Language |
| A | LABBench2 Source Reliability | Biology |
| A | Multilingual Factual Questions Net Score | Multilingual |
| A | OpenUGI | Alignment |
| A | SWE-ProSoftware performance optimization is a notoriously complex and manual task. Despite the growing use of Large… | Software Engineering |
| B | Excel Modeling Benchmark | |
| B | SWE-Marathon | Coding |
| B | OpenThoughts | Coding |
| B | OpenReward | Finance |
| B | MazeBenchTests whether a coding agent can explore a 256-room 3D puzzle world and collect its 100 hidden gems, using… | Agentic |
| B | ParseBenchScores document parsers and vision-language models on 2,078 human-verified enterprise pages across tables,… | Multimodal |
| B | MultiChallengeTests instruction retention, memory of user information, versioned editing, and coherence over multi-turn… | Instruction Following |
| B | Web Search Index | |
| B | Autoresearch | ML Science |
| B | benchmarks.bio | Science |
| B | CREW | Coding Enterprise Workflows |
| B | FrontierSWE v2 | Coding Long-Horizon+1 |
| B | CUA data | Computer Use Tool Use |
| B | Gandalf the Grader | Multi-Domain |
| B | Human Creativity Benchmark | Creative Coding |
| B | AIMSWe argue that safety classifiers should model user intent as an explicit signal between the prompt and the… | Safety |
| B | EnterpriseOps-GymServiceNow benchmark of stateful planning and tool use across simulated enterprise systems, as run by model… | Agentic |
| B | ToolSandboxApple benchmark of stateful, conversational tool use, where tool calls change state and a simulated user takes… | Agentic |
| B | RedlineBenchNegotiating and revising contracts across multiple turns. This table shows Mercor-run configurations for… | Agentic |
| B | FinanceArenaAn AfterQuery benchmark of open-ended financial analysis that requires models to read financial data, make… | Knowledge |
| B | Multi-IFMeta benchmark of instruction following over multi-turn conversations in eight languages. | Instruction Following |
| B | AgentIFInstruction-following benchmark built from real agentic applications, with long instructions that carry many… | Instruction Following |
| B | Beyond Literal MappingLarge Language Models (LLMs) have significantly advanced Machine Translation (MT), applying them to… | Translation |
| B | BLXBench | Coding |
| B | A Positive Case for FaithfulnessLLM self-explanations are often presented as a promising tool for AI oversight, yet their faithfulness to the… | Alignment |
| B | AI-Secure LLM Trustworthy LeaderboardTrustworthiness benchmark for LLMs covering toxicity, stereotypes, adversarial robustness, out-of-distribution… | Safety |
| B | Chart-FR1Multimodal large language models (MLLMs) have shown considerable potential in chart understanding and… | Reasoning |
| B | Clembench Multimodal v1.6.5Clembench Multimodal evaluates chat-optimized multimodal models as conversational agents through visual… | Agentic |
| B | FinSTaRTime series (TS) reasoning models (TSRMs) have shown promising capabilities in general domains, yet they… | Finance |
| B | GeoDialSeveral educational domains rely heavily on diagrams and visual cues, yet most existing tutoring datasets are… | Education |
| B | HindiGen v1Hindi generative-task benchmark for chat and instruct models, evaluated with the 3C3H rubric across Hindi QA,… | Language |
| B | InfiniteBM Liar's DiceHead-to-head LLM game-arena ladder for Liar's Dice, using InfiniteBM's per-game Bradley-Terry Elo ratings… | Game |
| B | kluster.ai LLM Hallucination Detection Leaderboard | Behavioral |
| B | LanguageBenchLanguageBench evaluates AI models across many languages and tasks, including translation, classification,… | Multilingual |
| B | Med-CoReasonerWhile reasoning-enhanced large language models perform strongly on English medical tasks, a persistent… | Biomedicine |
| B | Mixing Expert KnowledgeLarge language models (LLMs) have demonstrated exceptional performance in reasoning tasks such as mathematics… | Game |
| B | PlanAheadDespite recent advances, LLM-based web agents still struggle with limited exploration, omission of critical… | Agentic |
| B | PROBE Family ClassificationPROBE drug-target protein family classification benchmark, reporting MCC across random and sequence-identity… | Science |
| B | PROBE Semantic SimilarityPROBE semantic similarity benchmark for protein representation methods, measuring correlation with Gene… | Science |
| B | Reeve Foundation Multilingual CorpusBuilding on our previous work, this paper develops practical, low-barrier methods for freelance translators… | Translation |
| B | RoboAlign-R1 | Robotics |
| B | SearchGymSearch agents have emerged as a pivotal paradigm for solving open-ended, knowledge-intensive reasoning tasks.… | Search |
| B | TabEmbedFoundation models have established unified representations for natural language processing, yet this paradigm… | Tabular |
| B | Triaging Threats to Specialized Guardrails | Safety |
| C | Turing Terminal-Bench Data | Coding Enterprise Workflows+1 |
| C | BrowseComp+Evaluates research agents against a fixed document corpus, separately from live-web BrowseComp. | Agentic |
| C | OSWorld-Verified | Computer Use Enterprise Workflows |
| C | OSWorld Windows tasks on Daytona Windows sandboxes | Coding Computer Use |
| C | SWE-bench MultimodalJavaScript issue resolution where bug reports depend on screenshots, evaluated by Vals AI with no agent… | Coding |
| C | Technical Report | Enterprise Workflows |
| C | Agent-environment papers | Coding Computer Use+2 |
| C | MMLU Professional MedicineMMLU Professional Medicine: Evaluates clinical, biomedical, medical-exam, coding, or healthcare-document… | Healthcare |
| C | Firehorse open harness library | Coding Long-Horizon |
| C | inspect-openreward | Coding Long-Horizon |
| C | Public Benefits Bench | |
| C | KernelQuest | Coding Enterprise Workflows+1 |
| C | PLSQLBench | Coding Enterprise Workflows+1 |
| C | Turing Frontier Agentic STEM | Coding Enterprise Workflows+1 |
| C | ConjectureBench: 14,865 open math problems | Long-Horizon |
| C | Dynamic Semantic Tags technical report with Intuit | Long-Horizon |
| C | The Valley of Code ReasoningThe Valley of Code Reasoning (arXiv 2510.06101) | Coding Computer Use+2 |
| C | Cats Confuse Reasoning LLMCats Confuse Reasoning LLM (arXiv 2503.01781) | Coding Computer Use+2 |
| C | Self-rationalization improves LLM as a fine-grained judgeSelf-rationalization improves LLM as a fine-grained judge (arXiv 2410.05495) | Coding Computer Use+2 |
| C | AI bosses studies, parts 1 and 2 | Computer Use Long-Horizon |
| C | env0 SFT study on SkillsBench/ClawsBench | Coding Computer Use+1 |
| C | GenesisBench | Coding Computer Use+1 |
| C | FinQA / SnorkelFinance environment and benchmark | Coding Computer Use+2 |
| C | RIFT / Hitchhiker's Guide to Rubric QualityRIFT / Hitchhiker's Guide to Rubric Quality (arXiv 2604.01375) | Coding Computer Use+2 |
| C | Shrinking the Generation-Verification Gap with Weak Verifiers | Coding Computer Use+2 |
| C | Snorkel: Rapid Training Data Creation with Weak Supervision | Coding Computer Use+2 |
| C | Human Baselines for Benchmarks | Coding Enterprise Workflows+1 |
| C | INTELLECT-3 Technical ReportINTELLECT-3 Technical Report (https://arxiv.org/abs/2512.16144) | Coding Enterprise Workflows+2 |
| C | INTELLECT-2INTELLECT-2 (https://www.primeintellect.ai/blog/intellect-2) | Coding Enterprise Workflows+2 |
| C | Systematic Reward Hacking studySystematic Reward Hacking study (https://primeintellect.ai/blog/reward-hacking) | Coding Enterprise Workflows+2 |
| C | True Agents Model the World / ECHOTrue Agents Model the World / ECHO (https://primeintellect.ai/blog/true-agents-model-the-world) | Coding Enterprise Workflows+2 |
| C | Scaling Agentic RL: 365k+ SWE/terminal/search tasksScaling Agentic RL: 365k+ SWE/terminal/search tasks (https://primeintellect.ai/blog/scaling-agentic-rl) | Coding Enterprise Workflows+2 |
| C | No Last Mile: A Theory of the Human Data MarketNo Last Mile: A Theory of the Human Data Market (arXiv 2603.00932) | Enterprise Workflows |
| C | PII Transformation for Enterprise Datasets | Enterprise Workflows |
| C | AI Models Now Introduce Safety Risks in the Physical World | Enterprise Workflows |
| C | Data-Centric Interpretability for LLM-based Multi-Agent RLData-Centric Interpretability for LLM-based Multi-Agent RL (arXiv 2602.05183) | Long-Horizon |
| C | StarSkirmish Bench | Long-Horizon |
| C | 1830 railroad game to Finance-Agent transfer study | Long-Horizon |
| C | Cross-Benchmark Transfer from RL on Agentic Coding Tasks - arXiv | Enterprise Workflows Long-Horizon |
| C | Post-Training on Office Work Improves Software Engineering - arXiv | Enterprise Workflows Long-Horizon |
| C | Cross-Benchmark Generalization in Long-Horizon Agents - arXiv | Enterprise Workflows Long-Horizon |
| C | Robo-benchTests agents controlling a simulated robot arm and humanoid through motor commands, sensor readings, and… | Agentic |
| C | RULERSynthetic long-context benchmark of retrieval, multi-hop tracing, aggregation and question answering, scored… | Reasoning |
| C | Simulated-to-Real Transfer for Web Agents (2026-07-27): | Computer Use Enterprise Workflows |
| C | BackSearch point-in-time search for leak-free forecasting evals | Coding Long-Horizon |
| C | Galactica, Llama 2, Llama 3 | Coding Long-Horizon |
| C | OpenThoughts datasets | Long-Horizon |
| C | GEPA | Long-Horizon |
| C | Bespoke-Nimble-9B model | Long-Horizon |
| C | Bespoke Curator | Long-Horizon |
| C | Evalchemy | Long-Horizon |
| C | Bespoke-MiniCheck | Long-Horizon |
| C | Finance Agent Benchmark - arXiv:2508.00828; Finance Agent v2 | |
| C | Vals Index v2.1 and Vals Multimodal Index | |
| C | AI cheating audit | |
| C | Project Vend | Computer Use Long-Horizon |
| C | Cheating in Drone-Bench | Computer Use Long-Horizon |
| C | Multi-token Residual PredictionMulti-token Residual Prediction (arXiv 2605.18817; co-authored by Modal Research, 2026-05) | Coding |
| C | LLM Engineer's Almanac inference benchmarks | Coding |
| C | Quail open-source AI-SQL inference engine with CMU Full Stack Data Lab | Coding |
| C | FlashAttention-4 inference contributions | Coding |
| C | Trillion-token | Coding |
| C | ApexAgents SkyRL 397B RL training recipe | Coding Enterprise Workflows+1 |
| C | The Benchmark CeilingThe Benchmark Ceiling (arXiv 2607.01254) | Enterprise Workflows |
| C | COS-PLAYCOS-PLAY (arXiv 2604.20987) | Long-Horizon |
| C | Jev verifier cost study | Long-Horizon |
| C | Trinity: autoformalization system | Coding |
| C | Trinity/Gauss Lean verification environments infrastructure | Coding |
| C | MiniMuSiQue: multi-hop QA eval dataset | Coding |
| C | LiveWeb samples (with Browserbase) - 2026-09-22/28 | Computer Use Enterprise Workflows |
| C | Production-debugging RL environment (hud-ops-diagnostics) - 2026-01-20 | Computer Use Enterprise Workflows |
| C | Dojo-Bench-Mini public task set on Hugging Face | Computer Use |
| C | Pango crowdsourced computer-use trajectory datasets on Hugging Face | Computer Use |
| C | GLADOS-1 | Computer Use |
| C | PopuLoRAPopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Play (arXiv:2605.16727, 2026-05-16) | Coding Long-Horizon |
| C | ~1k JavaScript coding tasks in Harbor format | Coding Long-Horizon |
| C | SRE Bench (2026-08) and CyberBench v1.1 | |
| C | Vals Legal AI Report (VLAIR), Feb 2025 | |
| C | Gauntlet 4K RLVR dataset, access by request | Coding Computer Use+1 |
| C | OpenEnv Visual Memory and Spreadsheet environments on Hugging Face | Coding Computer Use+3 |
| C | Messier | Long-Horizon |
| C | Predicting Task Difficulty Without Rollouts | Long-Horizon |
| C | FINDFinancial decision-making in multilingual settings demands accurate numerical reasoning grounded in diverse… | Finance |
| C | MIRROR | Cybersecurity |
| C | House Rules | Long-Horizon |
| C | GAPJigsaw puzzle solving has been an increasingly popular task in the computer vision research community. Recent… | Vision |
| C | User-simulation realism: Veris vs τ²-bench | Enterprise Workflows |
| C | Printing Machines (2026-01-28) | Computer Use Enterprise Workflows |
| C | No peer-reviewed paper authored by Fleet found. Harbor | Computer Use Enterprise Workflows |
| C | Legal Research Bench | |
| C | Editorial and infrastructure posts on chakra.dev/blog | Computer Use |
| C | PROPELPROPEL: Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier (arXiv:2606.18284,… | Coding Long-Horizon |
| C | Robustly Evaluating Cyber Capabilities | Coding Long-Horizon+1 |
| C | Team research listed on the research page | |
| C | HuzzleWorld computer-use models | Coding Computer Use+3 |
| C | Searching Over Sandbox States for Coding Agents | Coding Computer Use |
| C | No Runloop-authored papers/benchmarks found. Platform supports running third-party academic coding benchmarks | Coding |
| C | SheetBench-50 (released 2025-10-01) | Enterprise Workflows Long-Horizon+1 |
| C | FinchA benchmark of real finance and accounting workflows built around spreadsheets: creating, editing and… | Agentic |
| C | Frontier FinanceAn investment-workflow benchmark covering complete tasks from screening to profitability analysis. | Agentic |
| C | CLUEWSC | Reasoning |
| C | Graphwalks BFS 128KLong-context graph traversal benchmark using breadth-first search tasks. | Reasoning |
| C | RAGTruthCorpus of retrieval-augmented responses annotated word by word for unsupported or contradicted content. | Knowledge |
| C | IFStructLiquid AI benchmark of following structured-output instructions, such as producing output that matches a… | Instruction Following |
| C | StructEval-TMeasures whether a model produces valid structured output in text formats such as JSON, YAML and CSV. | Instruction Following |
| C | Human EvaluationHuman evaluation programs for model quality, usefulness, safety, and subjective output characteristics. | Alignment Multi-domain |
| C | A Matter of TASTE | Agentic |
| C | ArchBench Traceability Link RecoveryArchitecture Traceability Link Recovery: Recover traceability links between software architecture… | Coding |
| C | ArtifactLinkerScientific artifacts such as models and datasets are foundations for research. With the rapid growth of… | Machine Learning |
| C | AutoDFTDensity functional theory (DFT) serves as the basis for computational discovery in materials science and… | Materials |
| C | Beyond PerplexityTraining-free AI text detection methods primarily rely on model log-probabilities, achieving strong… | Classification |
| C | CaptchaMind | Computer Use |
| C | Chain-of-ProcedureRecent advances in vision-language models (VLMs) have achieved impressive results on standard image-text… | Multimodal Understanding |
| C | DramaDirector: Geometry-Guided Short Drama GenerationShort dramas, with their rapid shot rhythms, dialogue-driven focus shifts, and demanding cinematographic… | Video |
| C | HELM MedQAHELM MedQA: Evaluates clinical, biomedical, medical-exam, coding, or healthcare-document reasoning. | Healthcare |
| C | MLPerf EndpointsMLCommons benchmark for API-hosted GenAI endpoints, measuring serving latency, throughput, concurrency, and… | Inference |
| C | Physics-R1We audit the multimodal-physics evaluation pipeline end-to-end and document three undetected construction… | Reasoning |
| C | Agent-as-a-Router: Agentic Model Routing for Coding Tasks | Coding |
| C | Compositional Skill Routing for LLM AgentsLLM agents increasingly rely on external skills -- reusable tool specifications -- but real-world tasks often… | Agentic |
| C | ConvReConvRe evaluates language models' understanding of converse relations through Re2Text and Text2Re tasks in… | Knowledge |
| C | DualEvalCurrent LLM evaluation relies on two complementary but often disconnected signals: static benchmarks with… | Meta |
| C | DVMapCurrent Large Language Models (LLMs) typically rely on coarse-grained national labels for pluralistic value… | Alignment |
| C | EmplifAIThis paper introduces EmplifAI, a Japanese empathetic dialogue dataset designed to support patients coping… | Healthcare |
| C | Endless TerminalsEnvironments are the bottleneck for self-improving agents. Current terminal benchmarks were built for… | Agentic |
| C | From Talking to SingingWith rapid advances in audio-visual generative models, reliable forgery detection becomes increasingly… | Multimodal Understanding |
| C | GaMMAIn this paper, we propose GaMMA, a state-of-the-art (SoTA) large multimodal model (LMM) designed to achieve… | Audio |
| C | Generate README EvalBenchmark for generating structured README files from entire GitHub repositories, evaluating long-context… | Summarization |
| C | Governance DecayModern LLM agents increasingly rely on context compaction, summarization, or eviction to keep long-running… | Safety |
| C | Grounding Agent Memory in Contextual IntentDeploying large language models in long-horizon, goal-oriented interactions remains challenging because… | Agentic |
| C | GroundShotGenerating visually consistent multi-shot videos remains an open challenge. As videos span more shots,… | Video |
| C | HelicaseLLM-based multi-agent systems have been widely adopted for knowledge retrieval and report generation,… | Agentic Business |
| C | HyperEyesExisting multimodal search agents process target entities sequentially, issuing one tool call per entity and… | Agentic |
| C | InfiniteBM CoupHead-to-head LLM game-arena ladder for Coup, using InfiniteBM's per-game Bradley-Terry Elo ratings across… | Game |
| C | InfiniteBM WerewolfHead-to-head LLM game-arena ladder for Werewolf, using InfiniteBM's per-game Bradley-Terry Elo ratings across… | Game |
| C | InteractiveBench LogicInteractive proof domain on Situation Puzzle logic instances. | Knowledge |
| C | InterGPSInterpretable Geometry Problem Solver (Inter-GPS) with Geometry3K dataset of 3,002 geometry problems with… | Spatial Reasoning |
| C | LID BenchmarkLanguage identification benchmark comparing LID models across FLORES+, MADAR, Gherbal-Multi, ATLASIA-LID,… | Language |
| C | LiveStarProDespite the remarkable progress of Video Large Language Models (Video-LLMs), current online architectures… | Uncategorized |
| C | MAAT: Multi-phase Adapter-Aware Targeted UnlearningMachine unlearning evaluation is structurally skewed: Why-type questions, which probe causal and relational… | Safety |
| C | Membership Inference on LLMs in the WildMembership Inference Attacks (MIAs) act as a crucial auditing tool for the opaque training data of Large… | Cybersecurity |
| C | MemDelta | Agentic |
| C | MindClawTheory of Mind (ToM) enables an agent to reason about another actor's beliefs, goals, and intentions, which is… | Embodied |
| C | MMFormalizer: Multimodal Autoformalization in the WildAutoformalization, which translates natural language mathematics into formal statements to enable machine… | Multimodal Understanding |
| C | Molmo2Today's strongest video-language models (VLMs) remain proprietary. The strongest open-weight models either… | Video |
| C | OrgForge-ITSynthetic insider-threat detection benchmark built from OrgForge organizational simulation telemetry with… | Cybersecurity |
| C | PanoWorldMultimodal large laboratory models (MLLMs) still struggle with spatial understanding under the dominant… | Spatial Reasoning |
| C | PerceptionDLM | Multimodal |
| C | PhotoFlow: Agentic 3D Virtual Photography MissionsVirtual photography asks an agent to enter a prepared 3D scene with no preselected camera pose or reference… | Agentic |
| C | RadLIT-9Radiology education retrieval benchmark with 420 multiple-choice questions from 36 open-access review articles… | Retrieval Healthcare |
| C | Relevance as a VulnerabilityAI agents augment large language models with external tools such as web retrieval, enabling grounded and… | Safety |
| C | Reproducing Complex Set-Compositional Information RetrievalComplex information needs may involve set-compositional queries using conjunction, disjunction, and exclusion,… | Retrieval |
| C | ResearchMath-14KThe frontier of mathematics is defined by problems whose solutions are not yet known, yet it remains unclear… | Math |
| C | Right KnowledgeLarge language models can store both outdated facts and newer superseding facts in their parameters, but… | Factuality |
| C | Seg-AgentLanguage-guided segmentation transcends the scope limitations of traditional semantic segmentation, enabling… | Vision |
| C | Send a SCOUT FirstPrompt-injection detectors are heterogeneous: each is strong on a different slice of attacks, and none is… | Cybersecurity |
| C | SenFlowSentence-level AI-generated text detection (S-AGTD) for hybrid documents, where humans and LLMs co-author one… | Classification |
| C | STReasonerSpatio-temporal reasoning in time series involves the explicit synthesis of temporal dynamics, spatial… | Spatial Reasoning |
| C | SynAE | Uncategorized |
| C | The World is Not Mono | Audio |
| C | To Call or Not to CallLLM agents exhibit a consistent tendency to over-call, invoking tools even in situations where none is needed.… | Tool Use |
| C | TSCGProduction agent frameworks (OpenAI Function Calling, Anthropic Tool Use, MCP) transmit tool schemas as JSON,… | Tool Use |
| C | When2SpeakLarge Language Models (LLMs) excel at generating contextually appropriate responses but remain poorly… | Behavioral |
| C | X-Risks Leaderboard | Safety |
| C | MIIT-datasetMulti-image content has become an increasingly prevalent form of visual communication in social media, giving… | |
| B | Global MMLU LiteKaggle reports an average across 16 language dimensions of Global MMLU Lite, with separate culturally… | Multilingual |
| D | Vals Terminal-Bench 1.0 mirrorA Vals-hosted view of Terminal-Bench 1.0 with easy, medium, and hard task splits. | Agentic |
| D | LiveCodeBench Pass@1-COTThis lane contains DeepSeek's LiveCodeBench Pass@1-COT results. The explicit metric and prompting label keeps… | Coding |
| D | Cloudflare Decision Index 0.2.1: GPQA DiamondPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: GSM8KPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: HLEPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: MMLU-ProPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: MMLUPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | SWE-bench ExtraSWE-bench Extra: Evaluates software-engineering agents on realistic issue resolution, repository navigation,… | Coding |
| C | Scale Labs (launched 2026-03-09 | Coding Computer Use+2 |
| D | EMBEvaluating agents on Excel-based financial modeling tasks | Agentic |
| D | Finance Agent v2.1Financial research with tools, revised questions and rubrics, a new judge jury, and stricter treatment of… | Agentic |
| D | EMB v1.1Excel financial modeling from a template or from scratch, with updated grading in v1.1. | Agentic |
| D | Code Migration v1.1 | Coding |
| D | vexp-swe-bench | Coding |
| D | Voyager MinecraftVoyager Minecraft: Measures embodied-agent, navigation, manipulation, or simulated robotics task success. | Agentic |
| C | PostTrain Arena | Coding Computer Use+1 |
| C | Learning from Less: RLVR in Low Data RegimesLearning from Less: RLVR in Low Data Regimes (arXiv 2604.18381; MLSys 2026 per Snorkel) | Coding Computer Use+2 |
| C | ACE (AI Consumer Index) | Coding Enterprise Workflows+1 |
| C | Universal offline sandbox escape disclosureUniversal offline sandbox escape disclosure (https://primeintellect.ai/blog/universal-offline-sandbox-escape) | Coding Enterprise Workflows+2 |
| D | Hermes BenchTests agents on 150 tasks across 25 categories, including Hermes skills, research, diagrams and art, memory,… | Agentic |
| D | VideoQADeepMind's 1H-VideoQA tests recall and reasoning over videos lasting 40 to 90 minutes. The current… | Multimodal |
| D | ProtocolsDetects and fixes errors in molecular-biology protocols using document, code, and web-search tools. | Knowledge |
| D | KnowledgeLiquid AI reports 23.8 skill points for d1-3B and 8.3 for d1-omni-600M on Knowledge with the official Decision… | Decision Models |
| D | LanguageLiquid AI reports 56.4 skill points for d1-3B and 12.9 for d1-omni-600M on Language with the official Decision… | Decision Models |
| D | RetrievalLiquid AI reports 52.8 skill points for d1-3B and 35.0 for d1-omni-600M on Retrieval with the official… | Decision Models |
| D | ToolsLiquid AI reports 74.5 skill points for d1-3B and 15.1 for d1-omni-600M on Tools with the official Decision… | Decision Models |
| D | ArtsLiquid AI reports 36.3 skill points for d1-3B and 6.8 for d1-omni-600M on Arts with the official Decision… | Decision Models |
| C | Teen AI safety study | |
| C | SOTA speculative decoding | Coding |
| C | Morph Prover v0 7B | Coding |
| B | Vals RSI Index | |
| C | computer-1 computer-use harness in Harbor | Coding Computer Use+1 |
| C | GPU Glossary | Coding |
| C | Rubric-based RL papers | Coding Computer Use+2 |
| C | Kubernetes Is the Wrong Primitive for Diverse Workloads | Coding Computer Use |
| C | Evaluations cited in 23 frontier model system cards and reports listed on the vendor homepage (Claude family through Opus/Sonnet 5.5 | |
| D | DECK-BenchMoonshot AI's internal benchmark for presentation and deck-production workflows. | Agentic |
| D | Financial ModelingMoonshot AI's internal benchmark for formula linkage, financial logic, and final workbook quality in… | Agentic |
| D | EEBenchAn electrical-engineering agent benchmark maintained by atopile. | Coding |
| D | De novo protein-binder designDesigns diverse miniprotein binders against target protein structures in a GPU-equipped research sandbox. | Knowledge |
| D | CircaCirca pairs polar questions with indirect answers and labels how each answer should be interpreted. It tests… | Decision Models |
| D | DistressBenchMeasures crisis-support replies against clinician-written rubrics. These published results are excluded from… | Cross-domain |
| D | ExplorationBench AlienCodeTests whether a system can discover a programming language's hidden rules through experiments and apply them… | Agentic |
| D | Toolathlon Verified Pass@3Fraction of Toolathlon Verified tasks solved in at least one of three trials. | Agentic |
| D | Toolathlon Verified Pass³Fraction of Toolathlon Verified tasks solved in all three independent trials. | Agentic |
| D | AA AutomationBenchAn independently evaluated automation benchmark from Artificial Analysis. | Agentic |
| D | AA EnterpriseOps-GymAn independently evaluated enterprise-operations benchmark from Artificial Analysis. | Agentic |
| D | AA Harvey LAB v1.0Archived v1.0 legal-agent results from Artificial Analysis, kept separate from v1.1. | Agentic |
| D | AA ITBenchAn independently evaluated IT-operations benchmark from Artificial Analysis. | Agentic |
| D | AA Agentic IndexA display-only Artificial Analysis agentic index. | Agentic |
| D | LifeValOpenAI's benchmark of practical everyday tasks people want a personal assistant to handle, such as finding… | Agentic |
| D | Kimi Claw 24/7A Moonshot AI internal long-horizon agent benchmark for persistent professional coworking tasks. | Agentic |
| D | Business OperationMoonshot AI's internal benchmark for long-horizon management and decision-making in a simulated business… | Agentic |
| D | Financial ResearchMoonshot AI's internal benchmark for auditable financial research under evidence and cutoff constraints. | Agentic |
| D | DECK-Arena | Agentic |
| D | Design Bench | Agentic |
| D | MLE-Bench LiteA lightweight machine-learning competition benchmark that measures whether models can iteratively train,… | Agentic |
| D | MCP-Tasks | Agentic |
| D | WideResearchA broad research-agent benchmark for open-ended information gathering, synthesis, and answer construction… | Agentic |
| D | MEWCA benchmark that evaluates AI agents on multi-environment web challenges, testing navigation and task… | Agentic |
| D | ACE solvedNumber of advanced cyber-range challenges solved in the joint NIST CAISI and UK AISI preliminary evaluation. | Agentic |
| D | The Last Ones stepsAverage step reached on a 32-step long-horizon cyber range. | Agentic |
| D | FrontierCyber | Agentic |
| D | CyScenarioBench successAverage success rate across realistic, long-horizon cybersecurity scenarios. | Agentic |
| D | Atomic vulnerability research | Agentic |
| D | Atomic evasion | Agentic |
| D | SCONE post-cutoff successShare of SCONE smart-contract vulnerabilities exploited on the 12-task post-cutoff set. | Agentic |
| D | Firefox 147 exploitsShare of patched Firefox 147 JavaScript-engine targets for which the model produced a working… | Agentic |
| D | Harvey's Legal Agent Benchmark v1Archived Vals legal-agent results before the tracked-changes grading fix. | Agentic |
| D | FACTS Search V1The archived FACTS Search V1 protocol measures factual answering with Brave Search and a Gemini 2.0 Flash… | Agentic |
| D | FrontierCode 1.1 MainCognition's 100-task software-engineering benchmark for whether coding agents produce mergeable,… | Coding |
| D | AA Coding Agents | Coding |
| D | VIBE V2A display-only MiniMax provider benchmark for end-to-end coding-agent and product-building tasks. | Coding |
| D | Biomedical image analysisQuantitative biomedical image analysis across segmentation, detection, measurement, classification,… | Multimodal |
| D | BabyVision w/ PythonA Python-assisted BabyVision evaluation for fine-grained visual perception and grounded reasoning. | Multimodal |
| D | ZeroBench w/ Python | Multimodal |
| D | MMMU-Pro w/ PythonTool-augmented MMMU-Pro variant that allows Python assistance during multimodal reasoning. | Multimodal |
| D | Liquid Extract JSON ValidityA display-only Liquid AI extraction metric measuring the share of image-to-JSON outputs that parse as strict… | Multimodal |
| D | Liquid Extract F1A display-only Liquid AI extraction metric measuring field-name agreement between requested schema fields and… | Multimodal |
| D | Liquid Extract VLM JudgeA display-only Liquid AI extraction metric measuring judged agreement between extracted values and the source… | Multimodal |
| D | MStarA general visual question-answering benchmark used in provider tables for real-image reasoning quality. | Multimodal |
| D | ChatCVQAA conversational visual QA benchmark that tests multi-turn grounded answering over images and documents. | Multimodal |
| D | AI2D_TESTA diagram understanding benchmark focused on scientific and educational visual question answering. | Multimodal |
| D | VoiceCodeBenchTests whether speech-to-text models preserve exact structured values in English workplace audio. | Multimodal |
| D | Medicinal chemistrySelects compounds with the highest or lowest measured ADME-related property from molecular structures and… | Knowledge |
| D | Organic chemistry V2Chemistry tasks covering spectroscopy, synthesis planning, reaction prediction, and chemical structure images. | Knowledge |
| D | AA-Omniscience Hallucination RateA display-only Artificial Analysis factuality metric for the rate of incorrect answers among non-correct… | Knowledge |
| D | Long-context translationCohere's long-document translation evaluation: two book chapters translated in a single call, scored per… | Multilingual |
| D | SysBenchTests how well a model follows the rules in a system message across single and multi-turn conversations. | Instruction Following |
| D | KindBenchA behavioral benchmark that tests psychological safety across sixteen adversarial multi-turn conversations… | Instruction Following |
| D | ACCR standardShare of approved advanced-cyber requests completed rather than refused under standard GPT-5.6 Sol safeguards. | Instruction Following |
| D | ACCR Daybreak RedShare of approved advanced-cyber requests completed rather than refused by GPT-5.6 Cyber under Daybreak Red… | Instruction Following |
| D | JevBenchTyped-decision evaluation across 1,624 decisions, combining chance-corrected intelligence, probability… | Decision Models |
| D | Cloudflare Decision Index 0.2.1: BPoMPPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: HumicroeditPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: POP909-CLPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: cfcolorPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ForecastBenchPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: Habermas MachinePublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: New YorkerPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ChessBenchPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: MuSRPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: SATA-BenchPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: CRUXEvalPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: CLadderPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: BBHPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ContractNLIPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ANLIPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: WinoGrandePublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: HellaSwagPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ACOSPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: FinEntityPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: iSarcasmEvalPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: VASTPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: NLI4CTPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: RAGTruthPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: BANKING77Published Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: CLINC150+OOSPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: BRIGHTPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: Amazon ESCIPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: HoVerPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: BFCLPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ToolRetPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: API-BankPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: Home appliance simulatorPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: When2CallPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: RouterBenchPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: SGD/SGD-XPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ARC-EasyPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: ARC-ChallengePublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: OpenBookQAPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: CommonsenseQAPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: Support-ticket calibrationPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | Cloudflare Decision Index 0.2.1: Email spam classificationPublished Cloudflare and community decision-model configurations. | Decision Models |
| D | JevBench public hard | Decision Models |
| D | Perplexity Decision PanelPerplexity reports accuracy across 11 decision tasks on a fixed 7,210-row panel. The overall percentage… | Decision Models |
| D | Knowledge and Reasoning | Decision Models |
| D | Language Understanding | Decision Models |
| D | Retrieval and Classification | Decision Models |
| D | Tools and Automation | Decision Models |
| D | Arts and Human Taste | Decision Models |
| D | Decision Index ToolsDecision Index 0.4 publishes chance-corrected tools scores for 209 evaluated systems. | Decision Models |
| D | Decision Index ArtsDecision Index 0.4 publishes chance-corrected arts scores for 209 evaluated systems. | Decision Models |
| D | AA-Omniscience-Decision accuracyAA-Omniscience-Decision accuracy is one component of the Artificial Analysis Decision Index v1.0, run by… | Decision Models |
| D | AA-Omniscience-Decision non-hallucination rateAA-Omniscience-Decision non-hallucination rate is one component of the Artificial Analysis Decision Index… | Decision Models |
| D | KMMLU-ProKorean National Professional Licensure exams evaluating professional-grade knowledge. | Korean |
| D | CLIcKEvaluates Korean culture and linguistics. | Korean |
| D | CAIS Text LeaderboardA Center for AI Safety dashboard view summarizing text capabilities across HLE and TextQuests. | Cross-domain |
| D | Cohere Embed 5 launch: ViDoRe V3 parsed-text average | Cross-domain |
| D | Cohere Embed 5 launch: ViDoRe V3 Finance | Cross-domain |
| D | Cohere Embed 5 launch: Parsed PDF full-suite average | Cross-domain |
| D | Cohere Embed 5 launch: Fused text-image average | Cross-domain |
| D | Cohere Embed 5 launch: Page-image finance average | Cross-domain |
| D | Cohere Embed 5 launch: European-language composite average | Cross-domain |
| D | Cohere Embed 5 launch: Japanese retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Korean retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Arabic retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Farsi retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Hindi retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Bengali retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Telugu retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Indonesian retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Thai retrieval composite | Cross-domain |
| D | Cohere Embed 5 launch: Document throughput | Cross-domain |
| D | Cohere Embed 5 launch: Shared-space corpus/query pairings | Cross-domain |
| D | Cohere Rerank 5 launch: ViDoRe V3 average | Cross-domain |
| D | Cohere Rerank 5 launch: BEIR average | Cross-domain |
| D | Cohere Rerank 5 launch: BRIGHT average | Cross-domain |
| D | Cohere Rerank 5 launch: nanoMTEB average | Cross-domain |
| D | Cohere Rerank 5 launch: Parsed PDFs average | Cross-domain |
| D | Cohere Rerank 5 launch: Spanish retrieval | Cross-domain |
| D | Cohere Rerank 5 launch: French retrieval | Cross-domain |
| D | Cohere Rerank 5 launch: German retrieval | Cross-domain |
| D | Cohere Rerank 5 launch: Arabic retrieval | Cross-domain |
| D | Cohere Rerank 5 launch: Japanese retrieval | Cross-domain |
| D | Cohere Rerank 5 launch: Korean retrieval | Cross-domain |
| D | Cohere Rerank 5 launch: calibration error vs human graders | Cross-domain |
| D | Cohere Rerank 5 launch | Cross-domain |
| D | A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models | Truthfulness |
| D | AMC 12AMC 12: Measures mathematical reasoning, symbolic problem solving, proof construction, or competition-style… | Math |
| D | An Empirical Study of On-Device Translation for Real-Time Live-Stream Chat on Mobile DevicesDespite its efficiency, there has been little research on the practical aspects required for real-world… | Translation |
| D | Beyond Global ReplanningReal-world computer-use tasks often span multiple applications and devices, requiring agents to coordinate… | Agentic |
| C | Beyond the GUI ParadigmRecent advances in mobile agents are dominated by the GUI paradigm, in which agents perceive UI information… | Mobile |
| D | CiteCheck | Retrieval |
| C | CORTEXThe continuous evolution of large language models drives escalating demands on data scale and quality, and as… | Retrieval |
| D | Cybersecurity CTFsCybersecurity Capture the Flag (CTF) benchmark for evaluating LLMs in offensive security challenges. Contains… | Cybersecurity |
| D | EvoTrace | Coding |
| D | InferenceLatency.comLive provider-level inference latency, throughput, cost, and reliability tracker for hosted language-model… | Inference |
| C | Lumos-NexusConnector-based video unified models have demonstrated strong capability in instruction-grounded video… | Video |
| D | MIMIC-IV-Ext-BenchMIMIC-IV-Ext-Bench: Evaluates clinical, biomedical, medical-exam, coding, or healthcare-document reasoning. | Healthcare |
| D | NodeSynth: Socially Aligned Synthetic Data for AI EvaluationRecent advancements in generative AI facilitate large-scale synthetic data generation for model evaluation.… | Safety |
| D | Open X-Embodiment / RT-XCross-robot embodiment benchmark and dataset collection for evaluating vision-language-action policies across… | Robotics |
| D | RAGASRAGAS: Measures long-context retrieval, needle finding, summarization, factual grounding, or… | Generalization |
| C | Sarashina2.2-TTSWhile large language model (LLM)-based text-to-speech (TTS) systems have achieved high-quality speech… | Speech |
| D | SWE-GymSWE-Gym: Evaluates software-engineering agents on realistic issue resolution, repository navigation, testing,… | Coding |
| D | SWE-smithSWE-smith: Evaluates software-engineering agents on realistic issue resolution, repository navigation,… | Coding |
| C | TailorMindPersonalized content systems depend on available UGC and struggle when suitable content is absent, delayed, or… | Multimodal |
| D | Design Arena — Agentic Game DevelopmentCategory: multimodal. Leader: . Rows: 28. | Multimodal |
| D | Design Arena — HTML SlidesCategory: multimodal. Leader: . Rows: 29. | Multimodal |
| A | BenchLM | General Knowledge |
| D | BBO-PileMost black-box optimization methods require extensive hyperparameter tuning, often limiting their ability to… | Machine Learning |
| C | Beyond Static ToolsThe central challenge of AI for Science is not reasoning alone, but the ability to create computational… | Tool Use |
| D | Decoupling Spatio-Temporal Adapter for Fine-Grained Badminton Action LocalizationTemporal Action Localization (TAL) has been extensively studied in generic video understanding, while… | Video |
| B | FLORES European Languages LeaderboardEuropean-language subset of FLORES-200 machine translation, focusing on translation pairs involving Polish and… | Multilingual |
| B | Galileo Agent Leaderboard | Agentic Business+1 |
| A | GSMA Open Telco Leaderboard | Domain |
| D | HH-RLHFHH-RLHF: Measures model robustness, truthfulness, calibration, bias, harmfulness, jailbreak resistance, or… | Generalization |
| C | Hista and NumcaReinforcement learning (RL) refines large language models (LLMs) by directly optimizing model behavior through… | Machine Learning |
| D | HypersimHypersim evaluates 3D grounding and depth understanding in synthetic indoor scenes. | Spatial Reasoning |
| D | InferBench Text-to-Image | Multimodal |
| C | InfiniteBM ChessHead-to-head LLM game-arena ladder for chess, using InfiniteBM's per-game Bradley-Terry Elo ratings across… | Game |
| D | Integrated Safety Evaluation | Safety |
| D | L-MARSLong-form Multi-hop Agentic Reasoning benchmark for legal search, centered on LegalSearchQA tasks requiring… | Legal |
| D | Large-Scale Aspect-Based Sentiment Analysis with Reasoning-Infused LLMsWe introduce Arctic-ABSA, a collection of powerful models for real-life aspect-based sentiment analysis… | Language-understanding |
| C | LLawCoEmbodied agents operating in decentralized and partially observable environments have attracted growing… | Embodied |
| D | (long-context agent monitoring benchmark — not explicitly named in abstract)Monitoring coding agents for dangerous behavior using language models requires classifying transcripts that… | Long Context |
| C | OMLAB Open Agent Multimodal Leaderboard | Multimodal |
| A | OpenVLM Leaderboard | Multimodal Understanding |
| C | Optimum LLM Perf Leaderboard | Inference |
| D | OS-AtlasGUI grounding benchmark and dataset for evaluating foundation models on operating-system interface… | GUI |
| D | OS-CopilotOS-Copilot: Measures browser, desktop, mobile, or GUI agents operating in interactive environments. | Agentic |
| D | Pokémon Autonomous Completion | Game |
| D | Red QueenCurrent approaches to LLM adversarial testing suffer from coverage gaps: manual red-teaming does not scale,… | Safety |
| D | Sandboxed Coding Agents are Competitive Omni-modal Task SolversAs multimodal LLMs increasingly target video and audio, it is often assumed that such tasks require native… | Agentic |
| D | SeeClickGUI grounding benchmark and model suite for visual GUI agents that must map natural-language instructions to… | GUI |
| D | Self-Ensembling Vision-Language Models for Chart Data ExtractionCharts effectively convey quantitative information, but the underlying data are often locked in image form,… | Document Understanding |
| D | Semantically Aware UAV Landing Site Assessment from Remote Sensing Imagery via Multimodal Large Language ModelsSafe UAV emergency landing requires more than just identifying flat terrain; it demands understanding complex… | Aerospace |
| D | SomaliBench EvalLarge language model safety evaluation remains heavily English-centered, leaving low-resource languages… | Safety |
| D | TAAFExecution traces are a critical source of information for understanding, debugging, and optimizing complex… | Software Engineering |
| D | TabArena All Tasks | Tabular |
| D | TemporalBenchTemporalBench evaluates LLM-based agents on contextual and event-informed time-series tasks spanning multiple… | Agentic |
| D | The Heterogeneous Safety Impacts of Benign Multilingual Fine-TuningFine-tuning a large language model is a ubiquitous method for enhancing its capability on a specific… | Safety |
| C | The Misattribution GapMulti-agent AI pipelines typically assume that agent misconduct originates from model misalignment. We… | Safety |
| D | Tools as Continuous Flow for Evolving Agentic ReasoningLarge Language Models (LLMs) have demonstrated remarkable capabilities in orchestrating tools for reasoning… | Agentic |
| D | TRACERData contamination is a known threat to the reliability of model evaluation. However, it remains underexplored… | Coding |
| C | Ukrainian LLM Leaderboard | Language |
| D | Uncertainty Decomposition for Clarification Seeking in LLM AgentsRecent position papers argue that the classical aleatoric/epistemic uncertainty framework is insufficient for… | Agentic |
| D | Urban Socio-Semantic Segmentation with Vision-Language ReasoningAs hubs of human activity, urban surfaces consist of a wealth of semantic entities. Segmenting these various… | Geospatial |
| C | VisualClawVision language models are serving as general-purpose interfaces for complex multimodal tasks. However,… | Agentic |
| D | Why LLMs Fail at Causal Discovery and How Interventional Agents EscapeCausal discovery is a cornerstone of scientific reasoning, yet whether large language models can perform it… | Reasoning |
| D | WM Bench | Embodied |
| A | YALL Nous Leaderboard | Knowledge |
| C | ZONOS2 Technical ReportWe present ZONOS2 8B, our latest TTS model, which achieves state-of-the-art naturalness, prosody, and voice… | Speech |
| C | Transformer Scalability Crisis | Efficiency |
No entries match. Clear the filter above to see all 3228.