MedStruct-S C

AI benchmark · Biomedicine

Publisher
Paper Arxiv
Domains
Biomedicine Healthcare
Tier
C why
Snapshot

What is MedStruct-S?

Semi-structured information extraction (IE) from OCR-derived clinical reports is crucial for efficiently reconstructing patients' longitudinal medical histories. In practice, this scenario commonly involves three tasks: (i) field-header (key) discovery, (ii) key-conditioned question answering (QA), and (iii) end-to-end key-value pair extraction. However, existing evaluations often under-model two factors: heterogeneous and incompletely known key representations, and OCR-induced noise. This makes it difficult to assess model robustness in real-world settings. We present MedStruct-S, a benchmark specifically designed to evaluate these tasks under unknown keys and OCR noise. MedStruct-S contains 3,582 annotated real-world clinical report pages. Using MedStruct-S, we benchmark two representative paradigms: encoder-only sequence labeling with post-processing and decoder-only structured generation, covering four encoder-only and five decoder-only models spanning 0.11B to 103B parameters. Our results show that encoder-only models achieve the best performance for non-null-value key-conditioned QA despite being substantially smaller than decoder-only models. When comparing models of similar order of magnitude, encoder-only models still perform better overall. Without controlling for model scale, fine-tuned decoder-only models deliver the strongest overall results. These findings show that the benchmark provides a reliable and practical basis for selecting and comparing models across different semi-structured IE settings.

Tier C

C

Niche: a handful of models scored, or built by an RL vendor.

How much stands behind a tier: usage, who backs it, a confirmed line from its own site or paper, and a clear read on X. High with three of the four, medium with two, low otherwise.

  • 9 models scored
  • in curated RL directories (1 pt)
How we got to tier C
  1. Adoption: 9 models scored. Tier C.
  2. Not yet read on X.
  3. In curated RL directories: 1 point, a floor of none.
  4. Not yet confirmed by a line from its own site or paper.

A clear view on X moves a tier one step up or down; most entries don't move. How tiers work.

Not on the map yet: no line from its own site or paper places it in a domain.

More Biomedicine AI benchmarks

Next: browse all benchmarks. Entry from the RL Research daily scrape of public sources, 2026-10-11 snapshot.