Medicine uses many types of information at once, such as images and text. Researchers from Yale and collaborators built MedPMC, an automated framework that pulls images and their descriptive text from permissively licensed articles in PubMed Central (PMC). Applied to 6.1 million PMC articles, MedPMC produced 11 million curated image–text pairs intended to train and evaluate medical multimodal models.

MedPMC works as a five-stage pipeline. The stages are initial screening, multi-panel figure detection, multi-panel figure separation, caption separation and alignment, and medical figure classification. Each stage uses purpose-built models and benchmarks, and the group reports strong component performance: initial screening F1 = 93.2, multi-panel figure detection F1 = 96.5, figure separation mean average precision = 89.8, caption separation and alignment F1 = 81.4 and ROUGE-L = 85.3, and medical figure classification F1 = 96.5.

To test whether higher-quality literature curation helps models, the team trained a CLIP-style vision–language encoder called MedPMC-CLIP. CLIP-style means the model learns to match images and text in a shared space. MedPMC-CLIP outperformed a strong biomedical CLIP baseline (BMC-CLIP) by an average of 7.1 percentage points in zero-shot area under the curve across 26 public benchmarks covering 11 specialties, even though it used fewer than half as many image–text pairs. As a vision encoder inside a multimodal large language model (MLLM), it improved medical visual question answering by 1.9 and 16.9 percentage points on two benchmarks. In a real clinical test using 10,524 dermatology photos from the Yale New Haven Health System, it improved morphology-to-image retrieval Recall@5 by 11.7 percentage points.

Human review supports the automated measures. Five annotators, including three with medical training, judged 95.3% of MedPMC images to be medically relevant. By contrast, a prior PMC-derived dataset contained many non-medical visuals: the authors report that 80.3% of images in a recent 24-million-pair dataset were non-medical and that only 19.7% of that dataset’s images were medically relevant by their review. The MedPMC corpus also spans many specialties and includes clinically important imaging types such as pathology, radiology, ophthalmology, dermatology, endoscopy, gross pathology photography, and functional imaging modalities.