Frontier models learn mostly from public internet data. However, clinical neuroimaging rarely appears there, because MRI and CT scans contain identifiable facial features. Consequently, general models underperform on brain-imaging tasks. A University of Michigan research team addresses this gap with NeuroVFM, published in Nature Medicine.
What is NeuroVFM?
At its core, NeuroVFM is a generalist visual foundation model for neuroimaging. Specifically, it was trained on 5.24 million clinical MRI and CT volumes. These came from 566,915 studies in the UM-NeuroImages dataset. That data spans over two decades of routine care at Michigan Medicine.
The research team call their approach ‘health system learning.’ In short, the model learns from uncurated data generated during normal clinical operations. Therefore, it avoids the bottleneck of paired radiology reports. It also avoids the disease-specific curation used in narrow classifiers.
Notably, the base model is called Vol-JEPA. It extends the earlier I-JEPA and V-JEPA methods to volumetric medical images. This reflects a wider trend: JEPA-style learning is expanding into medical imaging.
How Vol-JEPA Works?
Vol-JEPA is a self-supervised, vision-only algorithm. Rather than reconstructing pixels, it predicts representations in a learned latent space. As a result, it needs no labels, no report text, and no voxel decoder.
First, each 3D volume is tokenized into non-overlapping 4×16×16-voxel patches. Next, the volume is split into a small visible context and a larger masked target. A student encoder then processes the context patches.
Meanwhile, a predictor combines context latents with target position encodings. It predicts the masked-region latents. A teacher encoder generates the ground-truth target latents. This teacher is an exponential moving average (EMA) of the student. Training minimizes a smooth L1 loss between predicted and teacher latents, with gradients stopped through the teacher.
Importantly, masking is foreground-focused, using precomputed head masks. Context ratios are 25% for MRI and 20% for CT, with 20% patch dropout. This design encourages the encoder to model shared neuroanatomy rather than background shortcuts.
Performance
To measure this, the research team froze every encoder and trained identical study-level attentive probes. The primary endpoint was macro-averaged AUROC across 156 diagnostic tasks. These cover 74 MRI and 82 CT diagnoses.
Consequently, NeuroVFM reached 92.68 AUROC on CT and 92.49 on MRI. Moreover, it outperformed every baseline on the aggregate endpoint.
| Model | Pretraining data | Learning objective | AUROC margin vs NeuroVFM |
|---|---|---|---|
| NeuroVFM | UM-NeuroImages (health system) | Vol-JEPA latent prediction | — (92.68 CT / 92.49 MRI) |
| HLIP | UM-NeuroImages | Report / language supervision | −0.98 |
| PRIMA | UM-NeuroImages (MRI only) | Report / language supervision | −3.87 |
| NeuroMAE | UM-NeuroImages | Voxel reconstruction (MAE) | −1.55 |
| DINOv3 | 1.7B natural images | Self-supervision (2D) | −2.24 |
| BiomedCLIP | 15M image–text pairs | Vision–language (2D) | −2.88 |
Because PRIMA, HLIP, and NeuroMAE share the same training data, these gaps isolate the objective. In other words, latent prediction outperformed both report supervision and voxel reconstruction here.
Training efficiency was also reported. For instance, a full Vol-JEPA run used fewer than 1,000 GPU hours. It trained over 7× faster than a 3DINO baseline. It also fit 16× larger batches at equal memory. The base encoder has 85.8M parameters; the small variant has 21.7M.
What the Model Enables?
Beyond diagnosis, NeuroVFM supports several downstream tasks. Each one reuses the same frozen visual tokens.
- Report generation: The team paired frozen NeuroVFM with Qwen3-14B in a LLaVA-1.5 style. This NeuroVFM-LLaVA system generated structured key findings.
- Triage: Findings passed to a reasoning model produced acuity levels: unremarkable, routine, or urgent.
- Grounded predictions: An attention-based MIL pooler maps findings to image regions, without region-level annotations.
- Cross-modal transfer.:A CT-trained probe ran on MRI with under a 5-point AUROC drop.
On generation and triage, NeuroVFM-LLaVA outperformed frontier baselines.
| Metric | NeuroVFM-LLaVA | GPT-5 | Claude Sonnet 4.5 |
|---|---|---|---|
| Three-tier acuity accuracy | Best | −11.0 | −20.3 |
| Prospective balanced accuracy | 92.6% | 71.2% | — |
| Critical-finding miss rate | 13.5% (21/155) | 50.3% (78/155) | — |
| Report inference cost | Baseline | >24× higher | — |
Furthermore, the research team ran a silent one-week prospective study across the health system (n=1,155). There, NeuroVFM reached 92.6% balanced triage accuracy versus 71.2% for GPT-5. However, its sensitivity was 86.5%, so 21 of 155 critical findings were missed. The authors therefore frame it as decision support, not autonomous screening.
Running NeuroVFM
In practice, using the released stack is straightforward. You install the package, then call the pipeline helpers. The snippet below uses the repository’s own API.
# git clone https://github.com/MLNeurosurg/neurovfm.git && cd neurovfm && pip install -e .
from neurovfm.pipelines import (
load_encoder, load_diagnostic_head, load_vlm, interpret_findings,
)
encoder, preprocessor = load_encoder("mlinslab/neurovfm-encoder")
dx_head = load_diagnostic_head("mlinslab/neurovfm-dx-ct")
generator, gen_preproc = load_vlm("mlinslab/neurovfm-llm")
# Encode a study, then predict diagnoses
batch = preprocessor.load_study("/path/to/ct/study/", modality="ct")
embeddings = encoder.embed(batch) # [N_tokens, 768]
predictions = dx_head.predict(embeddings, batch) # [(label, prob, pred), ...]
# Generate preliminary findings, then optionally triage
vols = gen_preproc.load_study("/path/to/ct/study/", modality="ct")
findings = generator.generate(vols, clinical_context="LOC and nausea.")
triage = interpret_findings(findings, "LOC and nausea.", api_key="...")Note that the stack requires FlashAttention-2 (v2.6.3) built from source. Code ships under an MIT license. Weights use CC-BY-NC-SA-4.0, and some require access approval with an institutional email.
Strengths and Limitations
Taken together, the research paper reports advantages alongside constraints. Both are grounded in its own evaluations.
Strengths
- Learns from uncurated scans, without report or label supervision.
- Shares one latent space across CT and MRI.
- Report generation was reported as >24× cheaper and >23× less carbon-intensive than GPT-5.
- Performance held across manufacturers, field strengths, and demographic subgroups.
Limitations
- 86.5% triage sensitivity means real critical-finding misses remain.
- Weights are non-commercial, and the model is not FDA-approved.
- The model is susceptible to dataset, architecture, and objective bias.
- Results come from a single academic health system.
Check out the Paper and Repo. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
Credit: Source link
