Microsoft’s MAI-DxO diagnostic system outperformed participating generalist physicians on complex diagnostic cases in a peer-reviewed study published October 22, 2026 in Science, which reported 83.2% accuracy for the system on New England Journal of Medicine case records against 36.1% for physicians using reference resources.
The study, “Sequential diagnosis with language models,” appears in Science Volume 394, Issue 6822, pages 423–428; it was received April 4, 2026 and accepted August 10, 2026. All 15 authors are Microsoft employees, and the list includes Microsoft AI chief executive Mustafa Suleyman alongside Dominic King and Eric Horvitz. In a same-day announcement, Microsoft said the work “is published today in Science” in the journal’s “AI in Medicine” issue.
A Benchmark Built Around Step-by-Step Diagnosis
The paper introduces the Sequential Diagnosis Benchmark, or SDBench, which converts 304 New England Journal of Medicine clinicopathological conference cases and 325 emergency department-to-admission encounters from the MIMIC-IV-ED dataset into interactive, stepwise diagnostic encounters. Each case opens with a brief clinical vignette, from which the diagnostician, whether a physician or an AI system, can ask history or examination questions, order diagnostic tests, or commit to a final diagnosis. Performance is scored on diagnostic accuracy against the reference diagnosis and on the cumulative estimated cost of the tests ordered.
Two supporting agents structure each encounter. A Gatekeeper agent mediates every information request against the underlying case record and generates diagnostically neutral synthetic results when a requested finding is absent, so missing data does not signal the answer. A Judge agent scores the committed diagnosis against the reference diagnosis on a five-level physician-authored rubric, with scores of 4 or 5, representing diagnoses for which management would be unchanged, counted as correct. Test costs were estimated using CPT-based pricing derived from the Kaiser Permanente Washington Central Hospital Standard Charges file, with history and examination questions folded into a fixed physician-visit cost.
Within the benchmark, the authors evaluated the Microsoft AI Diagnostic Orchestrator, abbreviated DxO in the paper, a model-agnostic system that simulates a panel of clinical experts with complementary roles, applies cost-conscious test stewardship at each step, and aggregates diagnoses across independent runs. Microsoft’s announcement describes the system as operating like a panel of virtual physicians debating over several rounds. The authors argue that static benchmarks, which present all case information upfront and ask for a single answer, cannot measure whether a system asks the right questions, avoids indiscriminate testing, or recognizes when it has enough certainty to conclude a workup.
Results on NEJM and Emergency-Department Cases
On the clinicopathological conference cases, DxO paired with GPT-5 and run without budget constraints achieved 83.2% accuracy at an average cost of $5,431 per case. Cost-constrained configurations reached 80.9% accuracy at $2,241 and 79.6% at $1,303, while a configuration aimed at diagnostic excellence reached 84.5% at $8,945. In the accompanying physician study, conducted on 56 cases held out from engineering work, participants produced 764 closed-book and 662 open-book diagnoses, averaging 22.3% accuracy closed-book at $3,049 per case and 36.1% open-book at $4,725. The single best physician reached 51.8%. The paper also reports differing workup styles: physicians asked a mean of 5.6 questions before ordering their first test, while DxO with GPT-5 asked a mean of 25.5, a strategy the authors describe as extensive structured history-taking.
On the emergency-department cases, DxO without budget constraints achieved 80.3% accuracy at $4,979 per case and 73.2% at $2,634 under budget constraints. Generalist physicians averaged 46.8% at $3,400, and the best individual physician reached 62.5%. Across all models tested with minimal baseline prompts, DxO orchestration raised diagnostic accuracy by an average of 7.9 percentage points on the conference cases and 4.4 points on the emergency-department cases. Microsoft’s announcement states that no other evaluated human or AI approach was simultaneously more accurate and less costly than MAI-DxO, describing the system as 1.7 times more accurate than physicians on the emergency-department presentations, more than twice as accurate as physicians using textbooks and the internet on the journal cases, and nearly four times as accurate as physicians diagnosing without those resources, at half to one third of the cost.
DxO was initially developed when GPT-4.1 was the strongest model available, was subsequently evaluated with GPT-5, and the full evaluation suite was repeated with GPT-5.5 before publication without modifying prompts or orchestration, with its accuracy advantage over the baseline prompt increasing. An earlier version of the work was posted as a preprint on June 27, 2025, reporting 80% accuracy with OpenAI’s o3 model, four times the 20% physician average, and 85.5% in a maximum-accuracy configuration; the peer-reviewed version adds the emergency-department case set and updated evaluations with GPT-5 and GPT-5.5.
Stated Limitations and Author Disclosures
The authors state that the comparisons do not show superiority over specialists, multidisciplinary teams, or real-world clinical practice, and write that they would expect the performance gap to narrow against expert specialists or teams matched to each case. They note the measured advantages are amplified within a controlled simulation that sets aside uncertain patient histories, incomplete documentation, rapport, longitudinal follow-up, and multimodal data such as imaging and pathology slides, and that the journal cases may overrepresent diagnostically challenging presentations. A further stated limitation is the comparison against individual physicians rather than multispecialty collaboration, and the Judge rubric evaluates diagnostic correctness at the disease level rather than therapeutic appropriateness, safety, or longitudinal management.
In the paper’s disclosures, the authors state that all of them are Microsoft employees holding equity in the company, that several are inventors on a Microsoft patent application related to an automated evaluation framework for sequential diagnosis, and that no funds external to Microsoft were acquired for the studies.
Open-Source Release and Clinical Partnerships
Microsoft said it is open-sourcing SDBench for select NEJM cases, including the benchmark, Gatekeeper, Judge, and cost-estimation code. The code is available on GitHub under the MIT License, with a configuration named “science” that pins the gatekeeper and judge used for the paper’s results. According to the SDBench project site, the dataset derived from the journal cases requires licensing from NEJM Group, the MIMIC subset is distributed through PhysioNet subject to credentialing, and five fully synthetic cases ship as the default open dataset.
A companion position paper, “Medical Superintelligence: Foundations and Aspirations,” introduces six interdependent dimensions of medical intelligence and states that crossing the threshold to medical superintelligence will require “more than superhuman performance on isolated tasks,” namely systems that surpass today’s strongest human-led health systems in integrated medical reasoning across the full arc of care. Microsoft’s announcement states that the paper was published in BMJ Digital Health and AI. The company also said it is bringing health teams from across Microsoft, including Microsoft Research scientists and the teams behind Dragon Copilot and PowerScribe, into an integrated effort at Microsoft AI, and it referenced its previously announced Mayo Clinic collaboration; under a June 2, 2026 announcement, the organizations are developing a frontier AI model for healthcare that Mayo Clinic will own and that Microsoft plans to make available through Azure Foundry APIs.
Through a new partnership with Beth Israel Deaconess Medical Center in Boston, Microsoft said it will evaluate clinical reasoning capabilities across hospital settings in a prospective study in which patients interact with the system directly and performance is measured against physicians’ independent assessments.
Credit: Source link
