AI Diagnostic Panel Outperforms Doctors on Complex Cases by a Wide Margin

83 percent. That is the accuracy rate achieved by Microsoft’s new AI diagnostic system, MAI-DxO, when tested against a set of complex medical cases from the New England Journal of Medicine. The figure stands in stark contrast to the 36 percent accuracy rate recorded by generalist physicians, including primary care doctors and general internists, who worked individually with reference materials. This result, published in the journal Science on Thursday, highlights a significant performance gap in handling difficult diagnoses. However, researchers note that such complex cases are typically managed by specialists or teams in real-world settings, meaning the comparison is not directly equivalent to standard clinical practice.

The system operates by coordinating multiple AI agents that function like a panel of physicians. These agents interrogate the patient, order necessary tests, and engage in internal debates to reach a consensus on the diagnosis. Mustafa Suleyman, the CEO of Microsoft AI, described this capability as a critical step toward achieving medical superintelligence. The concept involves creating systems that can surpass the reasoning and coordination abilities of current human-led health systems across the entire arc of patient care. This marks a shift from simple task completion to integrated clinical support.

A key insight from the study is that much of the performance improvement stems from the underlying power of the AI models themselves rather than the orchestration system alone. Microsoft tested the framework using models from various companies, including OpenAI, Google, Anthropic, Meta, and DeepSeek. The highest scores were achieved using OpenAI’s GPT-5 and GPT-5.5. While the orchestration layer added gains on top of these powerful models, the data suggests that the base model capability is the primary driver of accuracy. This distinction is crucial for understanding where future improvements will likely come from.

Cost and efficiency were also central metrics in the evaluation. When using GPT-5 with a basic prompt, the system cost approximately $8,400 per case and achieved a 78.9 percent accuracy rate. The DxO orchestration reduced this cost by about $3,000 while raising accuracy by only four points. In emergency department scenarios, the difference in both cost and accuracy between the standalone model and the orchestrated system was minimal. This suggests that for routine or emergency cases, the added complexity of the multi-agent panel may not offer significant advantages over single-model approaches.

The impact of the orchestration system becomes more pronounced with less capable models or in highly complex scenarios. For instance, Gemini 2.5 Flash scored 46.4 percent on its own but nearly 69 percent when assisted by DxO. Despite this boost, the gap between weaker and stronger models remained larger than the improvement provided by the orchestration. Microsoft researchers explained that accuracy gains naturally diminish as models approach theoretical maximums. The system’s primary value, therefore, lies in optimizing the balance between accuracy and cost, particularly when using mid-tier models for difficult cases.

Microsoft is currently collaborating with the Mayo Clinic to train a specific foundation model for healthcare, with a release planned for next year. Suleyman believes this dedicated model will enable substantial leaps in medical AI. He anticipates that by 2027, the industry will see a coding-style breakthrough in medical records, allowing for the streaming of highly accurate predictions about how a patient’s condition might unfold. This foresight could provide clinicians with valuable insights that are difficult to obtain through traditional methods alone.

The company is moving from simulation to real-world application through a new partnership with Beth Israel Deaconess Medical Center in Boston. This study will evaluate the clinical reasoning capabilities of DxO across various hospital settings, including emergency departments and urgent care environments. Patients will receive standard treatment while consenting to parallel evaluation by the AI. Microsoft aims to compare the AI’s recommendations against those of human doctors to gather compelling evidence on the system’s deployability at scale. Dominic King, vice president of Health at Microsoft AI, emphasized that the technology is ready for these rigorous real-world evaluations.

This initiative places Microsoft in direct competition with peers like Google and OpenAI, who have also claimed their AI can match or outperform doctors. Google recently published a study in The Lancet evaluating its AMIE diagnostic AI in real-world urgent care visits. Microsoft’s strategy focuses on becoming the leading provider of state-of-the-art medical clinical decision support products in the coming years. By integrating these systems with existing dictation and radiology tools, the company aims to offer a comprehensive suite that enhances clinical outcomes and efficiency. The goal is to accelerate the introduction of these capabilities into the broader healthcare ecosystem.