Google AMIE: AI Doctor Learns to ‘See’ Medical Images

From Text to Multimodal Understanding

While AMIE has already shown promise in text-based medical chats, Google recognizes that real-world medicine relies heavily on visual data—skin conditions, lab reports, and machine readings. To bridge this gap, Google integrated its Gemini 2.0 Flash model with a “state-aware reasoning framework,” allowing AMIE to adapt its conversations dynamically, request relevant visual inputs, and refine diagnoses based on multimodal evidence.

The AI mimics a clinician’s workflow: gathering patient history, analyzing visual data, and offering management suggestions. To train and test this capability, Google created a simulation lab using realistic patient cases, medical images, and data from sources like the PTB-XL ECG database and SCIN dermatology set.

Testing AMIE in a Simulated Clinic

Google evaluated AMIE using a setup akin to the Objective Structured Clinical Examination (OSCE), a standard method for assessing medical students. In a study involving 105 scenarios, patient actors interacted with either AMIE or human primary care physicians (PCPs) via a chat interface that supported image uploads. Specialist doctors and patient actors then reviewed the interactions, assessing diagnostic accuracy, communication skills, and empathy.

Promising Results

AMIE often outperformed human PCPs in interpreting multimodal data and generating accurate differential diagnoses. Specialists praised its image analysis, diagnostic thoroughness, and management plans. Surprisingly, patient actors frequently rated AMIE as more empathetic and trustworthy than human doctors in text-based interactions. Importantly, the AI’s error rate in interpreting images was comparable to that of human physicians.

Early tests with the newer Gemini 2.5 Flash model showed further improvements in diagnostic accuracy and management suggestions, though Google emphasizes the need for expert review to validate these findings.

Challenges and Next Steps

Despite these promising results, Google acknowledges the limitations of simulated scenarios, which lack the complexity of real-world clinical settings. The company is partnering with Beth Israel Deaconess Medical Center to test AMIE in actual practice with patient consent. Future developments aim to incorporate real-time video and audio, aligning with telehealth trends.

While AMIE’s ability to interpret visual medical evidence marks a significant step forward, its journey from research to reliable clinical tool will require rigorous testing and careful implementation.