MedGemma 1.5: What Google’s Open Medical Model Can and Cannot Do

MedGemma 1.5 is the strongest open multimodal medical model for its size, and Google is explicit that it is not a clinical component: its outputs “are not intended to directly inform clinical diagnosis, patient management decisions, treatment recommendations, or any other direct clinical practice applications.” Its real value is not benchmark supremacy — a 4B general-purpose model beats it on several medical exams. Therefore, its value comes from running inside your own infrastructure, fine-tuning on your own data, and remaining frozen while you validate it.

---

The 91% problem, and what it actually teaches

A figure of “91%” attaches itself to MedGemma 1.5 across model round-ups and consultancy pages. Most of the time it appears next to the word MedQA, or next to no word at all.

Here is what the model card actually says. Under Document understanding evaluations — PDF-to-JSON Lab Test Data Conversion:

Task / Dataset	Metric	Gemma 3 4B	MedGemma 1 4B	MedGemma 1.5 4B	MedGemma 1 27B
EHR Dataset 2 (raw PDF to JSON)	Macro F1	84.0	78.0	91.0	76.0
EHR Dataset 2	Micro F1	81.0	75.0	88.0	70.0
EHR Dataset 3 (raw PDF to JSON)	Macro F1	61.0	50.0	71.0	66.0
EHR Dataset 3	Micro F1	61.0	51.0	70.0	69.0
Mendeley Clinical Lab Test Reports (PNG to JSON)	Macro F1	83.0	85.0	85.0	69.0
Mendeley	Micro F1	78.0	81.0	83.0	68.0
EHR Dataset 4	Macro F1	41.0	25.0	64.0	—
EHR Dataset 4	Micro F1	41.0	33.0	67.0	—

And under Text evaluations:

Dataset	Gemma 3 4B	MedGemma 1 4B	MedGemma 1.5 4B	MedGemma 1 27B
MedQA (4-option)	50.7	64.4	69.1	85.3

Therefore, the 91.0 figure represents Macro F1 for structured extraction from lab report PDFs on an internal Google dataset. 69.1 is accuracy on a medical licensing exam benchmark. Both are MedGemma 1.5 4B. Neither is wrong. They measure entirely different capabilities.

This is the actual failure mode in model coverage — metrics travelling without their tasks. A number detached from what it measured is not a shortcut, it is a different claim. We got this wrong in an earlier version of this article, assuming the 91% figure was a corrupted 87.7%. However, it was not, and the correction is more useful than our original point.

The practical rule: before quoting any model figure in a business case, write down the dataset and the metric. If you cannot, you do not have a number, you have a rumour. And in this case, note that EHR Datasets 2, 3 and 4 are all internal Google datasets, so the 91.0 is not independently reproducible.

---

What MedGemma actually is

Google Research and Google DeepMind trained MedGemma, a collection of Gemma 3 variants, for medical text and image comprehension and released it through the Health AI Developer Foundations programme.

MedGemma 1 offers three variants: a 4B multimodal model, a 27B text-only model, and a 27B multimodal model. The 27B text-only variant was trained exclusively on medical text and, per the model card, “has slightly higher performance on some text benchmarks than MedGemma 27B multimodal.” Both 27B variants are available only instruction-tuned. FHIR-based electronic health record data was included in the training of the 27B multimodal model only.

MedGemma 1.5, released 13 January 2026, is a 4B multimodal instruction-tuned model only.

The architecture is Gemma 3’s decoder-only transformer with grouped-query attention, images normalised to 896×896 and encoded to 256 tokens each, 128k input context, 8,192 token output. Training was done in JAX.

The multimodal variants use a SigLIP image encoder pre-trained on de-identified chest X-rays, dermatology, ophthalmology and histopathology images. Google also publishes the encoder separately as MedSigLIP, a 400M-parameter model. In addition, for image tasks that do not require text generation, such as classification or retrieval, Google recommends using MedSigLIP instead of MedGemma.

Public training datasets include MIMIC-CXR, Chest ImaGenome, SLAKE, PAD-UFES-20, SCIN, TCGA, CAMELYON, PMC-OA and Mendeley Digital Knee X-Ray, alongside licensed proprietary datasets covering CT, MRI, dermatology (Colombia, Australia, Japan), pathology (Europe, US) and lab reports.

---

The full benchmark picture

Text and reasoning

Dataset	Gemma 3 4B	MedGemma 1 4B	MedGemma 1.5 4B	MedGemma 1 27B
MedQA (4-op)	50.7	64.4	69.1	85.3
MedMCQA	45.4	55.7	59.8	70.2
PubMedQA	68.4	73.4	68.2 ↓	77.2
MMLU Med	67.2	70.0	69.6 ↓	86.2
MedXpertQA (text only)	11.6	14.2	16.4	23.7
AfriMed-QA	48.0	52.0	56.0	72.0

The 27B text model is a different proposition again: Google reports 87.7% on MedQA with test-time scaling, which it describes as within three points of DeepSeek R1 at approximately one tenth the inference cost.

Imaging

Task	Gemma 3 4B	MedGemma 1 4B	MedGemma 1.5 4B	MedGemma 1 27B
CT Dataset 1, macro accuracy	54.5	58.2	61.1	57.8
CT-RATE, Macro F1	—	23.5	27.0	—
MRI Dataset 1, macro accuracy	51.1	51.3	64.7 ↑	57.4
MIMIC CXR, Macro F1 (top 5)	81.2	88.9	89.5	90.0
CheXpert CXR, Macro F1	32.6	48.1	48.2	49.9
CXR14, Macro F1	32.0	50.1	48.4 ↓	45.3
PathMCQA	37.1	69.8	70.0	71.6
WSI-Path, ROUGE	2.3	2.2	49.4 ↑↑	4.1
US-DermMCQA	52.5	71.8	73.5	71.7
EyePACS (fundus)	14.4	64.9	76.8 ↑	75.3
MS-CXR-T, macro accuracy	59.0	61.11	65.7	50.1
SLAKE, tokenized F1	40.2	72.3	59.7 ↓	70.3
VQA-RAD, tokenized F1	33.6	49.9	48.1 ↓	46.7
Chest ImaGenome, IoU	5.7	3.1	38.0 ↑↑	16.0
MedXpertQA (text + multimodal)	16.4	18.8	20.9	26.8

Electronic health records

Dataset	Gemma 3 4B	MedGemma 1 4B	MedGemma 1.5 4B	MedGemma 1 27B
EHRQA	70.9	67.6	89.6 ↑↑	90.5
EHRNoteQA	78.0	79.4	80.4	90.7

Chest X-ray report generation

Metric	MedGemma 1 4B (tuned for CXR)	MedGemma 1.5 4B	MedGemma 1 27B
MIMIC-CXR RadGraph F1	30.3	27.2	27.0

Note the direction here. The best CXR report generation result belongs to MedGemma 1 4B fine-tuned for the task, not to MedGemma 1.5. Therefore, if report generation is your use case, do not assume the newer model is automatically the better choice.

---

Four things in those tables worth acting on

MedGemma 1.5 regressed on four benchmarks. PubMedQA, MMLU Med, CXR14 and SLAKE all fell relative to MedGemma 1 4B. This is a normal consequence of retraining a 4B model to absorb 3D imaging, whole-slide pathology and document understanding within a fixed parameter budget. However, it means upgrading is a decision, not a default. Therefore, if your product depends on literature-style reasoning, MedGemma 1 4B is the stronger model.

Google flags the SLAKE drop directly, which is unusually candid: MedGemma 1.5 4B “was less optimized for the SLAKE Q&A format compared to MedGemma 1 4B. Fine-tuning on SLAKE may improve results.”

The base model beat the medical model, twice. Gemma 3 4B — no medical training at all — scored 70.9 on EHRQA against MedGemma 1 4B’s 67.6, and 84.0 on EHR Dataset 2 against 78.0. MedGemma 1.5 corrected both, decisively. However, the lesson remains the same: benchmark the base model as a control. Domain tuning is not automatically an advantage, and on your specific task it may be a disadvantage.

The 27B models are not simply bigger. MedGemma 1 27B beats the 4B models on nearly every text benchmark by a wide margin, scoring 85.3 versus 69.1 on MedQA. However, it performs worse on several imaging and document tasks. It scores 50.1 on MS-CXR-T against 1.5’s 65.7, 16.0 IoU against 38.0, and 76.0 on EHR Dataset 2 against 91.0. Choose by task, not by parameter count.

Some absolute numbers are low, and the deltas hide it. CT-RATE Macro F1 rose from 23.5 to 27.0 — a real improvement on a benchmark where the model gets most things wrong. MedXpertQA text sits at 16.4. However, improvement percentages can look impressive in a launch post. Therefore, you should focus on the absolute score when deciding whether the model is suitable for deployment.

---

The limitations Google publishes and most coverage omits

Importantly, the model card’s limitations section provides more decision-relevant information than the benchmark scores alone.

Outputs are not for clinical use, stated plainly. “The outputs generated by MedGemma are not intended to directly inform clinical diagnosis, patient management decisions, treatment recommendations, or any other direct clinical practice applications. All outputs from MedGemma should be considered preliminary and require independent verification, clinical correlation, and further investigation.”

Multi-image comprehension is untested. “MedGemma’s multimodal capabilities have been primarily evaluated on single-image tasks. MedGemma has not been evaluated in use cases that involve comprehension of multiple images.” Prior-comparison workflows — the thing radiologists do constantly — sit outside what has been evaluated, with the partial exception of the MS-CXR-T longitudinal benchmark.

Multi-turn is untested. “MedGemma has not been evaluated or optimized for multi-turn applications.” Most clinical products are conversational.

Prompt sensitivity. “MedGemma’s training may make it more sensitive to the specific prompt used than Gemma 3.”

Agentic tasks fail at 4B. The technical report is explicit that the 4B variants “were not well suited for this task, demonstrating difficulty following system instructions for the agentic framework as provided by AgentClinic.” The 27B performed well, exceeding human physician performance on AgentClinic-MedQA. If your architecture relies on tool use and multi-step planning on modest hardware, therefore, test this capability before committing.

Data contamination. Google’s own warning: the model “might have inadvertently seen related medical information during its pre-training, potentially overestimating its true ability to generalize to novel medical concepts.” The recommendation is to validate on datasets not publicly available or otherwise available to non-institutional researchers.

Importantly, many evaluation plans overlook the cost implications of this point. As a result, published benchmark results may not reflect how the model performs on your own data. Therefore, the most reliable way to check performance is to validate the model against a held-out dataset from your own environment. However, building that dataset can become one of the longest parts of a clinical AI project.

Safety evaluation was primarily English. The model card notes as a limitation of its safety testing that evaluations “included primarily English language prompts.”

And there have been two release-note changes worth pinning. Google updated the model to MedGemma 1.5 on 13 January 2026. Ten days later, Google changed the default generation configuration to greedy decoding while keeping sampling available. Therefore, if you benchmarked the model between those dates without pinning the configuration, you should not directly compare those results with later runs.

---

What the 81% chest X-ray figure actually says

One number does more work in MedGemma coverage than any other. Google’s wording is worth reading exactly:

“In an unblinded study, 81% of MedGemma 4B–generated chest X-ray reports were judged by a US board certified radiologist to be of sufficient accuracy to result in similar patient management compared to the original radiologist reports.”

Three qualifications sit inside that sentence, and they are usually dropped.

It was unblinded. The reviewer knew which reports were AI-generated.

One radiologist. Elsewhere Google specifies a US board-certified cardiothoracic radiologist. A single-reviewer, unblinded assessment is a legitimate early signal, not a validation.

“Similar patient management” is not diagnostic equivalence. In other words, the measure asks whether a clinician would make roughly the same management decision. However, this criterion is broader than accuracy alone.

It remains a strong data point when quoted with those attached. Quoted bare, it becomes the next 91%.

---

Who is actually using it

Google names three developers in its announcement. Google describes these examples as early-stage explorations rather than deployed clinical systems. In addition, the organisations themselves report the use cases.

DeepHealth, Massachusetts, has been exploring MedSigLIP to improve chest X-ray triaging and nodule detection. Chang Gung Memorial Hospital, Taiwan, noted MedGemma works well with traditional Chinese-language medical literature and responds well to medical staff questions. For example, Tap Health in Gurgaon highlighted MedGemma’s medical grounding and its reliability on tasks that require sensitivity to clinical context, such as summarising progress notes.

On the Hugging Face side, the 1.5 4B model recorded 318,735 downloads in the past month, with 83 fine-tunes, 64 adapters and around 100 Spaces built on it. That is a genuine developer ecosystem, though downloads measure interest rather than deployment.

---

The licence, precisely

MedGemma is open-weight. It is not Apache 2.0 — the single most common misconception about it.

Use is governed by Google’s Health AI Developer Foundations terms of use, which Google describes as “an enabling license.” Access on Hugging Face is gated, so users must log in and accept the terms before accessing the model. In addition, Google applies a prohibited-use policy.

Therefore, commercial deployment should start with a licensing review rather than an assumption. Moreover, if your product becomes part of a regulated medical device, you must account for the licence in your regulatory documentation and change-control plan. Therefore, a genuinely permissive licence can reduce this burden.

In addition, MedGemma has another important advantage: Google distributes it as fixed model snapshots, so its parameters remain unchanged over time. Therefore, teams can achieve more consistent and reproducible results, which Google highlights as especially important for medical applications.

That is the strongest argument for open weights in a regulated setting, and it has nothing to do with benchmarks. A validated model that silently changes underneath you is a clinical safety problem. A frozen snapshot is not.

Our comparison of open medical AI models sets MedGemma’s licence against MeditronFO, Meditron-3, gpt-oss and the rest.

---

You are the manufacturer, not Google

Google’s disclaimer is unambiguous, and it is tempting to read it as Google’s problem with Google’s software. Regulators read it the other way.

The TGA’s February 2026 guidance is explicit: incorporating a large language model into a product with a medical purpose makes you the manufacturer, with full regulatory responsibility. The guidance makes the principle clear: therapeutic goods regulation depends on the manufacturer’s intended purpose for the product, not simply on whether the underlying technology uses AI.

The MHRA and FDA frameworks assign obligations the same way. The MHRA’s July 2026 guidance focuses on what the product is intended to do, rather than on the specific AI technology or model that powers it.

So the disclaimer describes an allocation that applies regardless. Google validated nothing for your use case. You will.

Therefore, model selection should start with a practical question: what does your intended purpose require you to validate, and does MedGemma shorten or lengthen that path? For example, MedGemma may shorten validation for imaging-adjacent products because the alternative could involve training an encoder. By contrast, for 4B text reasoning, a general model may perform better and come with a simpler licence.

---

When MedGemma is the right choice

When the task is document extraction. This is MedGemma 1.5’s strongest published result and the most under-reported. 91.0 Macro F1 on lab report PDF-to-JSON, up from 78.0, with EHR Dataset 4 rising from 25.0 to 64.0. If you are turning unstructured clinical documents into structured data, this is the model.

When the task is multimodal. X-ray, CT, MRI, histopathology, dermatology, fundus. Whole-slide pathology ROUGE went from 2.2 to 49.4 and anatomical localisation IoU from 3.1 to 38.0 — those are capability changes, not increments.

When data cannot leave your infrastructure. Usually the deciding factor. If DSPT obligations, data residency rules or internal governance keep patient data inside the estate, the comparison is not MedGemma against a frontier API — it is MedGemma against nothing.

When you need a frozen, reproducible model. The snapshot property matters more in regulated settings than any benchmark.

When you intend to fine-tune. Google’s strongest results all involve it: RadGraph F1 30.3 on CXR generation after fine-tuning, and a 50% reduction in EHR information retrieval errors.

Choose 27B text for pure text reasoning. 85.3 against 69.1 on MedQA is not marginal. The trade is hardware and the loss of 1.5’s newer modalities.

When it is not

When a cloud API is lawful for your data. On-premise is a response to a constraint, not a virtue.

When your task is one of the four that regressed. PubMedQA, MMLU Med, CXR14, SLAKE — use MedGemma 1 4B.

When you need CXR report generation. MedGemma 1 4B fine-tuned scores 30.3 RadGraph F1 against 1.5’s 27.2.

When you need agentic behaviour on small hardware. Use a general reasoning model for orchestration and call MedGemma for the medical subtask.

When the image task needs no text generation. For example, Google recommends MedSigLIP for classification and retrieval tasks. As a result, teams can use a 400M-parameter model instead of MedGemma’s 4B model when text generation is unnecessary.

---

A MedGemma evaluation checklist

Seven questions, in the order that saves the most rework.

• Write down the dataset and metric for every figure you are relying on. If you cannot, you do not have a number.
• Is the task multimodal or document-based? If neither, test a general model of the same size first — Gemma 3 4B beat MedGemma 1 4B on EHRQA.
• Can patient data lawfully leave your infrastructure? If yes, benchmark a frontier API before committing to self-hosting. If no, this is settled.
• Which version and variant? MedGemma 1 4B, 1.5 4B, 27B text and 27B multimodal are different products. Four benchmarks regressed between 1 and 1.5.
• Do you need text generation from images? If not, MedSigLIP at 400M is Google’s recommendation.
• Have you read the HAI-DEF terms? Not Apache 2.0. Budget for a licence review.
• What will you validate on? Google’s contamination warning means public benchmarks may overstate generalisation, and three of the document-understanding datasets are internal to Google and not reproducible.

---

How KastHunt uses MedGemma

KastHunt uses MedGemma 1.5 in its Diagnostic Research Engine as a self-hosted, grounded explanation layer. Therefore, the model explains retrieved evidence rather than calculating probabilities itself. That distinction determines what the model is permitted to do.

The engine first assembles encounter-scoped EHR facts alongside live, condition-specific evidence with exact citations. Deterministic code performs the prediction-model and Bayesian calculations before the case reaches MedGemma. Only then does the model receive the locked context, and its task is narrow: turning that context into supporting, contradicting and missing-information arguments a clinician can inspect.

The ordering is the design decision. MedGemma provides medical-language capability while running within the organisation’s environment. It does not calculate probabilities or introduce new evidence; it only works with information that the system has already retrieved, verified and cited.

That architecture follows directly from what Google documents about the model. A system that let MedGemma produce the probabilities would be relying on exactly the capability the model card declines to vouch for.

---

Working out whether MedGemma belongs in your build?

The right choice usually depends more on your compliance position than on the model itself. During the Discovery Workshop, we assess your data residency and governance requirements. If a cloud API and a simpler build provide the better fit, we will recommend that instead.

---

Frequently asked questions

What is MedGemma?

Google Research and Google DeepMind trained MedGemma, a collection of Gemma 3 variants, to understand medical text and images and released it through the Health AI Developer Foundations programme. MedGemma 1 comes as a 4B multimodal model, a 27B text-only model and a 27B multimodal model. Google released MedGemma 1.5 on 13 January 2026 as a 4B multimodal, instruction-tuned model.

What does MedGemma 1.5 score on MedQA?

69.1 on the four-option MedQA benchmark, per the Text evaluations table on Google’s model card. MedGemma 1 4B scores 64.4 and MedGemma 1 27B scores 85.3. Google’s abstract describes the 1.5 improvement as approximately 5% on MedQA accuracy, consistent with 64.4 to 69.1.

Where does the 91% MedGemma figure come from?

 It is a real result, but not a MedQA score. MedGemma 1.5 4B achieves 91.0 Macro F1 on EHR Dataset 2, an internal Google dataset for converting raw lab report PDFs into structured JSON. It appears under Document understanding evaluations on the model card. Quoting it as a general accuracy figure, or as a MedQA score, is incorrect.

Is MedGemma better than a frontier model for medical tasks?

However, MedGemma does not lead on text benchmarks. MedGemma 1 27B reaches 87.7% on MedQA with test-time scaling, which Google describes as within three points of DeepSeek R1 at roughly one-tenth of the inference cost. By contrast, frontier proprietary models still score higher. Therefore, MedGemma’s main advantages are self-hosting, fine-tuning, frozen parameters and cost.

Is MedGemma free to use commercially?

It is open-weight but not Apache-licensed. Google governs MedGemma use through its Health AI Developer Foundations terms, which it describes as an enabling licence. Users must also accept those terms before accessing the model on Hugging Face. A commercial deployment should include a licence review.

Is MedGemma ready for clinical use?

Google states that MedGemma outputs should not directly inform clinical diagnosis, patient management, treatment recommendations or other clinical decisions. It considers all outputs preliminary and requires users to independently verify them and interpret them in the appropriate clinical context.

Did MedGemma 1.5 improve on MedGemma 1 across the board?

No. It regressed on PubMedQA (73.4 to 68.2), MMLU Med (70.0 to 69.6), CXR14 (50.1 to 48.4) and SLAKE tokenized F1 (72.3 to 59.7). In addition, Google explicitly notes the SLAKE result, explaining that the model was less optimised for that Q&A format. Therefore, Google suggests that fine-tuning may improve its performance.

What did MedGemma 1.5 add?

Native 3D CT and MRI volume interpretation, whole-slide histopathology, anatomical localisation via bounding boxes on chest X-rays, longitudinal chest X-ray comparison, and medical document understanding for lab reports and EHR data. The largest gains were whole-slide pathology (ROUGE 2.2 to 49.4), anatomical localisation (IoU 3.1 to 38.0), EHRQA (67.6 to 89.6) and lab report extraction (78.0 to 91.0 Macro F1).

Which MedGemma variant is best for text reasoning?

MedGemma 1 27B text-only. It scores 85.3 on MedQA against MedGemma 1.5 4B’s 69.1, and 86.2 on MMLU Med against 69.6. All published 27B results use test-time scaling, which adds inference cost.

Can MedGemma be used for agentic workflows?

The 4B variants are poorly suited. The technical report states they demonstrated difficulty following system instructions in the AgentClinic framework. MedGemma 27B performed well in the same tests, exceeding human physician performance on AgentClinic-MedQA. On modest hardware, use a general reasoning model for orchestration and call MedGemma for the medical subtask.

What is MedSigLIP and when should I use it instead?

MedSigLIP is the 400M-parameter image encoder that powers MedGemma’s multimodal variants, released separately. Google recommends it over MedGemma for image tasks that do not require text generation — data-efficient classification, zero-shot classification and semantic image retrieval.

Who is liable if MedGemma produces a wrong output in our product?

The product manufacturer. The TGA’s guidance is explicit that incorporating a large language model into a product with a medical purpose makes you the manufacturer with full regulatory responsibility. The MHRA and FDA frameworks assign obligations the same way, based on stated intended purpose.

---

About the author

Sourabh Bisht is an AI Agentic Lead at KastHunt Consulting LLP, working on agentic AI, multi-agent systems, and healthcare-focused AI products. His work centers on prompt engineering, retrieval-augmented generation, Model Context Protocol, and practical AI system design, with an emphasis on building intelligent, reliable, and scalable healthcare technology.

He works on agentic AI, healthcare AI, multi-agent workflows, RAG systems, and the practical challenges of designing and deploying intelligent software products. 

Questions or corrections: info@kasthunt.com

---

---