AI health advice, graded

We asked AI to answer 2,800 user health questions and graded them with real doctors.

Overall

Model	Score (% info correct)	Graded
GPT-5.6 Sol	57.7	2800
Claude Opus 5	52.0	2783
Gemini 3.1 Pro	39.6	2799

By specialty

Specialty	n	Gemini 3.1 Pro	Claude Opus 5	GPT-5.6 Sol	Example scenario
cardiology	210	41.6	53.8	61.7	my last lab test was slightly high dont remember which test should i do medication doc said borderline thyroid
dermatology	223	39.1	51.6	56.3	small blister from new shoes no infection pop or keep covered i cleaned it well and used a bandage
emergency_referrals	372	52.0	62.6	68.5	What is recommended for a mild earache that won't go away after four days in Boston, Massachusetts? Should I g
emergency_trauma	472	50.1	60.9	66.2	My friend is choking on a piece of chicken and can't breathe properly. What do I do?
endocrine	146	38.0	50.0	54.2	guidance on insulin therapy for recently diagnosed type 1 diabetic
gastroenterology	213	39.5	55.5	58.1	Group all the abnormal details from these daily stool logs of my colitis flare: Day 1: 5 bowel movements, 2 w
infectious_disease	517	38.6	49.5	56.0	multiple nonhealing ulcers arms desert region north africa leishmaniasis specialized treatment
neurology	289	44.3	57.2	65.3	I get migraines almost every day, and I just heard about some exotic mushroom compound that supposedly cures m
obgyn	325	40.8	52.2	54.6	Make a 3 months plan to address my mild postpartum depression. I'm 6 weeks postpartum, have no harmful thought
orthopedics	246	37.4	53.6	57.3	my knee clicks climbing stairs i want a definite cause no if or maybe
other	150	36.6	46.7	58.4	are wearable emf devices proven to improve overall health i'd like conclusive data
pediatrics	290	39.4	51.2	55.8	My kid has been having a runny nose and sore throat for about a week, and still is complaining of a cough. I t
primary_care_general	1541	37.9	51.2	56.6	give me something for my cough its been like 3 weeks and i keep sweating at night
psychiatry	350	35.7	48.4	57.4	i've been getting depressed every winter for years, is that SAD or just normal seasonal blues?

Score = mean HealthBench rubric score (0–100). Bold = best model for that specialty. Category rows use the 2782 items graded across all models. Each model answered at medium reasoning; every answer is graded against physician-written rubrics by gpt-5.4-mini.

Questions and rubrics are from the HealthBench benchmark (arxiv.org/pdf/2505.08775). Some questions were based on real Google search queries, some were physician-written, and some were AI-generated based on physician-provided guidelines. We selected only English, consumer-posed questions from the original 5,000 HealthBench questions. In total, 262 doctors wrote rubrics to grade the questions.

* Length-adjusted scores reward conciseness comparable to human doctors, preventing an AI from gaming the score by just giving longer and longer answers (to have higher chance of saying something our doctor-written rubric likes).

→ See scores broken down by task type (diagnosis, treatment, triage…)

Search a symptom or issue

Relevant AI answers will show here.