MedicalBenchmark
EleutherAI: Llemma 7b provider

Llemma 7b

289

#289 of 290 models — MIR 2026

Net score

9.33 pts

Accuracy

8.5%

Correct / Incorrect

17 / 23

Total Cost

$0.68

Overall Performance

(vs. average)
Accuracy

8.5%

avg: 81.6%

Net score

9.33 pts

avg: 154.00 pts

Correct

17

avg: 163

Incorrect

23

avg: 28

Total Cost

$0.68

avg: $3.33

Average response time

72.4s

avg: 16.2s

Output Tokens

484K

avg: 430K

Reasoning Tokens

0

avg: 310K

Average confidence

19.4%

avg: 95.1%

Subject Breakdown

Allergology
Correct
0
Incorrect
0
Unanswered
1
Accuracy
0.0%
Average
96.9%
Anesthesiology and Resuscitation
Correct
0
Incorrect
1
Unanswered
6
Accuracy
0.0%
Average
69.6%
Cardiology
Correct
1
Incorrect
4
Unanswered
20
Accuracy
4.0%
Average
77.3%
Dermatology
Correct
1
Incorrect
0
Unanswered
10
Accuracy
9.1%
Average
72.3%
Endocrinology and Nutrition
Correct
1
Incorrect
0
Unanswered
14
Accuracy
6.7%
Average
84.0%
ENT
Correct
0
Incorrect
3
Unanswered
5
Accuracy
0.0%
Average
84.7%
Epidemiology
Correct
1
Incorrect
1
Unanswered
5
Accuracy
14.3%
Average
80.2%
Gastroenterology
Correct
2
Incorrect
6
Unanswered
22
Accuracy
6.7%
Average
79.3%
Genetics
Correct
2
Incorrect
0
Unanswered
9
Accuracy
18.2%
Average
78.7%
Geriatrics
Correct
2
Incorrect
2
Unanswered
9
Accuracy
15.4%
Average
83.0%
Gynecology and Obstetrics
Correct
1
Incorrect
1
Unanswered
10
Accuracy
8.3%
Average
84.3%
Health Planning and Management
Correct
2
Incorrect
0
Unanswered
8
Accuracy
20.0%
Average
78.4%
Hematology
Correct
0
Incorrect
0
Unanswered
9
Accuracy
0.0%
Average
76.6%
Immunology
Correct
0
Incorrect
1
Unanswered
5
Accuracy
0.0%
Average
91.4%
Infectious Diseases
Correct
2
Incorrect
2
Unanswered
10
Accuracy
14.3%
Average
77.9%
Legal Medicine and Bioethics
Correct
2
Incorrect
0
Unanswered
9
Accuracy
18.2%
Average
82.9%
Medical Oncology
Correct
3
Incorrect
4
Unanswered
16
Accuracy
13.0%
Average
83.0%
Nephrology
Correct
0
Incorrect
1
Unanswered
9
Accuracy
0.0%
Average
85.1%
Neurology
Correct
0
Incorrect
3
Unanswered
10
Accuracy
0.0%
Average
88.6%
Ophthalmology
Correct
0
Incorrect
0
Unanswered
5
Accuracy
0.0%
Average
83.7%
Palliative Care
Correct
1
Incorrect
1
Unanswered
4
Accuracy
16.7%
Average
80.2%
Pediatrics
Correct
2
Incorrect
3
Unanswered
17
Accuracy
9.1%
Average
87.6%
Pharmacology
Correct
4
Incorrect
0
Unanswered
7
Accuracy
36.4%
Average
78.6%
Psychiatry
Correct
0
Incorrect
0
Unanswered
8
Accuracy
0.0%
Average
87.9%
Pulmonology
Correct
1
Incorrect
2
Unanswered
13
Accuracy
6.3%
Average
82.8%
Radiology-Emergency
Correct
1
Incorrect
2
Unanswered
10
Accuracy
7.7%
Average
67.7%
Rheumatology
Correct
2
Incorrect
0
Unanswered
9
Accuracy
18.2%
Average
88.4%
Statistics
Correct
1
Incorrect
1
Unanswered
0
Accuracy
50.0%
Average
83.8%
Traumatology
Correct
1
Incorrect
0
Unanswered
10
Accuracy
9.1%
Average
65.2%
Urology
Correct
1
Incorrect
1
Unanswered
6
Accuracy
12.5%
Average
82.5%

Question Type Breakdown

Anatomy
Correct
0
Incorrect
0
Unanswered
3
Accuracy
0.0%
Average
82.6%
Biostatistics
Correct
1
Incorrect
1
Unanswered
0
Accuracy
50.0%
Average
83.8%
Diagnosis
Correct
5
Incorrect
11
Unanswered
65
Accuracy
6.2%
Average
82.2%
Epidemiology
Correct
1
Incorrect
1
Unanswered
7
Accuracy
11.1%
Average
88.7%
Ethics
Correct
1
Incorrect
0
Unanswered
5
Accuracy
16.7%
Average
92.0%
Interpretation
Correct
3
Incorrect
4
Unanswered
30
Accuracy
8.1%
Average
72.0%
Legal
Correct
2
Incorrect
0
Unanswered
7
Accuracy
22.2%
Average
82.4%
Pathophysiology
Correct
1
Incorrect
1
Unanswered
24
Accuracy
3.8%
Average
84.3%
Pharmacology
Correct
3
Incorrect
2
Unanswered
10
Accuracy
20.0%
Average
82.3%
Prevention
Correct
2
Incorrect
1
Unanswered
13
Accuracy
12.5%
Average
80.6%
Prognosis
Correct
0
Incorrect
2
Unanswered
3
Accuracy
0.0%
Average
93.2%
Risk
Correct
2
Incorrect
0
Unanswered
13
Accuracy
13.3%
Average
84.3%
Tests
Correct
4
Incorrect
7
Unanswered
22
Accuracy
12.1%
Average
80.3%
Treatment
Correct
4
Incorrect
10
Unanswered
58
Accuracy
5.6%
Average
80.1%
#AnswerCorrectStatus
1—A
2BB
3CD
4—D
5—B
6—C
7—B
8—D
9—D
10AA
11CB
12—A
13——Annulled
14CC
15—A
16—C
17—C
18—D
19—B
20—B
21—B
22—C
23—B
24—A
25AC
26—A
27—B
28—D
29AA
30—D
31—C
32—C
33DB
34—B
35DC
36—D
37—D
38AA
39—C
40—A
41—D
42—A
43—B
44CB
45AB
46—B
47—D
48—C
49—C
50——Annulled
51—A
52—C
53AC
54—C
55AA
56—A
57—A
58—A
59—B
60—B
61—C
62—B
63—B
64——Annulled
65—C
66CC
67—C
68—A
69—B
70—B
71BB
72—B
73—B
74AA
75—C
76—C
77—A
78—D
79—A
80—D
81AC
82—D
83—B
84—A
85—D
86AA
87—B
88DB
89—B
90—C
91—A
92—C
93—D
94—C
95—C
96—B
97AB
98—A
99—B
100—C
101BC
102—A
103BC
104CC
105—C
106—B
107—C
108—A
109AC
110—C
111—B
112—C
113CC
114—D
115—B
116—D
117—C
118—B
119—A
120—A
121—C
122—B
123AB
124—B
125—C
126—B
127—C
128CB
129—C
130—C
131—B
132—B
133—C
134—B
135—C
136BB
137—D
138—B
139——Annulled
140—D
141—A
142——Annulled
143—A
144—D
145—C
146—C
147—B
148—C
149AB
150—B
151—B
152—C
153CC
154—D
155—B
156—A
157BD
158—B
159—C
160—A
161A—Annulled
162AA
163DC
164DC
165—A
166—A
167AD
168—C
169—B
170—B
171—A
172—B
173—D
174—C
175—B
176—A
177—D
178—D
179BB
180—C
181—B
182—C
183—D
184—D
185—B
186—D
187—D
188—C
189—C
190—B
191—D
192—D
193—C
194—C
195—D
196AA
197AC
198—A
199—B
200—D
201—C
202—A
203—A
204—C
205—B
206BC
207—A
208D—Annulled
209—B
210—C