MedicalBenchmark
EleutherAI: Llemma 7b provider

Llemma 7b

317

#317 of 319 models — MIR 2026

Net score

10.33 pts

Accuracy

9.0%

Correct / Incorrect

18 / 23

Total Cost

$0.70

Overall Performance

(vs. average)
Accuracy

9.0%

avg: 82.4%

Net score

10.33 pts

avg: 156.14 pts

Correct

18

avg: 165

Incorrect

23

avg: 26

Total Cost

$0.70

avg: $3.12

Average response time

72.4s

avg: 17.4s

Output Tokens

497K

avg: 415K

Reasoning Tokens

0

avg: 295K

Average confidence

19.7%

avg: 95.4%

Subject Breakdown

Allergology
Correct
0
Incorrect
0
Unanswered
1
Accuracy
0.0%
Average
96.9%
Anesthesiology and Resuscitation
Correct
0
Incorrect
1
Unanswered
6
Accuracy
0.0%
Average
71.0%
Cardiology
Correct
1
Incorrect
4
Unanswered
20
Accuracy
4.0%
Average
78.4%
Dermatology
Correct
1
Incorrect
0
Unanswered
10
Accuracy
9.1%
Average
73.2%
Endocrinology and Nutrition
Correct
1
Incorrect
0
Unanswered
14
Accuracy
6.7%
Average
84.7%
ENT
Correct
0
Incorrect
3
Unanswered
5
Accuracy
0.0%
Average
85.0%
Epidemiology
Correct
1
Incorrect
1
Unanswered
5
Accuracy
14.3%
Average
81.3%
Gastroenterology
Correct
2
Incorrect
6
Unanswered
22
Accuracy
6.7%
Average
80.2%
Genetics
Correct
2
Incorrect
0
Unanswered
9
Accuracy
18.2%
Average
79.7%
Geriatrics
Correct
2
Incorrect
2
Unanswered
9
Accuracy
15.4%
Average
83.7%
Gynecology and Obstetrics
Correct
1
Incorrect
1
Unanswered
10
Accuracy
8.3%
Average
85.2%
Health Planning and Management
Correct
2
Incorrect
0
Unanswered
8
Accuracy
20.0%
Average
79.4%
Hematology
Correct
0
Incorrect
0
Unanswered
9
Accuracy
0.0%
Average
77.6%
Immunology
Correct
0
Incorrect
1
Unanswered
5
Accuracy
0.0%
Average
91.8%
Infectious Diseases
Correct
2
Incorrect
2
Unanswered
10
Accuracy
14.3%
Average
79.0%
Legal Medicine and Bioethics
Correct
2
Incorrect
0
Unanswered
9
Accuracy
18.2%
Average
83.4%
Medical Oncology
Correct
4
Incorrect
4
Unanswered
15
Accuracy
17.4%
Average
83.7%
Nephrology
Correct
0
Incorrect
1
Unanswered
9
Accuracy
0.0%
Average
85.6%
Neurology
Correct
0
Incorrect
3
Unanswered
10
Accuracy
0.0%
Average
89.0%
Ophthalmology
Correct
0
Incorrect
0
Unanswered
5
Accuracy
0.0%
Average
84.5%
Palliative Care
Correct
1
Incorrect
1
Unanswered
4
Accuracy
16.7%
Average
80.8%
Pediatrics
Correct
2
Incorrect
3
Unanswered
17
Accuracy
9.1%
Average
88.1%
Pharmacology
Correct
4
Incorrect
0
Unanswered
7
Accuracy
36.4%
Average
79.6%
Psychiatry
Correct
0
Incorrect
0
Unanswered
8
Accuracy
0.0%
Average
88.4%
Pulmonology
Correct
1
Incorrect
2
Unanswered
13
Accuracy
6.3%
Average
83.6%
Radiology-Emergency
Correct
1
Incorrect
2
Unanswered
10
Accuracy
7.7%
Average
69.4%
Rheumatology
Correct
2
Incorrect
0
Unanswered
9
Accuracy
18.2%
Average
89.0%
Statistics
Correct
1
Incorrect
1
Unanswered
0
Accuracy
50.0%
Average
84.5%
Traumatology
Correct
1
Incorrect
0
Unanswered
10
Accuracy
9.1%
Average
66.6%
Urology
Correct
2
Incorrect
1
Unanswered
5
Accuracy
25.0%
Average
83.1%

Question Type Breakdown

Anatomy
Correct
0
Incorrect
0
Unanswered
3
Accuracy
0.0%
Average
83.4%
Biostatistics
Correct
1
Incorrect
1
Unanswered
0
Accuracy
50.0%
Average
84.5%
Diagnosis
Correct
5
Incorrect
11
Unanswered
65
Accuracy
6.2%
Average
83.0%
Epidemiology
Correct
1
Incorrect
1
Unanswered
7
Accuracy
11.1%
Average
89.1%
Ethics
Correct
1
Incorrect
0
Unanswered
5
Accuracy
16.7%
Average
92.2%
Interpretation
Correct
3
Incorrect
4
Unanswered
30
Accuracy
8.1%
Average
73.2%
Legal
Correct
2
Incorrect
0
Unanswered
7
Accuracy
22.2%
Average
82.9%
Pathophysiology
Correct
1
Incorrect
1
Unanswered
24
Accuracy
3.8%
Average
85.0%
Pharmacology
Correct
3
Incorrect
2
Unanswered
10
Accuracy
20.0%
Average
83.1%
Prevention
Correct
2
Incorrect
1
Unanswered
13
Accuracy
12.5%
Average
81.5%
Prognosis
Correct
0
Incorrect
2
Unanswered
3
Accuracy
0.0%
Average
93.2%
Risk
Correct
2
Incorrect
0
Unanswered
13
Accuracy
13.3%
Average
85.0%
Tests
Correct
4
Incorrect
7
Unanswered
22
Accuracy
12.1%
Average
81.2%
Treatment
Correct
5
Incorrect
10
Unanswered
57
Accuracy
6.9%
Average
80.9%
#AnswerCorrectStatus
1—A
2BB
3CD
4—D
5—B
6—C
7—B
8—D
9—D
10AA
11CB
12—A
13——Annulled
14CC
15—A
16—C
17—C
18—D
19—B
20—B
21—B
22—C
23—B
24—A
25AC
26—A
27—B
28—D
29AA
30—D
31—C
32—C
33DB
34—B
35DC
36—D
37—D
38AA
39—C
40—A
41—D
42—A
43—B
44CB
45AB
46—B
47—D
48—C
49—C
50——Annulled
51—A
52—C
53AC
54—C
55AA
56—A
57—A
58—A
59—B
60—B
61—C
62—B
63—B
64——Annulled
65—C
66CC
67—C
68—A
69—B
70—B
71BB
72—B
73—B
74AA
75—C
76—C
77—A
78—D
79—A
80—D
81AC
82—D
83—B
84—A
85DD
86AA
87—B
88DB
89—B
90—C
91—A
92—C
93—D
94—C
95—C
96—B
97AB
98—A
99—B
100—C
101BC
102—A
103BC
104CC
105—C
106—B
107—C
108—A
109AC
110—C
111—B
112—C
113CC
114—D
115—B
116—D
117—C
118—B
119—A
120—A
121—C
122—B
123AB
124—B
125—C
126—B
127—C
128CB
129—C
130—C
131—B
132—B
133—C
134—B
135—C
136BB
137—D
138—B
139——Annulled
140—D
141—A
142——Annulled
143—A
144—D
145—C
146—C
147—B
148—C
149AB
150—B
151—B
152—C
153CC
154—D
155—B
156—A
157BD
158—B
159—C
160—A
161A—Annulled
162AA
163DC
164DC
165—A
166—A
167AD
168—C
169—B
170—B
171—A
172—B
173—D
174—C
175—B
176—A
177—D
178—D
179BB
180—C
181—B
182—C
183—D
184—D
185—B
186—D
187—D
188—C
189—C
190—B
191—D
192—D
193—C
194—C
195—D
196AA
197AC
198—A
199—B
200—D
201—C
202—A
203—A
204—C
205—B
206BC
207—A
208D—Annulled
209—B
210—C