MedicalBenchmark
EleutherAI: Llemma 7b provider

Llemma 7b

316

#316 of 319 models — MIR 2025

Net score

3.33 pts

Accuracy

8.5%

Correct / Incorrect

17 / 41

Total Cost

$0.64

Overall Performance

(vs. average)
Accuracy

8.5%

avg: 77.9%

Net score

3.33 pts

avg: 143.96 pts

Correct

17

avg: 156

Incorrect

41

avg: 35

Total Cost

$0.64

avg: $3.36

Average response time

65.0s

avg: 19.0s

Output Tokens

446K

avg: 430K

Reasoning Tokens

0

avg: 306K

Average confidence

29.0%

avg: 95.2%

Subject Breakdown

Allergology
Correct
0
Incorrect
1
Unanswered
3
Accuracy
0.0%
Average
87.9%
Anesthesiology and Resuscitation
Correct
2
Incorrect
1
Unanswered
3
Accuracy
33.3%
Average
82.3%
Cardiology
Correct
0
Incorrect
3
Unanswered
19
Accuracy
0.0%
Average
78.6%
Dermatology
Correct
2
Incorrect
4
Unanswered
6
Accuracy
16.7%
Average
69.4%
Endocrinology and Nutrition
Correct
1
Incorrect
3
Unanswered
12
Accuracy
6.3%
Average
83.5%
ENT
Correct
1
Incorrect
1
Unanswered
6
Accuracy
12.5%
Average
74.8%
Epidemiology
Correct
1
Incorrect
1
Unanswered
5
Accuracy
14.3%
Average
69.1%
Gastroenterology
Correct
1
Incorrect
4
Unanswered
16
Accuracy
4.8%
Average
74.1%
Genetics
Correct
0
Incorrect
1
Unanswered
5
Accuracy
0.0%
Average
69.5%
Geriatrics
Correct
0
Incorrect
3
Unanswered
8
Accuracy
0.0%
Average
77.5%
Gynecology and Obstetrics
Correct
1
Incorrect
4
Unanswered
14
Accuracy
5.3%
Average
86.7%
Health Planning and Management
Correct
0
Incorrect
0
Unanswered
2
Accuracy
0.0%
Average
82.6%
Hematology
Correct
2
Incorrect
3
Unanswered
6
Accuracy
18.2%
Average
82.7%
Immunology
Correct
2
Incorrect
1
Unanswered
6
Accuracy
22.2%
Average
83.3%
Infectious Diseases
Correct
5
Incorrect
4
Unanswered
18
Accuracy
18.5%
Average
74.9%
Legal Medicine and Bioethics
Correct
2
Incorrect
0
Unanswered
3
Accuracy
40.0%
Average
68.4%
Medical Oncology
Correct
4
Incorrect
6
Unanswered
15
Accuracy
16.0%
Average
87.2%
Nephrology
Correct
0
Incorrect
3
Unanswered
11
Accuracy
0.0%
Average
84.8%
Neurology
Correct
0
Incorrect
4
Unanswered
16
Accuracy
0.0%
Average
77.3%
Ophthalmology
Correct
0
Incorrect
3
Unanswered
2
Accuracy
0.0%
Average
74.2%
Palliative Care
Correct
1
Incorrect
0
Unanswered
3
Accuracy
25.0%
Average
78.6%
Pediatrics
Correct
2
Incorrect
3
Unanswered
21
Accuracy
7.7%
Average
71.9%
Pharmacology
Correct
2
Incorrect
5
Unanswered
10
Accuracy
11.8%
Average
74.1%
Psychiatry
Correct
1
Incorrect
1
Unanswered
6
Accuracy
12.5%
Average
83.0%
Pulmonology
Correct
0
Incorrect
4
Unanswered
10
Accuracy
0.0%
Average
80.4%
Radiology-Emergency
Correct
0
Incorrect
3
Unanswered
11
Accuracy
0.0%
Average
69.4%
Rheumatology
Correct
1
Incorrect
5
Unanswered
9
Accuracy
6.7%
Average
76.6%
Statistics
Correct
0
Incorrect
1
Unanswered
2
Accuracy
0.0%
Average
76.6%
Traumatology
Correct
2
Incorrect
4
Unanswered
12
Accuracy
11.1%
Average
79.3%
Urology
Correct
0
Incorrect
3
Unanswered
4
Accuracy
0.0%
Average
80.7%

Question Type Breakdown

Anatomy
Correct
0
Incorrect
1
Unanswered
6
Accuracy
0.0%
Average
78.6%
Biostatistics
Correct
0
Incorrect
1
Unanswered
3
Accuracy
0.0%
Average
79.8%
Diagnosis
Correct
7
Incorrect
18
Unanswered
63
Accuracy
8.0%
Average
79.9%
Epidemiology
Correct
1
Incorrect
0
Unanswered
4
Accuracy
20.0%
Average
76.7%
Ethics
Correct
0
Incorrect
0
Unanswered
3
Accuracy
0.0%
Average
74.1%
Interpretation
Correct
2
Incorrect
7
Unanswered
33
Accuracy
4.8%
Average
70.7%
Legal
Correct
2
Incorrect
0
Unanswered
2
Accuracy
50.0%
Average
64.6%
Pathophysiology
Correct
2
Incorrect
6
Unanswered
19
Accuracy
7.4%
Average
76.1%
Pharmacology
Correct
1
Incorrect
5
Unanswered
7
Accuracy
7.7%
Average
83.3%
Prevention
Correct
2
Incorrect
2
Unanswered
8
Accuracy
16.7%
Average
75.6%
Prognosis
Correct
1
Incorrect
1
Unanswered
5
Accuracy
14.3%
Average
80.8%
Risk
Correct
0
Incorrect
1
Unanswered
4
Accuracy
0.0%
Average
85.2%
Tests
Correct
1
Incorrect
5
Unanswered
21
Accuracy
3.7%
Average
77.9%
Treatment
Correct
6
Incorrect
19
Unanswered
56
Accuracy
7.4%
Average
77.3%
#AnswerCorrectStatus
1DB
2DA
3—C
4—B
5—A
6—C
7—C
8AA
9—A
10AD
11—D
12—D
13—B
14—D
15——Annulled
16—B
17—B
18—A
19DC
20—A
21—B
22—D
23—C
24—D
25—C
26——Annulled
27—C
28A—Annulled
29—D
30DB
31—D
32—A
33—D
34—D
35BB
36—D
37—C
38—C
39—D
40—A
41—D
42—C
43—B
44—D
45BD
46BA
47—A
48AA
49—D
50—B
51—C
52AB
53CD
54—B
55AA
56——Annulled
57—C
58—B
59AD
60AA
61DA
62—D
63—B
64—D
65—A
66—A
67—B
68—B
69—B
70—A
71—D
72AA
73—D
74—C
75—A
76—B
77AB
78—B
79—C
80AC
81—C
82—D
83—B
84—D
85—C
86—C
87—A
88—D
89—B
90—A
91—B
92AC
93DB
94AC
95AA
96—C
97—D
98—C
99—A
100—C
101—B
102—D
103CA
104—C
105—A
106—C
107—B
108—D
109—B
110DC
111DA
112DC
113—B
114—D
115—D
116—C
117—A
118—D
119—C
120—B
121—D
122—C
123—C
124—C
125—D
126AD
127—B
128DD
129—A
130AD
131—D
132—A
133—B
134—C
135DB
136—C
137—A
138—D
139—D
140AB
141AA
142AA
143AB
144—B
145BD
146—C
147AB
148—A
149—A
150—D
151—A
152DA
153DB
154—B
155—B
156CC
157—A
158—C
159AC
160—A
161AA
162C—Annulled
163—D
164BC
165DA
166—B
167—C
168—D
169—B
170—B
171BC
172AA
173—A
174—B
175—B
176—C
177—C
178—A
179—D
180AA
181BB
182—C
183AB
184—B
185BB
186——Annulled
187DC
188AD
189—D
190—A
191—B
192BA
193—C
194AA
195—A
196—A
197AB
198BC
199BD
200—C
201—B
202CA
203—D
204—C
205—B
206—D
207DA
208AC
209—C
210AB