MedicalBenchmark
EleutherAI: Llemma 7b provider

Llemma 7b

320

#320 of 320 models — MIR 2024

Net score

0.00 pts

Accuracy

3.5%

Correct / Incorrect

7 / 37

Total Cost

$0.65

Overall Performance

(vs. average)
Accuracy

3.5%

avg: 81.3%

Net score

0.00 pts

avg: 153.08 pts

Correct

7

avg: 163

Incorrect

37

avg: 29

Total Cost

$0.65

avg: $3.09

Average response time

66.2s

avg: 17.7s

Output Tokens

455K

avg: 414K

Reasoning Tokens

0

avg: 296K

Average confidence

22.9%

avg: 95.7%

Subject Breakdown

Allergology
Correct
0
Incorrect
1
Unanswered
2
Accuracy
0.0%
Average
90.8%
Anesthesiology and Resuscitation
Correct
0
Incorrect
0
Unanswered
4
Accuracy
0.0%
Average
87.7%
Cardiology
Correct
1
Incorrect
7
Unanswered
13
Accuracy
4.8%
Average
80.4%
Dermatology
Correct
1
Incorrect
2
Unanswered
11
Accuracy
7.1%
Average
81.0%
Endocrinology and Nutrition
Correct
1
Incorrect
2
Unanswered
16
Accuracy
5.3%
Average
85.1%
ENT
Correct
0
Incorrect
2
Unanswered
5
Accuracy
0.0%
Average
75.1%
Epidemiology
Correct
1
Incorrect
1
Unanswered
6
Accuracy
12.5%
Average
89.7%
Gastroenterology
Correct
1
Incorrect
4
Unanswered
17
Accuracy
4.5%
Average
71.5%
Genetics
Correct
0
Incorrect
1
Unanswered
6
Accuracy
0.0%
Average
87.1%
Geriatrics
Correct
0
Incorrect
0
Unanswered
10
Accuracy
0.0%
Average
87.7%
Gynecology and Obstetrics
Correct
0
Incorrect
2
Unanswered
12
Accuracy
0.0%
Average
82.0%
Health Planning and Management
Correct
0
Incorrect
1
Unanswered
1
Accuracy
0.0%
Average
75.1%
Hematology
Correct
0
Incorrect
2
Unanswered
11
Accuracy
0.0%
Average
82.4%
Immunology
Correct
0
Incorrect
1
Unanswered
7
Accuracy
0.0%
Average
89.7%
Infectious Diseases
Correct
0
Incorrect
6
Unanswered
17
Accuracy
0.0%
Average
82.5%
Legal Medicine and Bioethics
Correct
0
Incorrect
1
Unanswered
1
Accuracy
0.0%
Average
91.8%
Medical Oncology
Correct
1
Incorrect
5
Unanswered
15
Accuracy
4.8%
Average
80.9%
Nephrology
Correct
0
Incorrect
1
Unanswered
12
Accuracy
0.0%
Average
81.8%
Neurology
Correct
2
Incorrect
3
Unanswered
17
Accuracy
9.1%
Average
84.5%
Ophthalmology
Correct
2
Incorrect
0
Unanswered
3
Accuracy
40.0%
Average
81.3%
Palliative Care
Correct
0
Incorrect
1
Unanswered
3
Accuracy
0.0%
Average
88.6%
Pediatrics
Correct
0
Incorrect
4
Unanswered
13
Accuracy
0.0%
Average
82.9%
Pharmacology
Correct
1
Incorrect
3
Unanswered
19
Accuracy
4.3%
Average
85.8%
Psychiatry
Correct
0
Incorrect
2
Unanswered
8
Accuracy
0.0%
Average
90.0%
Pulmonology
Correct
1
Incorrect
2
Unanswered
16
Accuracy
5.3%
Average
81.6%
Radiology-Emergency
Correct
0
Incorrect
1
Unanswered
13
Accuracy
0.0%
Average
66.0%
Rheumatology
Correct
0
Incorrect
2
Unanswered
12
Accuracy
0.0%
Average
82.4%
Statistics
Correct
0
Incorrect
1
Unanswered
2
Accuracy
0.0%
Average
91.6%
Traumatology
Correct
0
Incorrect
5
Unanswered
10
Accuracy
0.0%
Average
75.4%
Urology
Correct
0
Incorrect
1
Unanswered
5
Accuracy
0.0%
Average
79.0%

Question Type Breakdown

Anatomy
Correct
0
Incorrect
1
Unanswered
5
Accuracy
0.0%
Average
81.1%
Biostatistics
Correct
0
Incorrect
1
Unanswered
4
Accuracy
0.0%
Average
91.3%
Diagnosis
Correct
3
Incorrect
9
Unanswered
61
Accuracy
4.1%
Average
80.0%
Epidemiology
Correct
1
Incorrect
2
Unanswered
9
Accuracy
8.3%
Average
82.1%
Ethics
Correct
0
Incorrect
1
Unanswered
0
Accuracy
0.0%
Average
94.0%
Interpretation
Correct
1
Incorrect
3
Unanswered
33
Accuracy
2.7%
Average
70.5%
Pathophysiology
Correct
0
Incorrect
9
Unanswered
24
Accuracy
0.0%
Average
86.1%
Pharmacology
Correct
1
Incorrect
2
Unanswered
22
Accuracy
4.0%
Average
84.7%
Prevention
Correct
1
Incorrect
2
Unanswered
9
Accuracy
8.3%
Average
90.3%
Prognosis
Correct
1
Incorrect
0
Unanswered
6
Accuracy
14.3%
Average
84.6%
Risk
Correct
0
Incorrect
1
Unanswered
12
Accuracy
0.0%
Average
84.5%
Tests
Correct
0
Incorrect
4
Unanswered
17
Accuracy
0.0%
Average
75.0%
Treatment
Correct
4
Incorrect
16
Unanswered
51
Accuracy
5.6%
Average
82.1%
#AnswerCorrectStatus
1—B
2—D
3—B
4—C
5—C
6—B
7—D
8—C
9—A
10—D
11—D
12DA
13—C
14—A
15—B
16—A
17DC
18—A
19—B
20—C
21—D
22—B
23AA
24—A
25—C
26CB
27—C
28—A
29DB
30—C
31AD
32BA
33—C
34—B
35—D
36—D
37—A
38—A
39—C
40—B
41—C
42—D
43—A
44—D
45CD
46—B
47—C
48—C
49—B
50—C
51AA
52BD
53—C
54—B
55AC
56—D
57—A
58AA
59—A
60—A
61—A
62—D
63AD
64——Annulled
65—D
66—C
67—B
68——Annulled
69—A
70—B
71—B
72BD
73DB
74—C
75—B
76—A
77—D
78—C
79—B
80—A
81—C
82—C
83—B
84DC
85—A
86—A
87—B
88—D
89CB
90—A
91AD
92—A
93BC
94—B
95DD
96—B
97—B
98CB
99—A
100CB
101—A
102—D
103—B
104CD
105AB
106—C
107—C
108—B
109—D
110—D
111—B
112—C
113A—Annulled
114—D
115AD
116CA
117—D
118—D
119—A
120—C
121—A
122DB
123—D
124—D
125—B
126—D
127AA
128—B
129BD
130—C
131—C
132—D
133CA
134AC
135CA
136—D
137—A
138—C
139AA
140—C
141—B
142—C
143—A
144—D
145—C
146—C
147AC
148—A
149—C
150—D
151—A
152—A
153—C
154—B
155BD
156—C
157—C
158—D
159—D
160—B
161—B
162—B
163AB
164DB
165—A
166—C
167—A
168—B
169—C
170—A
171BD
172AB
173—A
174—B
175AA
176—C
177—C
178—B
179—C
180——Annulled
181—B
182AD
183—C
184—A
185—C
186—D
187BA
188—C
189CD
190—D
191—B
192—B
193—C
194—C
195—C
196—B
197—A
198AB
199—D
200—A
201—B
202—D
203—B
204—D
205—D
206——Annulled
207—A
208—A
209—B
210AD