MedicalBenchmark
EleutherAI: Llemma 7b provider

Llemma 7b

291

#291 of 291 models — MIR 2024

Net score

0.00 pts

Accuracy

3.5%

Correct / Incorrect

7 / 35

Total Cost

$0.63

Overall Performance

(vs. average)
Accuracy

3.5%

avg: 80.5%

Net score

0.00 pts

avg: 150.85 pts

Correct

7

avg: 161

Incorrect

35

avg: 30

Total Cost

$0.63

avg: $3.32

Average response time

66.5s

avg: 16.4s

Output Tokens

444K

avg: 427K

Reasoning Tokens

0

avg: 310K

Average confidence

22.7%

avg: 95.4%

Subject Breakdown

Allergology
Correct
0
Incorrect
1
Unanswered
2
Accuracy
0.0%
Average
90.5%
Anesthesiology and Resuscitation
Correct
0
Incorrect
0
Unanswered
4
Accuracy
0.0%
Average
87.1%
Cardiology
Correct
1
Incorrect
6
Unanswered
14
Accuracy
4.8%
Average
79.7%
Dermatology
Correct
1
Incorrect
1
Unanswered
12
Accuracy
7.1%
Average
80.2%
Endocrinology and Nutrition
Correct
1
Incorrect
2
Unanswered
16
Accuracy
5.3%
Average
84.2%
ENT
Correct
0
Incorrect
2
Unanswered
5
Accuracy
0.0%
Average
74.4%
Epidemiology
Correct
1
Incorrect
1
Unanswered
6
Accuracy
12.5%
Average
89.3%
Gastroenterology
Correct
1
Incorrect
4
Unanswered
17
Accuracy
4.5%
Average
70.5%
Genetics
Correct
0
Incorrect
1
Unanswered
6
Accuracy
0.0%
Average
86.5%
Geriatrics
Correct
0
Incorrect
0
Unanswered
10
Accuracy
0.0%
Average
86.9%
Gynecology and Obstetrics
Correct
0
Incorrect
2
Unanswered
12
Accuracy
0.0%
Average
81.2%
Health Planning and Management
Correct
0
Incorrect
1
Unanswered
1
Accuracy
0.0%
Average
73.2%
Hematology
Correct
0
Incorrect
2
Unanswered
11
Accuracy
0.0%
Average
81.5%
Immunology
Correct
0
Incorrect
1
Unanswered
7
Accuracy
0.0%
Average
89.1%
Infectious Diseases
Correct
0
Incorrect
5
Unanswered
18
Accuracy
0.0%
Average
81.8%
Legal Medicine and Bioethics
Correct
0
Incorrect
1
Unanswered
1
Accuracy
0.0%
Average
91.7%
Medical Oncology
Correct
1
Incorrect
5
Unanswered
15
Accuracy
4.8%
Average
80.2%
Nephrology
Correct
0
Incorrect
1
Unanswered
12
Accuracy
0.0%
Average
80.8%
Neurology
Correct
2
Incorrect
3
Unanswered
17
Accuracy
9.1%
Average
83.7%
Ophthalmology
Correct
2
Incorrect
0
Unanswered
3
Accuracy
40.0%
Average
80.0%
Palliative Care
Correct
0
Incorrect
1
Unanswered
3
Accuracy
0.0%
Average
88.2%
Pediatrics
Correct
0
Incorrect
4
Unanswered
13
Accuracy
0.0%
Average
82.0%
Pharmacology
Correct
1
Incorrect
3
Unanswered
19
Accuracy
4.3%
Average
85.4%
Psychiatry
Correct
0
Incorrect
2
Unanswered
8
Accuracy
0.0%
Average
89.5%
Pulmonology
Correct
1
Incorrect
1
Unanswered
17
Accuracy
5.3%
Average
80.6%
Radiology-Emergency
Correct
0
Incorrect
1
Unanswered
13
Accuracy
0.0%
Average
64.9%
Rheumatology
Correct
0
Incorrect
2
Unanswered
12
Accuracy
0.0%
Average
81.4%
Statistics
Correct
0
Incorrect
1
Unanswered
2
Accuracy
0.0%
Average
91.1%
Traumatology
Correct
0
Incorrect
5
Unanswered
10
Accuracy
0.0%
Average
74.5%
Urology
Correct
0
Incorrect
1
Unanswered
5
Accuracy
0.0%
Average
78.2%

Question Type Breakdown

Anatomy
Correct
0
Incorrect
1
Unanswered
5
Accuracy
0.0%
Average
79.8%
Biostatistics
Correct
0
Incorrect
1
Unanswered
4
Accuracy
0.0%
Average
90.7%
Diagnosis
Correct
3
Incorrect
8
Unanswered
62
Accuracy
4.1%
Average
79.2%
Epidemiology
Correct
1
Incorrect
2
Unanswered
9
Accuracy
8.3%
Average
81.2%
Ethics
Correct
0
Incorrect
1
Unanswered
0
Accuracy
0.0%
Average
94.5%
Interpretation
Correct
1
Incorrect
3
Unanswered
33
Accuracy
2.7%
Average
69.6%
Pathophysiology
Correct
0
Incorrect
9
Unanswered
24
Accuracy
0.0%
Average
85.4%
Pharmacology
Correct
1
Incorrect
2
Unanswered
22
Accuracy
4.0%
Average
84.0%
Prevention
Correct
1
Incorrect
2
Unanswered
9
Accuracy
8.3%
Average
89.8%
Prognosis
Correct
1
Incorrect
0
Unanswered
6
Accuracy
14.3%
Average
83.9%
Risk
Correct
0
Incorrect
1
Unanswered
12
Accuracy
0.0%
Average
83.6%
Tests
Correct
0
Incorrect
4
Unanswered
17
Accuracy
0.0%
Average
73.9%
Treatment
Correct
4
Incorrect
14
Unanswered
53
Accuracy
5.6%
Average
81.3%
#AnswerCorrectStatus
1—B
2—D
3—B
4—C
5—C
6—B
7—D
8—C
9—A
10—D
11—D
12DA
13—C
14—A
15—B
16—A
17DC
18—A
19—B
20—C
21—D
22—B
23AA
24—A
25—C
26CB
27—C
28—A
29DB
30—C
31AD
32BA
33—C
34—B
35—D
36—D
37—A
38—A
39—C
40—B
41—C
42—D
43—A
44—D
45CD
46—B
47—C
48—C
49—B
50—C
51AA
52BD
53—C
54—B
55AC
56—D
57—A
58AA
59—A
60—A
61—A
62—D
63AD
64——Annulled
65—D
66—C
67—B
68——Annulled
69—A
70—B
71—B
72BD
73DB
74—C
75—B
76—A
77—D
78—C
79—B
80—A
81—C
82—C
83—B
84DC
85—A
86—A
87—B
88—D
89CB
90—A
91AD
92—A
93BC
94—B
95DD
96—B
97—B
98CB
99—A
100CB
101—A
102—D
103—B
104CD
105AB
106—C
107—C
108—B
109—D
110—D
111—B
112—C
113A—Annulled
114—D
115AD
116CA
117—D
118—D
119—A
120—C
121—A
122—B
123—D
124—D
125—B
126—D
127AA
128—B
129BD
130—C
131—C
132—D
133CA
134AC
135CA
136—D
137—A
138—C
139AA
140—C
141—B
142—C
143—A
144—D
145—C
146—C
147AC
148—A
149—C
150—D
151—A
152—A
153—C
154—B
155BD
156—C
157—C
158—D
159—D
160—B
161—B
162—B
163AB
164DB
165—A
166—C
167—A
168—B
169—C
170—A
171—D
172AB
173—A
174—B
175AA
176—C
177—C
178—B
179—C
180——Annulled
181—B
182AD
183—C
184—A
185—C
186—D
187BA
188—C
189CD
190—D
191—B
192—B
193—C
194—C
195—C
196—B
197—A
198AB
199—D
200—A
201—B
202—D
203—B
204—D
205—D
206——Annulled
207—A
208—A
209—B
210AD