MedicalBenchmark
AlfredPros: CodeLLaMa 7B Instruct Solidity provider

CodeLLaMa 7B Instruct Solidity

290

#290 of 291 models — MIR 2024

Net score

2.33 pts

Accuracy

5.5%

Correct / Incorrect

11 / 26

Total Cost

$0.34

Overall Performance

(vs. average)
Accuracy

5.5%

avg: 80.5%

Net score

2.33 pts

avg: 150.85 pts

Correct

11

avg: 161

Incorrect

26

avg: 30

Total Cost

$0.34

avg: $3.32

Average response time

30.6s

avg: 16.4s

Output Tokens

205K

avg: 427K

Reasoning Tokens

0

avg: 310K

Average confidence

24.0%

avg: 95.4%

Subject Breakdown

Allergology
Correct
0
Incorrect
1
Unanswered
2
Accuracy
0.0%
Average
90.5%
Anesthesiology and Resuscitation
Correct
0
Incorrect
0
Unanswered
4
Accuracy
0.0%
Average
87.1%
Cardiology
Correct
0
Incorrect
2
Unanswered
19
Accuracy
0.0%
Average
79.7%
Dermatology
Correct
0
Incorrect
1
Unanswered
13
Accuracy
0.0%
Average
80.2%
Endocrinology and Nutrition
Correct
0
Incorrect
3
Unanswered
16
Accuracy
0.0%
Average
84.2%
ENT
Correct
0
Incorrect
3
Unanswered
4
Accuracy
0.0%
Average
74.4%
Epidemiology
Correct
1
Incorrect
1
Unanswered
6
Accuracy
12.5%
Average
89.3%
Gastroenterology
Correct
0
Incorrect
5
Unanswered
17
Accuracy
0.0%
Average
70.5%
Genetics
Correct
0
Incorrect
1
Unanswered
6
Accuracy
0.0%
Average
86.5%
Geriatrics
Correct
1
Incorrect
0
Unanswered
9
Accuracy
10.0%
Average
86.9%
Gynecology and Obstetrics
Correct
1
Incorrect
2
Unanswered
11
Accuracy
7.1%
Average
81.2%
Health Planning and Management
Correct
0
Incorrect
0
Unanswered
2
Accuracy
0.0%
Average
73.2%
Hematology
Correct
2
Incorrect
0
Unanswered
11
Accuracy
15.4%
Average
81.5%
Immunology
Correct
0
Incorrect
1
Unanswered
7
Accuracy
0.0%
Average
89.1%
Infectious Diseases
Correct
1
Incorrect
2
Unanswered
20
Accuracy
4.3%
Average
81.8%
Legal Medicine and Bioethics
Correct
0
Incorrect
0
Unanswered
2
Accuracy
0.0%
Average
91.7%
Medical Oncology
Correct
2
Incorrect
3
Unanswered
16
Accuracy
9.5%
Average
80.2%
Nephrology
Correct
1
Incorrect
0
Unanswered
12
Accuracy
7.7%
Average
80.8%
Neurology
Correct
3
Incorrect
3
Unanswered
16
Accuracy
13.6%
Average
83.7%
Ophthalmology
Correct
0
Incorrect
1
Unanswered
4
Accuracy
0.0%
Average
80.0%
Palliative Care
Correct
0
Incorrect
1
Unanswered
3
Accuracy
0.0%
Average
88.2%
Pediatrics
Correct
1
Incorrect
0
Unanswered
16
Accuracy
5.9%
Average
82.0%
Pharmacology
Correct
3
Incorrect
0
Unanswered
20
Accuracy
13.0%
Average
85.4%
Psychiatry
Correct
0
Incorrect
1
Unanswered
9
Accuracy
0.0%
Average
89.5%
Pulmonology
Correct
2
Incorrect
2
Unanswered
15
Accuracy
10.5%
Average
80.6%
Radiology-Emergency
Correct
0
Incorrect
2
Unanswered
12
Accuracy
0.0%
Average
64.9%
Rheumatology
Correct
0
Incorrect
1
Unanswered
13
Accuracy
0.0%
Average
81.4%
Statistics
Correct
1
Incorrect
0
Unanswered
2
Accuracy
33.3%
Average
91.1%
Traumatology
Correct
0
Incorrect
3
Unanswered
12
Accuracy
0.0%
Average
74.5%
Urology
Correct
1
Incorrect
1
Unanswered
4
Accuracy
16.7%
Average
78.2%

Question Type Breakdown

Anatomy
Correct
0
Incorrect
3
Unanswered
3
Accuracy
0.0%
Average
79.8%
Biostatistics
Correct
1
Incorrect
0
Unanswered
4
Accuracy
20.0%
Average
90.7%
Diagnosis
Correct
3
Incorrect
13
Unanswered
57
Accuracy
4.1%
Average
79.2%
Epidemiology
Correct
1
Incorrect
1
Unanswered
10
Accuracy
8.3%
Average
81.2%
Ethics
Correct
0
Incorrect
0
Unanswered
1
Accuracy
0.0%
Average
94.5%
Interpretation
Correct
3
Incorrect
5
Unanswered
29
Accuracy
8.1%
Average
69.6%
Pathophysiology
Correct
2
Incorrect
4
Unanswered
27
Accuracy
6.1%
Average
85.4%
Pharmacology
Correct
2
Incorrect
1
Unanswered
22
Accuracy
8.0%
Average
84.0%
Prevention
Correct
0
Incorrect
1
Unanswered
11
Accuracy
0.0%
Average
89.8%
Prognosis
Correct
1
Incorrect
0
Unanswered
6
Accuracy
14.3%
Average
83.9%
Risk
Correct
1
Incorrect
0
Unanswered
12
Accuracy
7.7%
Average
83.6%
Tests
Correct
2
Incorrect
3
Unanswered
16
Accuracy
9.5%
Average
73.9%
Treatment
Correct
2
Incorrect
10
Unanswered
59
Accuracy
2.8%
Average
81.3%
#AnswerCorrectStatus
1DB
2AD
3—B
4—C
5—C
6—B
7CD
8—C
9—A
10DD
11—D
12—A
13—C
14—A
15—B
16AA
17—C
18DA
19—B
20CC
21—D
22—B
23—A
24—A
25—C
26—B
27AC
28—A
29—B
30AC
31—D
32—A
33—C
34—B
35—D
36—D
37—A
38—A
39AC
40—B
41—C
42—D
43CA
44—D
45—D
46BB
47—C
48—C
49—B
50—C
51—A
52DD
53—C
54—B
55—C
56—D
57CA
58CA
59—A
60—A
61BA
62—D
63AD
64——Annulled
65—D
66—C
67—B
68——Annulled
69—A
70—B
71DB
72—D
73—B
74—C
75—B
76—A
77—D
78—C
79—B
80—A
81—C
82—C
83—B
84—C
85—A
86AA
87DB
88—D
89—B
90—A
91—D
92—A
93—C
94—B
95CD
96—B
97—B
98—B
99—A
100—B
101DA
102—D
103—B
104—D
105—B
106CC
107—C
108—B
109—D
110CD
111—B
112—C
113——Annulled
114—D
115—D
116CA
117—D
118—D
119—A
120—C
121—A
122—B
123—D
124DD
125—B
126AD
127—A
128—B
129—D
130—C
131—C
132—D
133CA
134—C
135—A
136BD
137—A
138—C
139—A
140—C
141—B
142—C
143—A
144DD
145—C
146CC
147—C
148—A
149—C
150—D
151—A
152—A
153—C
154—B
155—D
156—C
157—C
158—D
159—D
160—B
161—B
162—B
163DB
164—B
165—A
166—C
167DA
168—B
169BC
170—A
171—D
172—B
173—A
174—B
175—A
176—C
177—C
178—B
179—C
180——Annulled
181—B
182—D
183AC
184—A
185—C
186—D
187—A
188—C
189—D
190—D
191—B
192—B
193—C
194—C
195—C
196—B
197AA
198—B
199—D
200—A
201—B
202—D
203CB
204—D
205—D
206——Annulled
207—A
208—A
209CB
210—D