MedicalBenchmark
AlfredPros: CodeLLaMa 7B Instruct Solidity provider

CodeLLaMa 7B Instruct Solidity

318

#318 of 320 models — MIR 2024

Net score

2.33 pts

Accuracy

5.5%

Correct / Incorrect

11 / 26

Total Cost

$0.35

Overall Performance

(vs. average)
Accuracy

5.5%

avg: 81.3%

Net score

2.33 pts

avg: 153.08 pts

Correct

11

avg: 163

Incorrect

26

avg: 29

Total Cost

$0.35

avg: $3.09

Average response time

31.1s

avg: 17.7s

Output Tokens

209K

avg: 414K

Reasoning Tokens

0

avg: 296K

Average confidence

24.0%

avg: 95.7%

Subject Breakdown

Allergology
Correct
0
Incorrect
1
Unanswered
2
Accuracy
0.0%
Average
90.8%
Anesthesiology and Resuscitation
Correct
0
Incorrect
0
Unanswered
4
Accuracy
0.0%
Average
87.7%
Cardiology
Correct
0
Incorrect
2
Unanswered
19
Accuracy
0.0%
Average
80.4%
Dermatology
Correct
0
Incorrect
1
Unanswered
13
Accuracy
0.0%
Average
81.0%
Endocrinology and Nutrition
Correct
0
Incorrect
3
Unanswered
16
Accuracy
0.0%
Average
85.1%
ENT
Correct
0
Incorrect
3
Unanswered
4
Accuracy
0.0%
Average
75.1%
Epidemiology
Correct
1
Incorrect
1
Unanswered
6
Accuracy
12.5%
Average
89.7%
Gastroenterology
Correct
0
Incorrect
5
Unanswered
17
Accuracy
0.0%
Average
71.5%
Genetics
Correct
0
Incorrect
1
Unanswered
6
Accuracy
0.0%
Average
87.1%
Geriatrics
Correct
1
Incorrect
0
Unanswered
9
Accuracy
10.0%
Average
87.7%
Gynecology and Obstetrics
Correct
1
Incorrect
2
Unanswered
11
Accuracy
7.1%
Average
82.0%
Health Planning and Management
Correct
0
Incorrect
0
Unanswered
2
Accuracy
0.0%
Average
75.1%
Hematology
Correct
2
Incorrect
0
Unanswered
11
Accuracy
15.4%
Average
82.4%
Immunology
Correct
0
Incorrect
1
Unanswered
7
Accuracy
0.0%
Average
89.7%
Infectious Diseases
Correct
1
Incorrect
2
Unanswered
20
Accuracy
4.3%
Average
82.5%
Legal Medicine and Bioethics
Correct
0
Incorrect
0
Unanswered
2
Accuracy
0.0%
Average
91.8%
Medical Oncology
Correct
2
Incorrect
3
Unanswered
16
Accuracy
9.5%
Average
80.9%
Nephrology
Correct
1
Incorrect
0
Unanswered
12
Accuracy
7.7%
Average
81.8%
Neurology
Correct
3
Incorrect
3
Unanswered
16
Accuracy
13.6%
Average
84.5%
Ophthalmology
Correct
0
Incorrect
1
Unanswered
4
Accuracy
0.0%
Average
81.3%
Palliative Care
Correct
0
Incorrect
1
Unanswered
3
Accuracy
0.0%
Average
88.6%
Pediatrics
Correct
1
Incorrect
0
Unanswered
16
Accuracy
5.9%
Average
82.9%
Pharmacology
Correct
3
Incorrect
0
Unanswered
20
Accuracy
13.0%
Average
85.8%
Psychiatry
Correct
0
Incorrect
1
Unanswered
9
Accuracy
0.0%
Average
90.0%
Pulmonology
Correct
2
Incorrect
2
Unanswered
15
Accuracy
10.5%
Average
81.6%
Radiology-Emergency
Correct
0
Incorrect
2
Unanswered
12
Accuracy
0.0%
Average
66.0%
Rheumatology
Correct
0
Incorrect
1
Unanswered
13
Accuracy
0.0%
Average
82.4%
Statistics
Correct
1
Incorrect
0
Unanswered
2
Accuracy
33.3%
Average
91.6%
Traumatology
Correct
0
Incorrect
3
Unanswered
12
Accuracy
0.0%
Average
75.4%
Urology
Correct
1
Incorrect
1
Unanswered
4
Accuracy
16.7%
Average
79.0%

Question Type Breakdown

Anatomy
Correct
0
Incorrect
3
Unanswered
3
Accuracy
0.0%
Average
81.1%
Biostatistics
Correct
1
Incorrect
0
Unanswered
4
Accuracy
20.0%
Average
91.3%
Diagnosis
Correct
3
Incorrect
13
Unanswered
57
Accuracy
4.1%
Average
80.0%
Epidemiology
Correct
1
Incorrect
1
Unanswered
10
Accuracy
8.3%
Average
82.1%
Ethics
Correct
0
Incorrect
0
Unanswered
1
Accuracy
0.0%
Average
94.0%
Interpretation
Correct
3
Incorrect
5
Unanswered
29
Accuracy
8.1%
Average
70.5%
Pathophysiology
Correct
2
Incorrect
4
Unanswered
27
Accuracy
6.1%
Average
86.1%
Pharmacology
Correct
2
Incorrect
1
Unanswered
22
Accuracy
8.0%
Average
84.7%
Prevention
Correct
0
Incorrect
1
Unanswered
11
Accuracy
0.0%
Average
90.3%
Prognosis
Correct
1
Incorrect
0
Unanswered
6
Accuracy
14.3%
Average
84.6%
Risk
Correct
1
Incorrect
0
Unanswered
12
Accuracy
7.7%
Average
84.5%
Tests
Correct
2
Incorrect
3
Unanswered
16
Accuracy
9.5%
Average
75.0%
Treatment
Correct
2
Incorrect
10
Unanswered
59
Accuracy
2.8%
Average
82.1%
#AnswerCorrectStatus
1DB
2AD
3—B
4—C
5—C
6—B
7CD
8—C
9—A
10DD
11—D
12—A
13—C
14—A
15—B
16AA
17—C
18DA
19—B
20CC
21—D
22—B
23—A
24—A
25—C
26—B
27AC
28—A
29—B
30AC
31—D
32—A
33—C
34—B
35—D
36—D
37—A
38—A
39AC
40—B
41—C
42—D
43CA
44—D
45—D
46BB
47—C
48—C
49—B
50—C
51—A
52DD
53—C
54—B
55—C
56—D
57CA
58CA
59—A
60—A
61BA
62—D
63AD
64——Annulled
65—D
66—C
67—B
68——Annulled
69—A
70—B
71DB
72—D
73—B
74—C
75—B
76—A
77—D
78—C
79—B
80—A
81—C
82—C
83—B
84—C
85—A
86AA
87DB
88—D
89—B
90—A
91—D
92—A
93—C
94—B
95CD
96—B
97—B
98—B
99—A
100—B
101DA
102—D
103—B
104—D
105—B
106CC
107—C
108—B
109—D
110CD
111—B
112—C
113——Annulled
114—D
115—D
116CA
117—D
118—D
119—A
120—C
121—A
122—B
123—D
124DD
125—B
126AD
127—A
128—B
129—D
130—C
131—C
132—D
133CA
134—C
135—A
136BD
137—A
138—C
139—A
140—C
141—B
142—C
143—A
144DD
145—C
146CC
147—C
148—A
149—C
150—D
151—A
152—A
153—C
154—B
155—D
156—C
157—C
158—D
159—D
160—B
161—B
162—B
163DB
164—B
165—A
166—C
167DA
168—B
169BC
170—A
171—D
172—B
173—A
174—B
175—A
176—C
177—C
178—B
179—C
180——Annulled
181—B
182—D
183AC
184—A
185—C
186—D
187—A
188—C
189—D
190—D
191—B
192—B
193—C
194—C
195—C
196—B
197AA
198—B
199—D
200—A
201—B
202—D
203CB
204—D
205—D
206——Annulled
207—A
208—A
209CB
210—D