はじめにリーダーボードモデルハードウェア評価マーケットプレイスレンタルProAPIドキュメント
言語
Atomic Chat campaign banner

HellaSwag

Official shard eval

HellaSwag deterministic disjoint eval shards from Rowan/hellaswag:validation. Running all 13 shards covers the full 10,042-question dataset; Postgres stores only shard metadata and pass/fail results.

Source
Category: ReasoningEval type: Shard evalQuestions: 10,042Shards: 13Runs: 34
Dry-run first: lmx eval shard hellaswag --base-url http://localhost:8000 --questions 371 --dry-run. Then submit with a real model and hardware profile: lmx eval shard hellaswag --base-url http://localhost:8000 --questions 371 --model <hfId> --hardware hardware.json --submit. Scores are pooled by unique question_id; the leaderboard is ranked by Wilson 95% lower bound. HellaSwag uses canonical continuation loglikelihood scoring. Use a logprob-capable /v1/completions endpoint, or pass --model-path model.gguf to use the bundled local llama.cpp scorer.

Leaderboard

Qwopus3.6-27B-Coder-MTP-GGUFQ5_K_M · ggufunknown protocol · unknown agent
Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF · 1 run · 1/13 shards · harness-scoped
92.9%
95% CI 90.894.5%
717/772 correct · 7.7% coverage
2
Gemma-4-31B-IT-NVFP4unspecified quantunknown protocol · unknown agent
nvidia/Gemma-4-31B-IT-NVFP4 · 12 runs · 12/13 shards · harness-scoped
80.9%
95% CI 79.782.0%
3600/4452 correct · 44.3% coverage
3
Qwen3.6-27B-NVFP4NVFP4unknown protocol · unknown agent
nvidia/Qwen3.6-27B-NVFP4 · 13 runs · 13/13 shards · harness-scoped
79.8%
95% CI 78.680.9%
3848/4823 correct · 48.0% coverage
4
Hy-MT2-7Bunspecified quantunknown protocol · unknown agent
tencent/Hy-MT2-7B · 2 runs · 2/13 shards · harness-scoped
68.5%
95% CI 63.473.2%
233/340 correct · 3.4% coverage
5
gemma-4-E4B-itunspecified quantunknown protocol · unknown agent
google/gemma-4-E4B-it · 1 run · 1/13 shards · harness-scoped
65.2%
95% CI 59.570.4%
189/290 correct · 2.9% coverage
6
MiniCPM3-4Bunspecified quantunknown protocol · unknown agent
openbmb/MiniCPM3-4B · 2 runs · 2/13 shards · harness-scoped
52.0%
95% CI 45.158.7%
106/204 correct · 2.0% coverage
7
gemma-4-12b-it-GGUFQ8_K_XL · gguf
unsloth/gemma-4-12b-it-GGUF · 1 run · 1/13 shards · harness-scoped
54.0%
95% CI 44.363.4%
54/100 correct · 1.0% coverage
8
unsloth/gemma-4-26B-A4B-it-qat-GGUF · 1 run · 1/13 shards · harness-scoped
52.0%
95% CI 42.361.5%
52/100 correct · 1.0% coverage
9
MiniCPM5-1Bunspecified quantunknown protocol · unknown agent
openbmb/MiniCPM5-1B · 1 run · 1/13 shards · harness-scoped
20.0%
95% CI 11.233.0%
10/50 correct · 0.5% coverage

Stability— historical rerun transparency

Leaderboard rank uses the canonical latest approved answer per question_id. These metrics include historical submissions too, so reruns and changed answers are visible but do not drive rank.

Qwopus3.6-27B-Coder-MTP-GGUFQ5_K_M · ggufunknown protocol · unknown agent
1 historical run · 772 unique questions
Canonical
92.9%
Row avg
92.9%
Run avg
92.9%
Repeated
0
Changed
0
Gemma-4-31B-IT-NVFP4unspecified quantunknown protocol · unknown agent
12 historical runs · 4,452 unique questions
Canonical
80.9%
Row avg
80.9%
Run avg
80.9%
Repeated
0
Changed
0
Qwen3.6-27B-NVFP4NVFP4unknown protocol · unknown agent
13 historical runs · 4,823 unique questions
Canonical
79.8%
Row avg
79.8%
Run avg
79.8%
Repeated
0
Changed
0
Hy-MT2-7Bunspecified quantunknown protocol · unknown agent
2 historical runs · 340 unique questions
Canonical
68.5%
Row avg
68.5%
Run avg
72.4%
Repeated
0
Changed
0
gemma-4-E4B-itunspecified quantunknown protocol · unknown agent
1 historical run · 290 unique questions
Canonical
65.2%
Row avg
65.2%
Run avg
65.2%
Repeated
0
Changed
0
MiniCPM3-4Bunspecified quantunknown protocol · unknown agent
2 historical runs · 204 unique questions
Canonical
52.0%
Row avg
52.0%
Run avg
75.6%
Repeated
0
Changed
0
gemma-4-12b-it-GGUFQ8_K_XL · gguf
0 historical runs · 100 unique questions
Canonical
54.0%
Row avg
Run avg
Repeated
0
Changed
0
0 historical runs · 100 unique questions
Canonical
52.0%
Row avg
Run avg
Repeated
0
Changed
0
MiniCPM5-1Bunspecified quantunknown protocol · unknown agent
1 historical run · 50 unique questions
Canonical
20.0%
Row avg
20.0%
Run avg
20.0%
Repeated
0
Changed
0

Runs— sample traces per run

MiniCPM5-1Bunspecified quantunknown protocol · unknown agent
by mortimer22oc-arch · shard 8 · 7/9/2026, 9:28:17 PM · cmre0qj2g01j3o401rmz5fo2f
20.0%
10/50 correct · 5 correct traces · 5 incorrect traces
Correct samples
sample 1 · hellaswag:00008131pass · 100.0% · 2209ms · c7b72ea1deda
Question
[header] How to flex lats [title] Learn how to isolate your lats when flexing. [step] Your lats are a pair of wide muscles that run from your armpits to your lower spine, wrapping around the back of your ribcage just beneath your shoulder blades. To figure out how to isolate your lats, first stand with your feet together.

A. Your toes should be together, with your butt facing downward. Close your abdominal muscles in the groin to get into this position.
B. Hold your ankles firmly in your right hand and your ankles together with your left hand with your hands overlapped, in a criss cross pattern of parallel and vertical lines. [title] Use a ruler to identify your lats.
C. Squat forward so that your back and front ribs are aligned and your toes are parallel with the floor. Raise your elbows at an angle and bend your elbow at 90 degrees.
D. Arch your lower back so that your chest is pushed forward and your butt is pushed out. Spread your upper back by lifting your arms slightly away from your sides and holding your hands about 6 inches (15 cm) in front of your hips.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to flex lats [title] Learn how to isolate your lats when flexing. [step] Your lats are a pair of wide muscles that run from your armpits to your lower spine, wrapping around the back of your ribcage just beneath your shoulder blades. To figure out how to isolate your lats, first stand with your feet together.

A. Your toes should be together, with your butt facing downward. Close your abdominal muscles in the groin to get into this position.
B. Hold your ankles firmly in your right hand and your ankles together with your left hand with your hands overlapped, in a criss cross pattern of parallel and vertical lines. [title] Use a ruler to identify your lats.
C. Squat forward so that your back and front ribs are aligned and your toes are parallel with the floor. Raise your elbows at an angle and bend your elbow at 90 degrees.
D. Arch your lower back so that your chest is pushed forward and your butt is pushed out. Spread your upper back by lifting your arms slightly away from your sides and holding your hands about 6 inches (15 cm) in front of your hips.
Model answer
D
Extracted: D · Gold: D
sample 5 · hellaswag:00005144pass · 100.0% · 510ms · af8bdb0d38dd
Question
[header] How to succeed in distance learning [title] Familiarize yourself with the syllabus. [step] Think of your syllabus as the blueprint for your class. It should contain all of the class guidelines and rules, as well as what your instructor expects from you over the course of the semester.

A. If you're having trouble understanding all of the syllabus instructions, ask your instructor for advice. [substeps] To become a successful athlete, you have to have a strong understanding of the game and field, and then, with that knowledge, bring out your best on a single outing.
B. Include these with your notes, papers, assignments and applications so you'll be up to date and able to fully experience your classroom. [substeps] Do a quick background check on your syllabus for your instructor.
C. Bring this information to the syllabus, stating all required daily requirements, as well as any tests you may have scheduled or deadlines you must meet for your session. This way, you can easily adjust any assignments you may finish before the next session starts.
D. Make sure you read through this document in its entirety and keep track of it all semester. [substeps] Ask your instructor any questions you might have about the syllabus.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to succeed in distance learning [title] Familiarize yourself with the syllabus. [step] Think of your syllabus as the blueprint for your class. It should contain all of the class guidelines and rules, as well as what your instructor expects from you over the course of the semester.

A. If you're having trouble understanding all of the syllabus instructions, ask your instructor for advice. [substeps] To become a successful athlete, you have to have a strong understanding of the game and field, and then, with that knowledge, bring out your best on a single outing.
B. Include these with your notes, papers, assignments and applications so you'll be up to date and able to fully experience your classroom. [substeps] Do a quick background check on your syllabus for your instructor.
C. Bring this information to the syllabus, stating all required daily requirements, as well as any tests you may have scheduled or deadlines you must meet for your session. This way, you can easily adjust any assignments you may finish before the next session starts.
D. Make sure you read through this document in its entirety and keep track of it all semester. [substeps] Ask your instructor any questions you might have about the syllabus.
Model answer
D
Extracted: D · Gold: D
sample 7 · hellaswag:00006318pass · 100.0% · 507ms · e875db09a616
Question
[header] How to get other guys to stop staring at your pretty wife [title] Get your wife's opinion. [step] Value how your wife feels about the situation and get an understanding of how if affects her. If she is bothered by the stares, discuss whether or not you should intervene on her behalf.

A. If you feel the stares are a direct threat, try to think about your wife's feelings. Most of all, give her the benefit of the doubt-no guy wants to get a first-degree stare.
B. If she says " i don't feel like i need your attention right now, " then you can bring it up with her and let her ask her opinions. [substeps] Try to use humor to break the ice.
C. [substeps] If you start to think her boyfriend is doing this, report it back to him. If he is only doing this to steal her attention, admit that you feel sorry for him.
D. [substeps] If you have addressed this issue in the past, then you may already know how your wife feels about the attention that she receives. If this is the case, make a decision in advance on how you will respond so that you do not overreact.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to get other guys to stop staring at your pretty wife [title] Get your wife's opinion. [step] Value how your wife feels about the situation and get an understanding of how if affects her. If she is bothered by the stares, discuss whether or not you should intervene on her behalf.

A. If you feel the stares are a direct threat, try to think about your wife's feelings. Most of all, give her the benefit of the doubt-no guy wants to get a first-degree stare.
B. If she says " i don't feel like i need your attention right now, " then you can bring it up with her and let her ask her opinions. [substeps] Try to use humor to break the ice.
C. [substeps] If you start to think her boyfriend is doing this, report it back to him. If he is only doing this to steal her attention, admit that you feel sorry for him.
D. [substeps] If you have addressed this issue in the past, then you may already know how your wife feels about the attention that she receives. If this is the case, make a decision in advance on how you will respond so that you do not overreact.
Model answer
D
Extracted: D · Gold: D
sample 15 · hellaswag:00008167pass · 100.0% · 557ms · 6a3a797910b0
Question
[header] How to get around munich on the cheap [title] Use an automated teller machine (atm) to get the best rates for withdrawing money. [step] Try to make few withdrawals for larger amounts of money to prevent several transaction fees from adding up. [title] Pay with local currency instead of traveler's checks or credit cards as some places offering a good deal only take cash for payment.

A. [title] Use your credit card to make a few turns on the street before you take a taxi or subway. [step] Use your atm to take the bus or subway to munich.
B. [step] Make a large amount out of either military advance, student-issued or visa money. [title] Take a bowl of candy available in the restaurant at the end of the day.
C. [step] Get some savings if you can, as you will most likely need a lot of this money when you leave. [title] Keep a record of the amount required and the type of transaction you're dealing with.
D. [title] Purchase an airport-city-day ticket if you landed at the airport in munich. [step] It is good on all means of transportation on munich's public transportation network from the moment of purchase until 6 a.m.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to get around munich on the cheap [title] Use an automated teller machine (atm) to get the best rates for withdrawing money. [step] Try to make few withdrawals for larger amounts of money to prevent several transaction fees from adding up. [title] Pay with local currency instead of traveler's checks or credit cards as some places offering a good deal only take cash for payment.

A. [title] Use your credit card to make a few turns on the street before you take a taxi or subway. [step] Use your atm to take the bus or subway to munich.
B. [step] Make a large amount out of either military advance, student-issued or visa money. [title] Take a bowl of candy available in the restaurant at the end of the day.
C. [step] Get some savings if you can, as you will most likely need a lot of this money when you leave. [title] Keep a record of the amount required and the type of transaction you're dealing with.
D. [title] Purchase an airport-city-day ticket if you landed at the airport in munich. [step] It is good on all means of transportation on munich's public transportation network from the moment of purchase until 6 a.m.
Model answer
D
Extracted: D · Gold: D
sample 28 · hellaswag:00005154pass · 100.0% · 510ms · 0c93137495f8
Question
[header] How to pass notes in class without the teacher spotting you [title] Make the notes as small as possible. [step] Smaller notes are easier to hide. Make small pieces of paper (about 3 by 5 inches/7.5 cm x 12. 5 cm) to pass in-between you and your friend.

A. Write things that your teacher might not notice. [substeps] For instance, write down spelling and grammar mistakes.
B. [substeps] If you're trying to pass notes by hand, make more small ones to be hidden. If your friend has a small notebook or pencil, write down how much of its contents they'll need to memorize.
C. This will make passing between your friend and the teacher much easier. [title] Feel for the subject as you pass.
D. Perhaps make the little note papers outside of class and keep them ready stashed in your pencil case, to avoid having the teacher ask you to stop tearing paper during class. [title] Write the message on a spare scrap of paper, ideally when the teacher is writing on the board or talking to a classmate across the room.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to pass notes in class without the teacher spotting you [title] Make the notes as small as possible. [step] Smaller notes are easier to hide. Make small pieces of paper (about 3 by 5 inches/7.5 cm x 12. 5 cm) to pass in-between you and your friend.

A. Write things that your teacher might not notice. [substeps] For instance, write down spelling and grammar mistakes.
B. [substeps] If you're trying to pass notes by hand, make more small ones to be hidden. If your friend has a small notebook or pencil, write down how much of its contents they'll need to memorize.
C. This will make passing between your friend and the teacher much easier. [title] Feel for the subject as you pass.
D. Perhaps make the little note papers outside of class and keep them ready stashed in your pencil case, to avoid having the teacher ask you to stop tearing paper during class. [title] Write the message on a spare scrap of paper, ideally when the teacher is writing on the board or talking to a classmate across the room.
Model answer
D
Extracted: D · Gold: D
Incorrect samples
sample 2 · hellaswag:00009130fail · 0.0% · 389ms · ea60b3974f44
Question
[header] How to apply makeup for a glamour photography shoot [title] Clean your face. [step] Always start with a clean face. Leave no traces of your previous makeup.

A. If you will use a pore-cleaning strip, do it the day before to avoid makeup get caught in your newly cleansed pores. Exfoliate to feel that silky smooth texture of your skin.
B. Do not blush, define, or dry dry your face before applying eyeliner in this photoshoot. [title] Apply foundation and concealer to your face.
C. Inspect your face for blemishes with a makeup primer or something like that. Use a gentle cleanser and carefully pat your face dry with a soft soft cloth.
D. A healthy glow has been built up in your face over the past day. In order to achieve the glow, begin by exfoliating your face with a gentle face cleanser.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to apply makeup for a glamour photography shoot [title] Clean your face. [step] Always start with a clean face. Leave no traces of your previous makeup.

A. If you will use a pore-cleaning strip, do it the day before to avoid makeup get caught in your newly cleansed pores. Exfoliate to feel that silky smooth texture of your skin.
B. Do not blush, define, or dry dry your face before applying eyeliner in this photoshoot. [title] Apply foundation and concealer to your face.
C. Inspect your face for blemishes with a makeup primer or something like that. Use a gentle cleanser and carefully pat your face dry with a soft soft cloth.
D. A healthy glow has been built up in your face over the past day. In order to achieve the glow, begin by exfoliating your face with a gentle face cleanser.
Model answer
D
Extracted: D · Gold: A
sample 3 · hellaswag:00009876fail · 0.0% · 487ms · 7349454cbaa5
Question
[header] How to show cleavage [title] Wear tops that show off your cleavage. [step] While this may seem like a given, the right top is absolutely essential when showing cleavage. You can't show something that's covered up.

A. Also, make sure your tops are not g-rated, meaning those aren't appropriate for a particular time of day, or if you have an event coming up soon. [substeps] If you love showing cleavage, you want to show off that.
B. Make sure you're comfortable with your body in the top so you can show it up. [substeps] Studies show that flat strapless tops are considered the sexiest choice for showing cleavage.
C. While a turtleneck sweater may make your bust look fantastic, it will hide your cleavage. Make sure what you're wearing is cut for visible cleavage.
D. Ask a friend if she'd be okay with a top showing their cleavage, and if she says yes! [title] Go for a sleek cut. [step] The summer and winter months are the ideal time to show your cleavage.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to show cleavage [title] Wear tops that show off your cleavage. [step] While this may seem like a given, the right top is absolutely essential when showing cleavage. You can't show something that's covered up.

A. Also, make sure your tops are not g-rated, meaning those aren't appropriate for a particular time of day, or if you have an event coming up soon. [substeps] If you love showing cleavage, you want to show off that.
B. Make sure you're comfortable with your body in the top so you can show it up. [substeps] Studies show that flat strapless tops are considered the sexiest choice for showing cleavage.
C. While a turtleneck sweater may make your bust look fantastic, it will hide your cleavage. Make sure what you're wearing is cut for visible cleavage.
D. Ask a friend if she'd be okay with a top showing their cleavage, and if she says yes! [title] Go for a sleek cut. [step] The summer and winter months are the ideal time to show your cleavage.
Model answer
D
Extracted: D · Gold: C
sample 4 · hellaswag:00003669fail · 0.0% · 552ms · b43cc7df4607
Question
[header] How to enjoy your birthday [title] If you like planning, then plan how you'll spend the day at least a week ahead. [step] Check that your friends and family are free for the day and make sure you choose things to do that can involve the people you'd like to spend time with. [title] Get plenty of sleep.

A. [step] A really good time to get a full night's sleep is about 7 am; the rest of the night should be uninterrupted. It is also good to unwind a bit in the evening and can be done so that you can think about each day.
B. [step] And make sure to get 7-9 hours of sleep a night-1 1/2 hours for adults and 4-9 hours for children. You can even make it up to four hours of sleep on weekends.
C. [step] The day before your birthday, go to bed at a reasonable hour so that you can get the sleep needed to guarantee that you'll have lots of energy on your birthday. [title] Treat yourself to an awesome birthday outfit.
D. [step] The best things you can do for a birthday is to get a good quality sleep. Your body clock is only going to be able to tell you the exact amount of sleep you need.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to enjoy your birthday [title] If you like planning, then plan how you'll spend the day at least a week ahead. [step] Check that your friends and family are free for the day and make sure you choose things to do that can involve the people you'd like to spend time with. [title] Get plenty of sleep.

A. [step] A really good time to get a full night's sleep is about 7 am; the rest of the night should be uninterrupted. It is also good to unwind a bit in the evening and can be done so that you can think about each day.
B. [step] And make sure to get 7-9 hours of sleep a night-1 1/2 hours for adults and 4-9 hours for children. You can even make it up to four hours of sleep on weekends.
C. [step] The day before your birthday, go to bed at a reasonable hour so that you can get the sleep needed to guarantee that you'll have lots of energy on your birthday. [title] Treat yourself to an awesome birthday outfit.
D. [step] The best things you can do for a birthday is to get a good quality sleep. Your body clock is only going to be able to tell you the exact amount of sleep you need.
Model answer
D
Extracted: D · Gold: C
sample 6 · hellaswag:00005768fail · 0.0% · 387ms · 73813eb994e7
Question
[header] How to make spicy chicken bites [title] Preheat the oven to 450º fahrenheit (230º celsius). [title] Line a large baking sheet with parchment paper. [title] Combine the chicken bites with the garlic and paprika.

A. [title] Season the chicken bites with salt and freshly ground black pepper. [title] Stir together the chickpeas, seasoning with nutmeg, currants, cumin and dried ginger in a mixing bowl.
B. [step] Toss the ingredients together in a large bowl until fully coated. [title] Toss in the flour and fold together until well-coated.
C. [title] Stir in ground beef and chicken bouillon and stir. [title] Add the oil and cook it for 2 minutes.
D. [title] Spread the chicken bites over the coated baking sheet. [title] Bake until they've cooked.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to make spicy chicken bites [title] Preheat the oven to 450º fahrenheit (230º celsius). [title] Line a large baking sheet with parchment paper. [title] Combine the chicken bites with the garlic and paprika.

A. [title] Season the chicken bites with salt and freshly ground black pepper. [title] Stir together the chickpeas, seasoning with nutmeg, currants, cumin and dried ginger in a mixing bowl.
B. [step] Toss the ingredients together in a large bowl until fully coated. [title] Toss in the flour and fold together until well-coated.
C. [title] Stir in ground beef and chicken bouillon and stir. [title] Add the oil and cook it for 2 minutes.
D. [title] Spread the chicken bites over the coated baking sheet. [title] Bake until they've cooked.
Model answer
D
Extracted: D · Gold: B
sample 8 · hellaswag:00004787fail · 0.0% · 507ms · b4018df2f67d
Question
[header] How to date a guy from another school [title] Schedule your dates in advance. [step] It's easy to get wrapped up in daily life at school. Since you won't see your guy everyday, make sure to schedule time for each other after school or on weekends.

A. That way, you have a few days free to do something nice together and check in on each other. [substeps] You can also ask your guy when he needs something and send him your numbers.
B. Once school starts, prioritize being with him and avoiding being alone together. Try to schedule outings with his friends if you're in school together.
C. Work out your other commitments so that you get to see him at least once a week, if not more. Having a plan to see each other will give you something to look forward to and make the relationship exciting.
D. [substeps] Even if your guy spends all day on school, try scheduling dates with your friends. You can take advantage of your free time to do something fun together, like have a swim, walk your dog, or watch a play or movie.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to date a guy from another school [title] Schedule your dates in advance. [step] It's easy to get wrapped up in daily life at school. Since you won't see your guy everyday, make sure to schedule time for each other after school or on weekends.

A. That way, you have a few days free to do something nice together and check in on each other. [substeps] You can also ask your guy when he needs something and send him your numbers.
B. Once school starts, prioritize being with him and avoiding being alone together. Try to schedule outings with his friends if you're in school together.
C. Work out your other commitments so that you get to see him at least once a week, if not more. Having a plan to see each other will give you something to look forward to and make the relationship exciting.
D. [substeps] Even if your guy spends all day on school, try scheduling dates with your friends. You can take advantage of your free time to do something fun together, like have a swim, walk your dog, or watch a play or movie.
Model answer
D
Extracted: D · Gold: C
Hy-MT2-7Bunspecified quantunknown protocol · unknown agent
by mortimer22oc-arch · shard 7 · 7/9/2026, 8:55:46 PM · cmrdzkpp801abo4012ru560pw
66.9%
194/290 correct · 5 correct traces · 5 incorrect traces
Correct samples
sample 1 · hellaswag:00007843pass · 100.0% · 2977ms · 5cc50137e5c7
Question
[header] How to cite a book [title] Familiarize yourself with the accepted citation styles. [step] Use only one style when citing a book reference-each style has very specific rules about capitalization, punctuation and placement of the facts. That said, all styles are designed for the same purpose: to give proper credit where credit is due.

A. It is not necessary to include more than your attention span. You could include reading journal articles (if they were used before) or writing your own start point book reviews (if they are not never published by the author).
B. Listed below are the most common styles : [substeps] Modern language association (mla) style. This is a citation style most often used within the liberal arts and humanities departments of universities.
C. [substeps] Use a capital letter or a comma after the reference terms of a book reference. This will refer to the same format in a sentence-to-sentence format.
D. [substeps] There are several great writers' styles for reference: paul golding, glenwhelp, dell, k. paul wilson (1960s innovator), william investments investors, allan cannon (1938), bob n.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to cite a book [title] Familiarize yourself with the accepted citation styles. [step] Use only one style when citing a book reference-each style has very specific rules about capitalization, punctuation and placement of the facts. That said, all styles are designed for the same purpose: to give proper credit where credit is due.

A. It is not necessary to include more than your attention span. You could include reading journal articles (if they were used before) or writing your own start point book reviews (if they are not never published by the author).
B. Listed below are the most common styles : [substeps] Modern language association (mla) style. This is a citation style most often used within the liberal arts and humanities departments of universities.
C. [substeps] Use a capital letter or a comma after the reference terms of a book reference. This will refer to the same format in a sentence-to-sentence format.
D. [substeps] There are several great writers' styles for reference: paul golding, glenwhelp, dell, k. paul wilson (1960s innovator), william investments investors, allan cannon (1938), bob n.
Model answer
B
Extracted: B · Gold: B
sample 2 · hellaswag:00006278pass · 100.0% · 1630ms · e34b53861a5e
Question
[header] How to be emo in middle school [title] Learn what it means to be emo. [step] Before you can become emo, you have to understand exactly what it means. " emo " is short for emotive, and people who consider themselves to be emo are usually in touch with their emotions.

A. They don't feel the need to hide how they feel, even when they are sad or angry. [substeps] While music, clothing, hair, and makeup can play a large role in being emo, being comfortable with your feelings and expressing them is the most important part.
B. This is an important step to emo, so have a good understanding of your gender. You don't have to have what everyone else has.
C. You don't need a dictionary dictionary and stick with the " emotive ". Instead, learn what emotive means, like " i am ", " i am skinny ", " i am strong and strong ", " i'm confident in my body ", etc.
D. You can't be emo without being emo. [substeps] If you're emo, you want an emo vibe that's unique to you.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to be emo in middle school [title] Learn what it means to be emo. [step] Before you can become emo, you have to understand exactly what it means. " emo " is short for emotive, and people who consider themselves to be emo are usually in touch with their emotions.

A. They don't feel the need to hide how they feel, even when they are sad or angry. [substeps] While music, clothing, hair, and makeup can play a large role in being emo, being comfortable with your feelings and expressing them is the most important part.
B. This is an important step to emo, so have a good understanding of your gender. You don't have to have what everyone else has.
C. You don't need a dictionary dictionary and stick with the " emotive ". Instead, learn what emotive means, like " i am ", " i am skinny ", " i am strong and strong ", " i'm confident in my body ", etc.
D. You can't be emo without being emo. [substeps] If you're emo, you want an emo vibe that's unique to you.
Model answer
A
Extracted: A · Gold: A
sample 4 · hellaswag:00003431pass · 100.0% · 1676ms · fda7a18bf9c9
Question
[header] How to apply shadow on hooded eyes [title] Prime your eyelids. [step] While this is important for everyone, it's especially critical for those with hooded eyes. Primer creates a base for your makeup that helps it stay put and last all day.

A. [substeps] If you use liquid primer, try it first before you start applying it on your eyes. Otherwise, you'll try a different primer beforehand.
B. Primer also keeps your eyelids from getting heavy and swollen. To prime your eyelids : [substeps] Apply an eyeshadow to your lids using a dark gray eye liner or gel instead of a cream or gel liner.
C. Because hooded eyes are prone to smudging and smearing, primer can make all the difference. [substeps] Use your fingertip to apply the primer, and allow it to absorb into your skin for a minute or so before proceeding.
D. Apply a primer to your eyelids and eyelid over the primer you applied earlier. [substeps] Use your powder brush, primer, or a makeup sponge to carefully prime your eyelids.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to apply shadow on hooded eyes [title] Prime your eyelids. [step] While this is important for everyone, it's especially critical for those with hooded eyes. Primer creates a base for your makeup that helps it stay put and last all day.

A. [substeps] If you use liquid primer, try it first before you start applying it on your eyes. Otherwise, you'll try a different primer beforehand.
B. Primer also keeps your eyelids from getting heavy and swollen. To prime your eyelids : [substeps] Apply an eyeshadow to your lids using a dark gray eye liner or gel instead of a cream or gel liner.
C. Because hooded eyes are prone to smudging and smearing, primer can make all the difference. [substeps] Use your fingertip to apply the primer, and allow it to absorb into your skin for a minute or so before proceeding.
D. Apply a primer to your eyelids and eyelid over the primer you applied earlier. [substeps] Use your powder brush, primer, or a makeup sponge to carefully prime your eyelids.
Model answer
C
Extracted: C · Gold: C
sample 5 · hellaswag:00004697pass · 100.0% · 1655ms · 002775c993f6
Question
[header] How to get favorites on your deviantart photo [title] Create your own username. [step] Have a name that can be easily recognized and defines you as a person. [substeps] Your username doesn't have to be your first name, but some well-known deviantart members remember their art style through their da names.

A. Avoid using names of famous celebrities or character from movies, video games, or animes. To change your username, you have to purchase premium membership.
B. If you don't want people to recognize you as a person, try to keep the name honest and easy to identify. You can do this by including your full name and what year it is, but stay away from androgynous names.
C. For instance, president christian founder of the white house had a username that " peter " (modern), and there are so many other names available today. In the wild, some names are simply variations of the last name.
D. [title] Create usernames for your favorite characters. [step] Depending on your theme and design, you may have many different options.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to get favorites on your deviantart photo [title] Create your own username. [step] Have a name that can be easily recognized and defines you as a person. [substeps] Your username doesn't have to be your first name, but some well-known deviantart members remember their art style through their da names.

A. Avoid using names of famous celebrities or character from movies, video games, or animes. To change your username, you have to purchase premium membership.
B. If you don't want people to recognize you as a person, try to keep the name honest and easy to identify. You can do this by including your full name and what year it is, but stay away from androgynous names.
C. For instance, president christian founder of the white house had a username that " peter " (modern), and there are so many other names available today. In the wild, some names are simply variations of the last name.
D. [title] Create usernames for your favorite characters. [step] Depending on your theme and design, you may have many different options.
Model answer
A
Extracted: A · Gold: A
sample 7 · hellaswag:00009247pass · 100.0% · 1840ms · d0632a00ca4b
Question
[header] How to decide whether to go abroad for dental treatment [title] Start your search on google or some other search engine. [step] Make a list of the clinics that are featured and that appeal to you. Remember the sponsored clinics have paid to be on the first position, they have not necessarily been selected by google as suiting your criteria.

A. [title] Find out how long the clinic has been offering treatment to overseas patients. [step] If they have been around a long time then they probably have a good track record.
B. [title] Ask around for " dentists for dental care " posters or " in-clinic treatments " flyers. [step] Also, make sure you get the name of appropriate dr.
C. [title] Leave notes at the clinic each day. [step] A written note is always needed in case you are unable to attend the appointment while you are away.
D. Checking the number of clinics you wish to visit and contacting them makes the decision much easier and easier. [title] Visit a conversation, tea or even facial shop to try to talk to a dentist, he or she will be able to check your dental work and provide you with mouth health options.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to decide whether to go abroad for dental treatment [title] Start your search on google or some other search engine. [step] Make a list of the clinics that are featured and that appeal to you. Remember the sponsored clinics have paid to be on the first position, they have not necessarily been selected by google as suiting your criteria.

A. [title] Find out how long the clinic has been offering treatment to overseas patients. [step] If they have been around a long time then they probably have a good track record.
B. [title] Ask around for " dentists for dental care " posters or " in-clinic treatments " flyers. [step] Also, make sure you get the name of appropriate dr.
C. [title] Leave notes at the clinic each day. [step] A written note is always needed in case you are unable to attend the appointment while you are away.
D. Checking the number of clinics you wish to visit and contacting them makes the decision much easier and easier. [title] Visit a conversation, tea or even facial shop to try to talk to a dentist, he or she will be able to check your dental work and provide you with mouth health options.
Model answer
A
Extracted: A · Gold: A
Incorrect samples
sample 3 · hellaswag:00004307fail · 0.0% · 1607ms · 24b28583cdae
Question
[header] How to make a smoke bomb [title] Gather three or four ping-pong balls. [step] For this method, you don't have to buy any extra ingredients other than basic household supplies, like aluminum foil, a pencil, screwdriver, and scissors. [title] Cut a hole in the top of one of the balls.

A. [step] You can poke a hole with a screwdriver or a knife. [title] Cut the other balls into small pieces and put them in the hole.
B. [step] Use a knick knack knife to create an x around one of the balls. [title] Punch a hole in the bottom of one of the balls.
C. [step] Align the hole in the top ball with the one that's down. [substeps] The hole needs to be about six-ten inches deep and about two inches wide.
D. [step] Grab a piece of foil about 15 inches from the edge of the ball. Get a zig-zag or intricate pattern.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to make a smoke bomb [title] Gather three or four ping-pong balls. [step] For this method, you don't have to buy any extra ingredients other than basic household supplies, like aluminum foil, a pencil, screwdriver, and scissors. [title] Cut a hole in the top of one of the balls.

A. [step] You can poke a hole with a screwdriver or a knife. [title] Cut the other balls into small pieces and put them in the hole.
B. [step] Use a knick knack knife to create an x around one of the balls. [title] Punch a hole in the bottom of one of the balls.
C. [step] Align the hole in the top ball with the one that's down. [substeps] The hole needs to be about six-ten inches deep and about two inches wide.
D. [step] Grab a piece of foil about 15 inches from the edge of the ball. Get a zig-zag or intricate pattern.
Model answer
C
Extracted: C · Gold: A
sample 6 · hellaswag:00002238fail · 0.0% · 751ms · 90522e4a1c7a
Question
The man lays down and had a hard time breathing. After putting down sand he drinks a warm beverage. we

A. see the man underwater swimming.
B. see the ending credits screen.
C. then see two guys swimming nak * d and drinking.
D. see no other man and the other man is shown shirtless.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

The man lays down and had a hard time breathing. After putting down sand he drinks a warm beverage. we

A. see the man underwater swimming.
B. see the ending credits screen.
C. then see two guys swimming nak * d and drinking.
D. see no other man and the other man is shown shirtless.
Model answer
D
Extracted: D · Gold: B
sample 14 · hellaswag:00002753fail · 0.0% · 753ms · 77953ff24c1d
Question
A mom is sitting at the top of a slide with her little baby. she

A. slides down one at a time, and she tends to make a lot of noise.
B. is moving backward on the playground, walking away.
C. pulls out several balls as she slides down.
D. is contemplating going down for a while.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

A mom is sitting at the top of a slide with her little baby. she

A. slides down one at a time, and she tends to make a lot of noise.
B. is moving backward on the playground, walking away.
C. pulls out several balls as she slides down.
D. is contemplating going down for a while.
Model answer
A
Extracted: A · Gold: D
sample 15 · hellaswag:00005437fail · 0.0% · 2233ms · f105409d2cb5
Question
[header] How to participate in clinical trials [title] Talk to your doctor. [step] If you have a chronic illness or medical condition, your primary treatment provider is your best source for clinical trials that might benefit you. Let your doctor know that you are interested in exploring the possibilities.

A. [substeps] Bring your ob/gyn, a professional, with all of your medical information if you are not sure where to begin. Either ask, " what would be the best treatment program for me? " or " do you know of any facilities near me that might benefit me? " [title] Educate others.
B. Make sure that your medical provider knows exactly what sort of information you'll want to explore and will be able to help you find the materials you want to explore. [substeps] Even if you don't get infected with lupus, know that you are still a patient for one year, and you will experience no symptoms over time.
C. [substeps] If you hear of a trial on your own, bring it up to your doctor and let them know you'd like to participate. They can advise you on whether they think you would benefit from the drug or treatment being studied.
D. If possible, contact your doctor for more information. [substeps] Keep in mind that clinical trials are usually subsidized by a state or federal government agency and some states will not approve conditions they deemed appropriate for clinical trials.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to participate in clinical trials [title] Talk to your doctor. [step] If you have a chronic illness or medical condition, your primary treatment provider is your best source for clinical trials that might benefit you. Let your doctor know that you are interested in exploring the possibilities.

A. [substeps] Bring your ob/gyn, a professional, with all of your medical information if you are not sure where to begin. Either ask, " what would be the best treatment program for me? " or " do you know of any facilities near me that might benefit me? " [title] Educate others.
B. Make sure that your medical provider knows exactly what sort of information you'll want to explore and will be able to help you find the materials you want to explore. [substeps] Even if you don't get infected with lupus, know that you are still a patient for one year, and you will experience no symptoms over time.
C. [substeps] If you hear of a trial on your own, bring it up to your doctor and let them know you'd like to participate. They can advise you on whether they think you would benefit from the drug or treatment being studied.
D. If possible, contact your doctor for more information. [substeps] Keep in mind that clinical trials are usually subsidized by a state or federal government agency and some states will not approve conditions they deemed appropriate for clinical trials.
Model answer
D
Extracted: D · Gold: C
sample 17 · hellaswag:00004677fail · 0.0% · 1536ms · da10bc594647
Question
[header] How to switch tabs in chrome [title] Switch to the next tab. [step] Press + to change to the next tab in the window. This will move you to the tab to the right of your current tab.

A. If you are already on the furthest right tab, this will send you to the one on the far left. This works on windows, mac, chromebook, or linux, but some operating systems have additional options : [substeps] You also have the option to use ctrl + pgdn.
B. [substeps] Click link/click next to update. [title] Press + to switch tabs in the tab to the left.
C. You may also need to scroll down to find it. [title] Click on the tab to go back into browsing mode.
D. Move to the tab to the right of your current tab. [title] Click the " pin up " tab.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to switch tabs in chrome [title] Switch to the next tab. [step] Press + to change to the next tab in the window. This will move you to the tab to the right of your current tab.

A. If you are already on the furthest right tab, this will send you to the one on the far left. This works on windows, mac, chromebook, or linux, but some operating systems have additional options : [substeps] You also have the option to use ctrl + pgdn.
B. [substeps] Click link/click next to update. [title] Press + to switch tabs in the tab to the left.
C. You may also need to scroll down to find it. [title] Click on the tab to go back into browsing mode.
D. Move to the tab to the right of your current tab. [title] Click the " pin up " tab.
Model answer
D
Extracted: D · Gold: A
Hy-MT2-7Bunspecified quantunknown protocol · unknown agent
by mortimer22oc-arch · shard 6 · 7/9/2026, 8:46:46 PM · cmrdz958n018ho401c83jt5s7
78.0%
39/50 correct · 5 correct traces · 5 incorrect traces
Correct samples
sample 1 · hellaswag:00000731pass · 100.0% · 8972ms · 315b3bc53367
Question
Maria is demonstrating how to make a nyquil cocktail drink. she

A. shows the ingredients required for the drink.
B. pours it through a straw into a glass and washes it down with a cup of soda water.
C. shows each scale of the cocktail by adding different colored liquids.
D. pours the ice in a small glass and squirts lemon juice into the glass.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

Maria is demonstrating how to make a nyquil cocktail drink. she

A. shows the ingredients required for the drink.
B. pours it through a straw into a glass and washes it down with a cup of soda water.
C. shows each scale of the cocktail by adding different colored liquids.
D. pours the ice in a small glass and squirts lemon juice into the glass.
Model answer
A
Extracted: A · Gold: A
sample 2 · hellaswag:00006203pass · 100.0% · 1845ms · 510fb1db449c
Question
[header] How to zone agricultural land for use as a wedding venue [title] Talk to your neighbors. [step] Your neighbors can be your strongest supporters or your most bitter rivals if you want to rezone your property. The best way to potentially get them on your side is to keep them informed of your plans every step of the way.

A. [substeps] In the early stages, let them know about your plans and reassure them that you will keep them informed on developments. Give them the chance to ask questions, and do everything you can to assure them that events you host won't cause any disturbances to them or their property.
B. [substeps] Make sure you know your neighbors' lives and can help them out if they disagree about your plan. Give them written correspondence so they can be helpful to you if needed.
C. This means that if your neighbors are anything but informed, you should talk to them as well. If you don't know them well enough to keep detailed notes, reach out to them and ask them to help keep a record of your plans.
D. [substeps] Don't forget that some of your family's acreage may need transplant rights. Always decide ahead of time what kind of liens will be put on what land you are planning on and why.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to zone agricultural land for use as a wedding venue [title] Talk to your neighbors. [step] Your neighbors can be your strongest supporters or your most bitter rivals if you want to rezone your property. The best way to potentially get them on your side is to keep them informed of your plans every step of the way.

A. [substeps] In the early stages, let them know about your plans and reassure them that you will keep them informed on developments. Give them the chance to ask questions, and do everything you can to assure them that events you host won't cause any disturbances to them or their property.
B. [substeps] Make sure you know your neighbors' lives and can help them out if they disagree about your plan. Give them written correspondence so they can be helpful to you if needed.
C. This means that if your neighbors are anything but informed, you should talk to them as well. If you don't know them well enough to keep detailed notes, reach out to them and ask them to help keep a record of your plans.
D. [substeps] Don't forget that some of your family's acreage may need transplant rights. Always decide ahead of time what kind of liens will be put on what land you are planning on and why.
Model answer
A
Extracted: A · Gold: A
sample 4 · hellaswag:00009360pass · 100.0% · 1865ms · 772bf3a3a4f6
Question
[header] How to get ready for a party [title] Prepare anything you need to bring. [step] If the party is a potluck, you'll want to make and bring a dish to share. If the party is a birthday party or some kind of shower, you'll need to bring a gift.

A. If you are throwing a birthday party or if your parents are fine with it, feel free to bring these items. [substeps] In a party, use sachets or strings to write things in.
B. If the party is a dinner party, you should bring a bottle of wine or a hostess gift. At more casual parties in high school or college, it's generally expected that guests won't bring anything unless it's been specified on the invitation.
C. You'll also want to bring a good notebook and a pen so you can write down ideas on how to best decorate the house. A classic approach would be to bring plastic utensils, such as wine glasses.
D. [title] Know what you're expected to wear. [step] If you're just going to school, you might wear a dress, but if you're doing other things, such as going to a dance, you might wear some pajamas or something a little fancier like a dress.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to get ready for a party [title] Prepare anything you need to bring. [step] If the party is a potluck, you'll want to make and bring a dish to share. If the party is a birthday party or some kind of shower, you'll need to bring a gift.

A. If you are throwing a birthday party or if your parents are fine with it, feel free to bring these items. [substeps] In a party, use sachets or strings to write things in.
B. If the party is a dinner party, you should bring a bottle of wine or a hostess gift. At more casual parties in high school or college, it's generally expected that guests won't bring anything unless it's been specified on the invitation.
C. You'll also want to bring a good notebook and a pen so you can write down ideas on how to best decorate the house. A classic approach would be to bring plastic utensils, such as wine glasses.
D. [title] Know what you're expected to wear. [step] If you're just going to school, you might wear a dress, but if you're doing other things, such as going to a dance, you might wear some pajamas or something a little fancier like a dress.
Model answer
B
Extracted: B · Gold: B
sample 5 · hellaswag:00004111pass · 100.0% · 1646ms · 804eac15909f
Question
[header] How to style mom jeans [title] Pick a pair of jeans that fit your waist well. [step] Mom jeans are meant to sit high on your waist. When picking out a new pair, try them on and see how the waist fits-you don't want the jeans too tight or too loose.

A. Try on any pair and see if it's even snug. [substeps] If your mom jeans tend to dig into your legs, look for jeans that come up above the knee.
B. [substeps] Pick out jeans that are at the leg length of your legs. If you have shorter legs, opt for what looks best to you, rather than what might look nice with longer pants.
C. [substeps] The high waist will show off your waist and elongate your legs. [title] Choose a leg style that's appropriate for your figure.
D. Try buying pairs that are so low that you can still walk in them but they are a little looser. [substeps] Ask your mom about how to size your jeans.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to style mom jeans [title] Pick a pair of jeans that fit your waist well. [step] Mom jeans are meant to sit high on your waist. When picking out a new pair, try them on and see how the waist fits-you don't want the jeans too tight or too loose.

A. Try on any pair and see if it's even snug. [substeps] If your mom jeans tend to dig into your legs, look for jeans that come up above the knee.
B. [substeps] Pick out jeans that are at the leg length of your legs. If you have shorter legs, opt for what looks best to you, rather than what might look nice with longer pants.
C. [substeps] The high waist will show off your waist and elongate your legs. [title] Choose a leg style that's appropriate for your figure.
D. Try buying pairs that are so low that you can still walk in them but they are a little looser. [substeps] Ask your mom about how to size your jeans.
Model answer
C
Extracted: C · Gold: C
sample 6 · hellaswag:00009418pass · 100.0% · 1597ms · 80921f961a36
Question
[header] How to diagnose heel spurs [title] Locate the pain. [step] Heel spurs can show up in multiple places on your heel. This can cause the pain to be slightly different depending on exactly where the heel spur is.

A. It can be hard to judge the strength of the spur and whether or not it is painful at all. When you just try to be more specific about where it is, your foot and heel will become increasingly entangled and there will be no wiggle room.
B. They can be located at the back of the heel or under the heel, near the sole of your foot. If you are experiencing pain along the back of your foot, up through your ankle, you might have a heel spur on the back of your heel.
C. The first position you identify is the thigh. You will see white or pinkish swelling and inflammation along the arch of the heel.
D. The most common places you might see the sensation of the pain are your heels and your nail. [substeps] You may discover that the pain is more acute in one heel then it is in another.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to diagnose heel spurs [title] Locate the pain. [step] Heel spurs can show up in multiple places on your heel. This can cause the pain to be slightly different depending on exactly where the heel spur is.

A. It can be hard to judge the strength of the spur and whether or not it is painful at all. When you just try to be more specific about where it is, your foot and heel will become increasingly entangled and there will be no wiggle room.
B. They can be located at the back of the heel or under the heel, near the sole of your foot. If you are experiencing pain along the back of your foot, up through your ankle, you might have a heel spur on the back of your heel.
C. The first position you identify is the thigh. You will see white or pinkish swelling and inflammation along the arch of the heel.
D. The most common places you might see the sensation of the pain are your heels and your nail. [substeps] You may discover that the pain is more acute in one heel then it is in another.
Model answer
B
Extracted: B · Gold: B
Incorrect samples
sample 3 · hellaswag:00002742fail · 0.0% · 712ms · 43625a13ba92
Question
. a man with a orange shirt and blue gloves

A. is standing in a snow covered parking lot talking while holding a skateboard and talking.
B. is doing gymnastics on an elliptical.
C. is shown operating a fire torch machine.
D. approaches a green motor vehicle that is in a pool.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

. a man with a orange shirt and blue gloves

A. is standing in a snow covered parking lot talking while holding a skateboard and talking.
B. is doing gymnastics on an elliptical.
C. is shown operating a fire torch machine.
D. approaches a green motor vehicle that is in a pool.
Model answer
D
Extracted: D · Gold: C
sample 18 · hellaswag:00002056fail · 0.0% · 938ms · 81e682a6e26a
Question
Another person moves with the camera as they watch from a window in an elevated room against a white window sill. the person blowing the leaves

A. gets closer to the camera as well as the person in the street behind him.
B. gives a final look like he is talking and ends with him dropping the leaves as soon as he makes it to the ground.
C. leaves are shown from a small angle.
D. moves around the yard and continues to do lawn maintenance.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

Another person moves with the camera as they watch from a window in an elevated room against a white window sill. the person blowing the leaves

A. gets closer to the camera as well as the person in the street behind him.
B. gives a final look like he is talking and ends with him dropping the leaves as soon as he makes it to the ground.
C. leaves are shown from a small angle.
D. moves around the yard and continues to do lawn maintenance.
Model answer
B
Extracted: B · Gold: D
sample 23 · hellaswag:00003513fail · 0.0% · 1739ms · 3ec0150cf69c
Question
[header] How to cope with your parents' custody battle (for kids ) [title] Recognize that it isn't your fault that your parents are having problems. [step] You are not the cause. They may not love each other anymore, but they love you, and that's why they are in this custody battle.

A. No matter what you think, your parents are fighting to uphold their rights and responsibilities. [title] Understand that actions can affect your consequences.
B. If you are in your parents' custody battle, your parents are not the reason why they are fighting. [substeps] They are fighting because they are angry with you.
C. Your friends could provide you a little moral support and that could be helpful. [title] Your parents feelings are not your responsibility.
D. However, appreciate that by recognizing what your parents are arguing about, you know that your perspective is the only way you can do better in this situation. Try your best to understand their perspective so that you have a better chance of changing your mind about custody.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to cope with your parents' custody battle (for kids ) [title] Recognize that it isn't your fault that your parents are having problems. [step] You are not the cause. They may not love each other anymore, but they love you, and that's why they are in this custody battle.

A. No matter what you think, your parents are fighting to uphold their rights and responsibilities. [title] Understand that actions can affect your consequences.
B. If you are in your parents' custody battle, your parents are not the reason why they are fighting. [substeps] They are fighting because they are angry with you.
C. Your friends could provide you a little moral support and that could be helpful. [title] Your parents feelings are not your responsibility.
D. However, appreciate that by recognizing what your parents are arguing about, you know that your perspective is the only way you can do better in this situation. Try your best to understand their perspective so that you have a better chance of changing your mind about custody.
Model answer
B
Extracted: B · Gold: C
sample 25 · hellaswag:00009494fail · 0.0% · 1739ms · 60faec525c6b
Question
[header] How to apply eyeshadow that lasts [title] Wash your face. [step] Starting with a clean and fresh face will make your makeup stand out more, help it last longer, and help prevent your makeup from clumping and flaking. Use your favorite gel or cream facial cleanser and lukewarm water.

A. Rub the cleanser in circular motions for 20 minutes before rinsing with warm water. [title] Apply your eyeliner to the crease of your eye.
B. [substeps] Wet your face and pour some cleanser into your hands. Work the cleanser into a lather, and then gently massage it into your face for about 30 seconds using circular motions.
C. Tap your face with your fingers to get the excess off, then pat your face dry with a clean towel. [substeps] Do not splash water onto your face.
D. Pat your face dry with a clean towel. [substeps] An eyeshadow primer also works just as well.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to apply eyeshadow that lasts [title] Wash your face. [step] Starting with a clean and fresh face will make your makeup stand out more, help it last longer, and help prevent your makeup from clumping and flaking. Use your favorite gel or cream facial cleanser and lukewarm water.

A. Rub the cleanser in circular motions for 20 minutes before rinsing with warm water. [title] Apply your eyeliner to the crease of your eye.
B. [substeps] Wet your face and pour some cleanser into your hands. Work the cleanser into a lather, and then gently massage it into your face for about 30 seconds using circular motions.
C. Tap your face with your fingers to get the excess off, then pat your face dry with a clean towel. [substeps] Do not splash water onto your face.
D. Pat your face dry with a clean towel. [substeps] An eyeshadow primer also works just as well.
Model answer
D
Extracted: D · Gold: B
sample 35 · hellaswag:00007831fail · 0.0% · 1838ms · 26ea288bfdca
Question
[header] How to apply eye black for baseball [title] Remove the eye black stick from its package. [title] Take off the lid to the eye black stick. [step] A lid to eye black is removed much in the same way you remove a lid from a stick of lip balm.

A. [title] Apply the eye black to your face. [substeps] Place the stick on the outside of either eye.
B. [substeps] Lipstick is removed from the mouth prior to applying makeup. If you don't want to remove the lid, use either a q-tip, toothbrush or mini bottle.
C. Go with the universal method: simply remove the lid. [title] File the top of the stick onto the lid.
D. The lid may fit in the palm of your hand, but it will slide across your eye to one side. [substeps] If your lid comes in a bottle or small round plastic container, make sure you grab that first.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to apply eye black for baseball [title] Remove the eye black stick from its package. [title] Take off the lid to the eye black stick. [step] A lid to eye black is removed much in the same way you remove a lid from a stick of lip balm.

A. [title] Apply the eye black to your face. [substeps] Place the stick on the outside of either eye.
B. [substeps] Lipstick is removed from the mouth prior to applying makeup. If you don't want to remove the lid, use either a q-tip, toothbrush or mini bottle.
C. Go with the universal method: simply remove the lid. [title] File the top of the stick onto the lid.
D. The lid may fit in the palm of your hand, but it will slide across your eye to one side. [substeps] If your lid comes in a bottle or small round plastic container, make sure you grab that first.
Model answer
C
Extracted: C · Gold: A
MiniCPM3-4Bunspecified quantunknown protocol · unknown agent
by mortimer22oc-arch · shard 5 · 7/9/2026, 8:12:50 PM · cmrdy1ig7012eo401ryrxkp0k
51.2%
103/201 correct · 5 correct traces · 5 incorrect traces
Correct samples
sample 1 · hellaswag:00004149pass · 100.0% · 6210ms · 320beff94af9
Question
[header] How to survive financial hardship in a marriage [title] Remember your wedding vows ,' for better, for worse, for richer, for poorer'. [step] When we make these vows to each other it probably doesn't enter our minds what could happen down the road, but nonetheless we made a promise that we had the best intentions of keeping. During difficult financial times those vows become far more prominent and necessary.

A. [title] Remember not to shut doors at all hours. [step] Shut off all lights at any time, especially those that serve to light up your wedding day.
B. Stick with it not because of the promise you made but because of the person you made it to. [title] Remember that as with everything in life, it will turn around again.
C. We finally remember that our intentions weren't necessarily good and neither were your intentions. A discharge of our vows is a second step towards firbility.
D. [substeps] Write them down for future reference (or on a reference sheet). The more you understand the meaning of financial hardship, the greater your chances at survival.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to survive financial hardship in a marriage [title] Remember your wedding vows ,' for better, for worse, for richer, for poorer'. [step] When we make these vows to each other it probably doesn't enter our minds what could happen down the road, but nonetheless we made a promise that we had the best intentions of keeping. During difficult financial times those vows become far more prominent and necessary.

A. [title] Remember not to shut doors at all hours. [step] Shut off all lights at any time, especially those that serve to light up your wedding day.
B. Stick with it not because of the promise you made but because of the person you made it to. [title] Remember that as with everything in life, it will turn around again.
C. We finally remember that our intentions weren't necessarily good and neither were your intentions. A discharge of our vows is a second step towards firbility.
D. [substeps] Write them down for future reference (or on a reference sheet). The more you understand the meaning of financial hardship, the greater your chances at survival.
Model answer
B
Extracted: B · Gold: B
sample 2 · hellaswag:00009152pass · 100.0% · 1657ms · 39119b83551d
Question
[header] How to plan your wedding music [title] Sit down together and talk about what music matters to you. [step] Each of you will have different ideas and preferences and this is about getting the main ideas heard, to allow both of you to plan in full knowledge of what matters to one another. [substeps] Write down a list of the songs and/or music that you would like to form a part of your wedding.

A. Make sure to decide ahead of time the location where you want to arrange your wedding music. You can discuss how many romantic themes you hope to include in each song and set up time for each venue.
B. Ask your future spouse to do the same. Bring the lists along to your discussion together.
C. Brainstorm a logical order together with your partner. Everyone should have their own wants and preferences and think about what exactly matters to them, in order to get included.
D. Or you can look through the music previously choreographed and combine the music together to put into song form. [title] Decide how the music matters to you.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to plan your wedding music [title] Sit down together and talk about what music matters to you. [step] Each of you will have different ideas and preferences and this is about getting the main ideas heard, to allow both of you to plan in full knowledge of what matters to one another. [substeps] Write down a list of the songs and/or music that you would like to form a part of your wedding.

A. Make sure to decide ahead of time the location where you want to arrange your wedding music. You can discuss how many romantic themes you hope to include in each song and set up time for each venue.
B. Ask your future spouse to do the same. Bring the lists along to your discussion together.
C. Brainstorm a logical order together with your partner. Everyone should have their own wants and preferences and think about what exactly matters to them, in order to get included.
D. Or you can look through the music previously choreographed and combine the music together to put into song form. [title] Decide how the music matters to you.
Model answer
B
Extracted: B · Gold: B
sample 3 · hellaswag:00009645pass · 100.0% · 1970ms · b4e5f821a16b
Question
[header] How to dress in knee length skirts [title] Pick a length that flatters your height. [step] Not everyone has the same body type, and there are some ways you can make a knee-length skirt look better for yours. [substeps] Short women or girls look really good in knee-length skirts that fall right below the knees.

A. When it comes to skirts that hit at your hip, go around the waist to accentuate it. Knee-length skirts are more flattering when they have structure and a twist.
B. Not everyone likes the hemlines of a waist-length skirt, and you don't want the skirt to stop too low on their legs, or they'll look overly babyish. If you're going out to dinner, go for a skirt that reaches the mid-thigh.
C. It's not true that short women can't wear knee-length skirts. If you're very tall, you will look great in a skirt that falls right at the mid-calf point.
D. Medium petite women or girls that are fairly tall don't care if their feet are bare, and may prefer wearing knee-length skirts with ankle-length tights. For a more feminine look, opt for a length that hits at the knee and curves over your thighs.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to dress in knee length skirts [title] Pick a length that flatters your height. [step] Not everyone has the same body type, and there are some ways you can make a knee-length skirt look better for yours. [substeps] Short women or girls look really good in knee-length skirts that fall right below the knees.

A. When it comes to skirts that hit at your hip, go around the waist to accentuate it. Knee-length skirts are more flattering when they have structure and a twist.
B. Not everyone likes the hemlines of a waist-length skirt, and you don't want the skirt to stop too low on their legs, or they'll look overly babyish. If you're going out to dinner, go for a skirt that reaches the mid-thigh.
C. It's not true that short women can't wear knee-length skirts. If you're very tall, you will look great in a skirt that falls right at the mid-calf point.
D. Medium petite women or girls that are fairly tall don't care if their feet are bare, and may prefer wearing knee-length skirts with ankle-length tights. For a more feminine look, opt for a length that hits at the knee and curves over your thighs.
Model answer
C
Extracted: C · Gold: C
sample 5 · hellaswag:00009851pass · 100.0% · 1656ms · cc7ee9946aef
Question
[header] How to set a digital watch [title] Contact the manufacturer or look up a manual for complicated watches. [step] If your watch is especially complicated and you don't have instructions, look up a digital manual online. Digital manuals can be found with an online keyword search for your watch brand and model.

A. [substeps] Some watches turn off automatically during the day. If yours is in the morning, switch it to the middle of the night.
B. Your watch may take you a while, or it may take a little while to get to a level where you won't experience any problem. [substeps] Digital watches come with online cameras and camera boxes.
C. [substeps] If a digital manual is impossible to find, look up and call the customer service number for your watch. Model and brand information are usually written or engraved on the back of the watch or somewhere on the watch face.
D. They are usually $15-$40. They cover the exact specs of your watch and include instructions to change them.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to set a digital watch [title] Contact the manufacturer or look up a manual for complicated watches. [step] If your watch is especially complicated and you don't have instructions, look up a digital manual online. Digital manuals can be found with an online keyword search for your watch brand and model.

A. [substeps] Some watches turn off automatically during the day. If yours is in the morning, switch it to the middle of the night.
B. Your watch may take you a while, or it may take a little while to get to a level where you won't experience any problem. [substeps] Digital watches come with online cameras and camera boxes.
C. [substeps] If a digital manual is impossible to find, look up and call the customer service number for your watch. Model and brand information are usually written or engraved on the back of the watch or somewhere on the watch face.
D. They are usually $15-$40. They cover the exact specs of your watch and include instructions to change them.
Model answer
C
Extracted: C · Gold: C
sample 6 · hellaswag:00009474pass · 100.0% · 1599ms · 57db1a1fe5a4
Question
[header] How to make a birthday mug cake [title] Microwave the butter in a large, microwave-safe mug. [step] Melt it for about 30-45 seconds until it turns into a liquid. [title] Add the flour, egg, sugar, milk, vanilla, and baking powder.

A. [step] Whisk gently with a fork until the ingredients have fully incorporated. [title] Pour in the sprinkles.
B. [step] Stir through to form smooth, well-blended, creamy cake batter. [title] Fill a 16 x 16 inch (33.3 x 27.6 cm) mug pan halfway with baking powder and cook for about 10 minutes.
C. [step] Stir with a mini whisk or flat spatula. [title] Pour the confectioners' sugar into the melted butter mixture.
D. [step] Stir on low speed for about 1 hour. [title] Pour the flour, milk, and baking powder into the mug.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to make a birthday mug cake [title] Microwave the butter in a large, microwave-safe mug. [step] Melt it for about 30-45 seconds until it turns into a liquid. [title] Add the flour, egg, sugar, milk, vanilla, and baking powder.

A. [step] Whisk gently with a fork until the ingredients have fully incorporated. [title] Pour in the sprinkles.
B. [step] Stir through to form smooth, well-blended, creamy cake batter. [title] Fill a 16 x 16 inch (33.3 x 27.6 cm) mug pan halfway with baking powder and cook for about 10 minutes.
C. [step] Stir with a mini whisk or flat spatula. [title] Pour the confectioners' sugar into the melted butter mixture.
D. [step] Stir on low speed for about 1 hour. [title] Pour the flour, milk, and baking powder into the mug.
Model answer
A
Extracted: A · Gold: A
Incorrect samples
sample 4 · hellaswag:00009316fail · 0.0% · 1909ms · 40e43f92db98
Question
[header] How to look like ashley barnum from' perfect ' [title] Get the look. [step] Ashley has blonde hair that ranges from light brown to almost white-blonde near her face. It's described as being' surfer girl' hair, so you should try using the product' surfer hair' by garnier.

A. She also has super-shiny and glossy hair, and it is revealed in the book that she rinses her hair with white vinegar so that it becomes shinier. Now this actually works, so consider it: you will need 1 part white vinegar mixed with three parts cold water.
B. For kit kerr, you need to mix emo' surfer' with ash blonde, but make sure you show your true color. Ashy hair works best, but you can try out the' pure girl' look for makeup and powdering.
C. If you can't find this extremely attractive look, use straighten-up shampoo and conditioner. If you don't like the appearance of her hair, have a good waxing or body hair section done, with a paddle-like shape and wavy roots, to create this look.
D. Hazel eyes that are never quite big are her signature. Look for a hair dye made for fair to medium hair.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to look like ashley barnum from' perfect ' [title] Get the look. [step] Ashley has blonde hair that ranges from light brown to almost white-blonde near her face. It's described as being' surfer girl' hair, so you should try using the product' surfer hair' by garnier.

A. She also has super-shiny and glossy hair, and it is revealed in the book that she rinses her hair with white vinegar so that it becomes shinier. Now this actually works, so consider it: you will need 1 part white vinegar mixed with three parts cold water.
B. For kit kerr, you need to mix emo' surfer' with ash blonde, but make sure you show your true color. Ashy hair works best, but you can try out the' pure girl' look for makeup and powdering.
C. If you can't find this extremely attractive look, use straighten-up shampoo and conditioner. If you don't like the appearance of her hair, have a good waxing or body hair section done, with a paddle-like shape and wavy roots, to create this look.
D. Hazel eyes that are never quite big are her signature. Look for a hair dye made for fair to medium hair.
Model answer
C
Extracted: C · Gold: A
sample 10 · hellaswag:00001958fail · 0.0% · 724ms · 3fedceffa364
Question
Two men wearing hats are seated on a couch. one of the men

A. throws the other one a ball.
B. stands up and grabs the back of the other man's shirt.
C. puts on a pair of stilts and the other mans a helmet.
D. is playing a guitar.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

Two men wearing hats are seated on a couch. one of the men

A. throws the other one a ball.
B. stands up and grabs the back of the other man's shirt.
C. puts on a pair of stilts and the other mans a helmet.
D. is playing a guitar.
Model answer
A
Extracted: A · Gold: D
sample 13 · hellaswag:00009888fail · 0.0% · 1722ms · dc5c17c10590
Question
[header] How to cut your own hair (men ) [title] Wash your hair before you cut it. [step] Your hair will be easier to work with if it's clean and snarl-free. Get your hair wet under running water and use shampoo and conditioner, rinsing both out after you've worked them through your hair.

A. [substeps] If you have no other hair options available, rinse your hair under running water for a minute to get out any tangles. [title] Use hair conditioner after showering.
B. [title] Soak your hair in apple cider vinegar for 20 to 30 minutes. [step] Mix a 1/2 cup (120 ml) of apple cider vinegar in 1 cup (226 ml) of water.
C. [title] Detangle your hair with a comb. [step] Run the comb through your hair and work out any knots with the comb or with your fingers.
D. [title] Rinse and dry your hair with a towel. [step] After you're finished working out your hair, take a good, hot shower.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to cut your own hair (men ) [title] Wash your hair before you cut it. [step] Your hair will be easier to work with if it's clean and snarl-free. Get your hair wet under running water and use shampoo and conditioner, rinsing both out after you've worked them through your hair.

A. [substeps] If you have no other hair options available, rinse your hair under running water for a minute to get out any tangles. [title] Use hair conditioner after showering.
B. [title] Soak your hair in apple cider vinegar for 20 to 30 minutes. [step] Mix a 1/2 cup (120 ml) of apple cider vinegar in 1 cup (226 ml) of water.
C. [title] Detangle your hair with a comb. [step] Run the comb through your hair and work out any knots with the comb or with your fingers.
D. [title] Rinse and dry your hair with a towel. [step] After you're finished working out your hair, take a good, hot shower.
Model answer
A
Extracted: A · Gold: C
sample 14 · hellaswag:00000895fail · 0.0% · 877ms · 81ab8ccd7391
Question
There's an open land area with greenery around and grassy surface area. there

A. is a tennis court and a tennis court on the side.
B. is a man wearing white shirt and jungle gym shorts exercising on the green surface.
C. 's a lawn trimmer lying on the ground near some over grown patches of grass.
D. are no trees, but there is a large green grassy area.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

There's an open land area with greenery around and grassy surface area. there

A. is a tennis court and a tennis court on the side.
B. is a man wearing white shirt and jungle gym shorts exercising on the green surface.
C. 's a lawn trimmer lying on the ground near some over grown patches of grass.
D. are no trees, but there is a large green grassy area.
Model answer
D
Extracted: D · Gold: C
sample 15 · hellaswag:00009257fail · 0.0% · 1244ms · 7c63b78868a2
Question
[header] How to remove turbidity [title] Remove organic debris. [step] Use a pool net to fish out any large, visible debris in the water. Organic debris commonly includes leaves, branches, and dead insects.

A. This debris will cause the turbidity to fade, and bacteria will be unable to grow. Not all organic matter is inherently easy to clean.
B. [substeps] Organic debris can decay and add sediment to the water. The dirt on the debris can also cause cloudiness.
C. If you live in a hot area, dry out your net in a garage, shed, or other combination of locations. By removing this debris, you can keep your home clean.
D. Collecting it in your garden is one way to remove it. [substeps] Collect as much debris as possible and throw it away.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to remove turbidity [title] Remove organic debris. [step] Use a pool net to fish out any large, visible debris in the water. Organic debris commonly includes leaves, branches, and dead insects.

A. This debris will cause the turbidity to fade, and bacteria will be unable to grow. Not all organic matter is inherently easy to clean.
B. [substeps] Organic debris can decay and add sediment to the water. The dirt on the debris can also cause cloudiness.
C. If you live in a hot area, dry out your net in a garage, shed, or other combination of locations. By removing this debris, you can keep your home clean.
D. Collecting it in your garden is one way to remove it. [substeps] Collect as much debris as possible and throw it away.
Model answer
A
Extracted: A · Gold: B
MiniCPM3-4Bunspecified quantunknown protocol · unknown agent
by mortimer22oc-arch · shard 4 · 7/9/2026, 8:06:53 PM · cmrdxtusj0122o401gx9fsuos
100.0%
3/3 correct · 3 correct traces · 0 incorrect traces
Correct samples
sample 1 · hellaswag:00008698pass · 100.0% · 8117ms · e7a500085361
Question
[header] How to get ready for summer using earl grey tea [title] Wash your face. [step] Your face should be squeaky clean. [title] Prepare your earl grey teabags, and a glass with a little hot water in it.

A. [step] Earl grey tea is good for any skin, not everyone's dry skin, and it's not bad for those with acne. [title] Dissolve the ginger and lemon zest.
B. [step] Use more than one teabag for optimal results. [title] Soak your teabags in the hot water to release the tea (or the color pigments you'll need).
C. [step] Take your shampoo and hair dryer and start to blast the hot water down towards your head. Since earl grey tea floats, it doesn't evaporate as quickly as you think.
D. [step] Add a spoonful of earl grey to the glass and mix well with your thumb and fingers. [title] Add a few drops of rosewater and blend well.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to get ready for summer using earl grey tea [title] Wash your face. [step] Your face should be squeaky clean. [title] Prepare your earl grey teabags, and a glass with a little hot water in it.

A. [step] Earl grey tea is good for any skin, not everyone's dry skin, and it's not bad for those with acne. [title] Dissolve the ginger and lemon zest.
B. [step] Use more than one teabag for optimal results. [title] Soak your teabags in the hot water to release the tea (or the color pigments you'll need).
C. [step] Take your shampoo and hair dryer and start to blast the hot water down towards your head. Since earl grey tea floats, it doesn't evaporate as quickly as you think.
D. [step] Add a spoonful of earl grey to the glass and mix well with your thumb and fingers. [title] Add a few drops of rosewater and blend well.
Model answer
B
Extracted: B · Gold: B
sample 2 · hellaswag:00003208pass · 100.0% · 1506ms · 904ec42df76f
Question
A small baby sits in a car seat and hols a cup of food. The light shines bright on the right and scares the baby who begins to cry hard. the light

A. finally dazzles the baby and the video ends.
B. fades and the baby calms down and looks all around the car.
C. moves down to the left to shift back and shines on a woman standing in the sitting room.
D. hurts the baby's eyes as she cries and begins to wake up.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

A small baby sits in a car seat and hols a cup of food. The light shines bright on the right and scares the baby who begins to cry hard. the light

A. finally dazzles the baby and the video ends.
B. fades and the baby calms down and looks all around the car.
C. moves down to the left to shift back and shines on a woman standing in the sitting room.
D. hurts the baby's eyes as she cries and begins to wake up.
Model answer
B. fade(s) and the baby calm(s) down and looks all around the car.
Extracted: B · Gold: B
sample 3 · hellaswag:00008219pass · 100.0% · 1618ms · 48bc2b3a77a0
Question
[header] How to lowlight hair yourself [title] Visit a drugstore or beauty supply store to choose your colors. [step] Some hair dye manufacturers already have products made especially for " at home " lowlights. Pick this if it is available.

A. [title] Buy two bags to color in your color. [step] Homemade dye uses dye directly from the bottle.
B. If not, choose your color carefully according to your natural hair color. [title] Pick a color that is two or three shades darker than your own hair.
C. [substeps] Apply semi-permanent remover to your hair. Then, use just a little of the product to mix all the way up to semi-permanent.
D. [substeps] For lowlights that aren't sold in salon or department stores, use your credit card to buy the color you want. You can also purchase green, red, or rose blushes to match your hair color, but don't overdo it.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to lowlight hair yourself [title] Visit a drugstore or beauty supply store to choose your colors. [step] Some hair dye manufacturers already have products made especially for " at home " lowlights. Pick this if it is available.

A. [title] Buy two bags to color in your color. [step] Homemade dye uses dye directly from the bottle.
B. If not, choose your color carefully according to your natural hair color. [title] Pick a color that is two or three shades darker than your own hair.
C. [substeps] Apply semi-permanent remover to your hair. Then, use just a little of the product to mix all the way up to semi-permanent.
D. [substeps] For lowlights that aren't sold in salon or department stores, use your credit card to buy the color you want. You can also purchase green, red, or rose blushes to match your hair color, but don't overdo it.
Model answer
B
Extracted: B · Gold: B
gemma-4-E4B-itunspecified quantunknown protocol · unknown agent
by mortimer22oc-arch · shard 3 · 7/9/2026, 6:58:57 PM · cmrdvehab00t1o401wi4am916
65.2%
189/290 correct · 5 correct traces · 5 incorrect traces
Correct samples
sample 2 · hellaswag:00003912pass · 100.0% · 2081ms · a38d0df1f39f
Question
[header] How to set up a wedding escort card table [title] Choose decorations that complement your wedding style. [step] Include different items on the escort card table that reflect your theme, like pots of flowers, glitter on the tablecloth, or ornate picture frames. Create different layers on the table and cluster your cards and items together for an interesting display.

A. Use candles or other lighting to brighten up your table. [substeps] Fill in open spots on the table with flowers, ivy, or other botanicals.
B. [substeps] This project also works well with formal wedding outfits. Add plastic floor lamps or built in lanterns for additional color.
C. [title] Purchase a larger, fancier table. [step] Include a ton of trimmings on the table to make it more interesting for guests to see the fancier items.
D. [substeps] Find a custom tablecloth that you can easily cut using scissors. Many people like to use ordinary table cloths but if you'd rather make your own, use latex papier mache for an easy-to-use option.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to set up a wedding escort card table [title] Choose decorations that complement your wedding style. [step] Include different items on the escort card table that reflect your theme, like pots of flowers, glitter on the tablecloth, or ornate picture frames. Create different layers on the table and cluster your cards and items together for an interesting display.

A. Use candles or other lighting to brighten up your table. [substeps] Fill in open spots on the table with flowers, ivy, or other botanicals.
B. [substeps] This project also works well with formal wedding outfits. Add plastic floor lamps or built in lanterns for additional color.
C. [title] Purchase a larger, fancier table. [step] Include a ton of trimmings on the table to make it more interesting for guests to see the fancier items.
D. [substeps] Find a custom tablecloth that you can easily cut using scissors. Many people like to use ordinary table cloths but if you'd rather make your own, use latex papier mache for an easy-to-use option.
Model answer
A
Extracted: A · Gold: A
sample 3 · hellaswag:00008804pass · 100.0% · 1807ms · 09535c730ea0
Question
[header] How to unhook a bra [title] Understand how the closing mechanism works. [step] A typical bra has two horizontal straps that fasten in the back. One strap has several pairs of eyelets.

A. The third strap has two double eyelet tabs that are close to the bra clasp. [substeps] The second strap wraps along the back of the bra and wraps down just over the clasp.
B. They each slide a little free and leave two eyelet-style holes in each shoulder section. The side straps also slide together and create clips, which will hold your bra together when you unclasp it.
C. The other has one pair of hooks that attach to the eyelets. To undo a bra, you must pinch both sides together and slide the hooks out of the eyelets.
D. The smaller strip attaches to the back of the bra through multiple eyelets on either side of the bra. This is the closing mechanism for a standard bra.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to unhook a bra [title] Understand how the closing mechanism works. [step] A typical bra has two horizontal straps that fasten in the back. One strap has several pairs of eyelets.

A. The third strap has two double eyelet tabs that are close to the bra clasp. [substeps] The second strap wraps along the back of the bra and wraps down just over the clasp.
B. They each slide a little free and leave two eyelet-style holes in each shoulder section. The side straps also slide together and create clips, which will hold your bra together when you unclasp it.
C. The other has one pair of hooks that attach to the eyelets. To undo a bra, you must pinch both sides together and slide the hooks out of the eyelets.
D. The smaller strip attaches to the back of the bra through multiple eyelets on either side of the bra. This is the closing mechanism for a standard bra.
Model answer
C
Extracted: C · Gold: C
sample 4 · hellaswag:00006691pass · 100.0% · 2032ms · 7467c511f95e
Question
[header] How to avoid eyebrow piercing scars [title] Choose a quality piercer. [step] The person doing your eyebrow piercing should be a trained professional. They should work in a sterile environment with sterile tools, know the proper piercing technique, be knowledgeable about the piercing industry and experienced in piercing eyebrows.

A. [title] Decide how often you want to see your piercer. [step] Each piercer will have its own schedule, so it should be understandable if you need to see a surgeon once a month.
B. You should feel comfortable with them and able to ask them any questions you have about the piercing. [substeps] Check to see if the piercer is certified by the state and if the piercing facility is licensed by the state.
C. If you're confused about what piercers specialize in, keep looking. If you're on a budget, look for a piercer who is experienced.
D. They should be experienced when piercing blood vessels, especially in specific parts of the body and should be able to take regular images of them. [title] Ask your piercer to do an imaging test.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to avoid eyebrow piercing scars [title] Choose a quality piercer. [step] The person doing your eyebrow piercing should be a trained professional. They should work in a sterile environment with sterile tools, know the proper piercing technique, be knowledgeable about the piercing industry and experienced in piercing eyebrows.

A. [title] Decide how often you want to see your piercer. [step] Each piercer will have its own schedule, so it should be understandable if you need to see a surgeon once a month.
B. You should feel comfortable with them and able to ask them any questions you have about the piercing. [substeps] Check to see if the piercer is certified by the state and if the piercing facility is licensed by the state.
C. If you're confused about what piercers specialize in, keep looking. If you're on a budget, look for a piercer who is experienced.
D. They should be experienced when piercing blood vessels, especially in specific parts of the body and should be able to take regular images of them. [title] Ask your piercer to do an imaging test.
Model answer
B
Extracted: B · Gold: B
sample 5 · hellaswag:00009553pass · 100.0% · 1549ms · e76b39559739
Question
[header] How to be a modern christian girl [title] Pray everyday. [step] Start off with reading a couple of passages in the bible. Then start praying.

A. You should start by praising god and marking him as authority. Then start thanking him, and pray for yourself, and others.
B. Talk it out with a friend and see how it helps you get into a good christian mindset. [title] Don't slouch or crouch too much when you pray.
C. God wants us to learn about god, so always read the books. The more you read, the wider your belly will become.
D. When you have difficulty reading, act enough like you are. After all, it is god's gift to us to learn the bible.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to be a modern christian girl [title] Pray everyday. [step] Start off with reading a couple of passages in the bible. Then start praying.

A. You should start by praising god and marking him as authority. Then start thanking him, and pray for yourself, and others.
B. Talk it out with a friend and see how it helps you get into a good christian mindset. [title] Don't slouch or crouch too much when you pray.
C. God wants us to learn about god, so always read the books. The more you read, the wider your belly will become.
D. When you have difficulty reading, act enough like you are. After all, it is god's gift to us to learn the bible.
Model answer
A
Extracted: A · Gold: A
sample 9 · hellaswag:00007802pass · 100.0% · 2009ms · fdddb6e5220f
Question
[header] How to make snowman earrings [title] Gather your supplies. [step] Some of the supplies you'll need, like the crystal encrusted beads, the headpins, and the earwires with mirror beads, probably aren't going to be lying around your home. Some craft stores will have everything you need, particularly any craft stores with large jewelry-making sections.

A. [substeps] The headpins will look like tiny pieces of silver and gold. The earshells will also be between the wires.
B. Everything comes in single tip sizes, so avoid getting [title] Thread the headpins onto your earrings. [step] Put one wire through one eye loop on the bottom and one in the other eye loop on the top.
C. Check your craft store's website to see what they have for you. [title] Unpin a 5-inch by 6-inch piece of metal from your craft store.
D. If you're unable to find these items in store, you can order everything you need online. [title] Slide your silver bead on the headpin.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to make snowman earrings [title] Gather your supplies. [step] Some of the supplies you'll need, like the crystal encrusted beads, the headpins, and the earwires with mirror beads, probably aren't going to be lying around your home. Some craft stores will have everything you need, particularly any craft stores with large jewelry-making sections.

A. [substeps] The headpins will look like tiny pieces of silver and gold. The earshells will also be between the wires.
B. Everything comes in single tip sizes, so avoid getting [title] Thread the headpins onto your earrings. [step] Put one wire through one eye loop on the bottom and one in the other eye loop on the top.
C. Check your craft store's website to see what they have for you. [title] Unpin a 5-inch by 6-inch piece of metal from your craft store.
D. If you're unable to find these items in store, you can order everything you need online. [title] Slide your silver bead on the headpin.
Model answer
D
Extracted: D · Gold: D
Incorrect samples
sample 1 · hellaswag:00004865fail · 0.0% · 3275ms · e70c2f5da6be
Question
[header] How to avoid teaching your kids prejudices [title] Consider your own biases. [step] Before you even begin to talk with your kids about prejudice and judgmental attitudes, you need to be aware of your own biases. Even if you don't consider yourself to be prejudiced against others, you might realize that you're uncomfortable around some people.

A. This may be a good time to educate yourself. [substeps] You might have been afraid of others by putting the prejudice on the people you're around.
B. Think about why you might feel this way. [substeps] For example, you might realize that you're uncomfortable around people of other religions.
C. [substeps] Rather than fearing how others perceive a particular person, people who are more prejudiced have more confidence in their own life choices. But don't give yourself the opportunity to feel bad for your own biases.
D. Learning to suppress the prejudice of others is not only important to your own personal well-being, but to your child's. [substeps] Developing your own personal biases takes time and effort.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to avoid teaching your kids prejudices [title] Consider your own biases. [step] Before you even begin to talk with your kids about prejudice and judgmental attitudes, you need to be aware of your own biases. Even if you don't consider yourself to be prejudiced against others, you might realize that you're uncomfortable around some people.

A. This may be a good time to educate yourself. [substeps] You might have been afraid of others by putting the prejudice on the people you're around.
B. Think about why you might feel this way. [substeps] For example, you might realize that you're uncomfortable around people of other religions.
C. [substeps] Rather than fearing how others perceive a particular person, people who are more prejudiced have more confidence in their own life choices. But don't give yourself the opportunity to feel bad for your own biases.
D. Learning to suppress the prejudice of others is not only important to your own personal well-being, but to your child's. [substeps] Developing your own personal biases takes time and effort.
Model answer
A
Extracted: A · Gold: B
sample 6 · hellaswag:00009666fail · 0.0% · 2231ms · 36b3b1c75cd2
Question
[header] How to bake chocolate chip cookies without them sticking [title] Use parchment paper. [step] This the best choice. The dough won't move off of it while in dough form, yet the cookies will come right off after only one or two minutes of cooling.

A. [title] Place the prepared baking sheets on the kitchen counter, and melt the butter and the cornstarch in a shallow bowl. [step] This could be used in place of cooking spray or butter in some recipes or just plain plain laying out a cookie sheet on the counter.
B. [title] Dust the parchment paper with powdered sugar and give it a quick wipe to get any sugar or any flour off. [title] Preheat the oven to 375 degrees f (180 c).
C. [title] Bake the dough for 8-10 minutes. [step] Overcooking it will make the cookies burn (duh) and stick to the baking tray.
D. [title] Make sure the parchment paper is a nice color! [step] If you remove the parchment paper it will create an impression. [title] Place the flour and baking mix on a cookie sheet.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to bake chocolate chip cookies without them sticking [title] Use parchment paper. [step] This the best choice. The dough won't move off of it while in dough form, yet the cookies will come right off after only one or two minutes of cooling.

A. [title] Place the prepared baking sheets on the kitchen counter, and melt the butter and the cornstarch in a shallow bowl. [step] This could be used in place of cooking spray or butter in some recipes or just plain plain laying out a cookie sheet on the counter.
B. [title] Dust the parchment paper with powdered sugar and give it a quick wipe to get any sugar or any flour off. [title] Preheat the oven to 375 degrees f (180 c).
C. [title] Bake the dough for 8-10 minutes. [step] Overcooking it will make the cookies burn (duh) and stick to the baking tray.
D. [title] Make sure the parchment paper is a nice color! [step] If you remove the parchment paper it will create an impression. [title] Place the flour and baking mix on a cookie sheet.
Model answer
B
Extracted: B · Gold: C
sample 7 · hellaswag:00008791fail · 0.0% · 1771ms · d3a31df93c40
Question
[header] How to make a gin martini [title] Gather your tools and ingredients. [step] To mix this drink you will need a chilled cocktail shaker, a chilled cocktail glass, a cocktail spoon, a cocktail strainer, a jigger, chilled gin, chilled vermouth, plenty of ice, and an olive. [substeps] Measure out 4 small jiggers of gin.

A. They should be about the same size as straws, plus an extra extra if needed. Optionally add 1 teaspoon of salt in.
B. Measure out 1 small jigger of vermouth. [title] Place your ingredients in the shaker.
C. Place the jiggers into a cocktail shaker and screw the lid on. [title] Measure and pour 2 ounces (120 ml) of gin in the shaker.
D. You should use one shaker for you gin, and one for your olive. Pour in all the ingredients and stir everything together well with a cocktail spoon.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to make a gin martini [title] Gather your tools and ingredients. [step] To mix this drink you will need a chilled cocktail shaker, a chilled cocktail glass, a cocktail spoon, a cocktail strainer, a jigger, chilled gin, chilled vermouth, plenty of ice, and an olive. [substeps] Measure out 4 small jiggers of gin.

A. They should be about the same size as straws, plus an extra extra if needed. Optionally add 1 teaspoon of salt in.
B. Measure out 1 small jigger of vermouth. [title] Place your ingredients in the shaker.
C. Place the jiggers into a cocktail shaker and screw the lid on. [title] Measure and pour 2 ounces (120 ml) of gin in the shaker.
D. You should use one shaker for you gin, and one for your olive. Pour in all the ingredients and stir everything together well with a cocktail spoon.
Model answer
C
Extracted: C · Gold: B
sample 8 · hellaswag:00002695fail · 0.0% · 1246ms · 8cf9019af1fb
Question
There are so many awesome and different things being made out of sand, it looks like it takes a lot of time and energy. there

A. are screens explaining what to do with the sand, how you should use it, and what you should wear to do it.
B. are people walking around and looking at all of them.
C. are so many plants and flowers on the sand, it's unbelievable.
D. are waterlog covered in sand, shingles, and areas filled with a lot of sand, sand castles shown, and miscellaneous sand on the ground and concrete.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

There are so many awesome and different things being made out of sand, it looks like it takes a lot of time and energy. there

A. are screens explaining what to do with the sand, how you should use it, and what you should wear to do it.
B. are people walking around and looking at all of them.
C. are so many plants and flowers on the sand, it's unbelievable.
D. are waterlog covered in sand, shingles, and areas filled with a lot of sand, sand castles shown, and miscellaneous sand on the ground and concrete.
Model answer
D
Extracted: D · Gold: B
sample 12 · hellaswag:00005949fail · 0.0% · 1811ms · 2ee548d660de
Question
[header] How to style wavy frizzy hair [title] Use a moisturizing shampoo and conditioner. [step] If you have naturally wavy and frizzy hair, then using the right shower products is very important. Look for a shampoo or conditioner that advertises its moisturizing qualities.

A. Make sure to massage these products into your scalp while taking a shower and fully rinse them out. [substeps] You can also comb your conditioner through your hair while taking a shower.
B. Make sure that it is water-based, and not shampoo containing sugar. It can make the detangling process time quicker, too.
C. A good moisturizing shampoo and conditioner might include : [substeps] Baby-friendly look: the jet-stream shampoo made of water and honey. Dove hair: the hydrating shampoo specifically designed for curly hair.
D. Remember that there are shampoos and conditioners meant to keep your hair moisturized and healthy. Ideally, use the spray or gel combination that's included in the kit.
Rendered prompt
Answer the question. Reply with only the letter of the correct choice.

[header] How to style wavy frizzy hair [title] Use a moisturizing shampoo and conditioner. [step] If you have naturally wavy and frizzy hair, then using the right shower products is very important. Look for a shampoo or conditioner that advertises its moisturizing qualities.

A. Make sure to massage these products into your scalp while taking a shower and fully rinse them out. [substeps] You can also comb your conditioner through your hair while taking a shower.
B. Make sure that it is water-based, and not shampoo containing sugar. It can make the detangling process time quicker, too.
C. A good moisturizing shampoo and conditioner might include : [substeps] Baby-friendly look: the jet-stream shampoo made of water and honey. Dove hair: the hydrating shampoo specifically designed for curly hair.
D. Remember that there are shampoos and conditioners meant to keep your hair moisturized and healthy. Ideally, use the spray or gel combination that's included in the kit.
Model answer
C
Extracted: C · Gold: A
Qwen3.6-27B-NVFP4NVFP4unknown protocol · unknown agent
by Stewart_of_Mars · shard 13 · 7/1/2026, 1:18:15 PM · cmr23piuc04nkpi01c9juykg5
77.9%
289/371 correct · 5 correct traces · 5 incorrect traces
Correct samples
sample 2 · hellaswag:00006547pass · 100.0% · 4401ms · e498ae6053dd
Question
[header] How to buy organic soil [title] Do your research. [step] Truly organic soil can be hard to come by since the market for organic potting media has only begun increasing in the past few years. Moreover, organic standards have also become stricter as the demand for organic goods has grown, so a soil that was once deemed organic may no longer fulfill those requirements legally.

A. To survive, the best option is to research the quality of organic soil you buy and see how they relate with your home or ecosystem. [substeps] Make sure that your soil is organic.
B. Determining what is organic will help you decide whether it would be a good idea to buy the soil you need. [substeps] There are two main kinds of organic soil: organic and non-organic.
C. Before shopping for an organic soil, inform yourself so that you know what to expect. Traces of chemicals can be common, especially in soils containing certain types of manure, but if the amount is low enough it could still qualify as organic.
D. [title] Buy organic potting soil instead of commercial ones. [step] Changing to commercial potting soil will reduce the amount of organic overgrowth in your field and reduce the amount of organic manure produced by your cultivars' buckers.
Rendered prompt
[header] How to buy organic soil [title] Do your research. [step] Truly organic soil can be hard to come by since the market for organic potting media has only begun increasing in the past few years. Moreover, organic standards have also become stricter as the demand for organic goods has grown, so a soil that was once deemed organic may no longer fulfill those requirements legally.
Model answer
C
Extracted: C · Gold: C
sample 3 · hellaswag:00008485pass · 100.0% · 4400ms · ef5ba648c703
Question
[header] How to burp a sleeping baby [title] Hold your baby and burp them. [step] This technique is good for babies who sleep on their stomach or who like to cuddle when they sleep. [substeps] Slowly move your baby next to your body so you do not wake them up.

A. Allow your baby's head or chin to rest on your shoulder and cup their bottom to support them so they don't slip as you hold them. Place your other hand on their back and gently pat it to help them burp.
B. You can also lower your baby's head and neck down into a deep belly position, which will help you burp your baby more easily. Gently nudge your baby at first and again until they are used to this position and begin to adjust to the feel of their neck and pillow.
C. Hold your baby close with your hand and record the amount of air they intake and how much you burp each time before you burp them. This way, you can do it for several days to see how comfortable they will be with your hand.
D. Then gently roll them over so they are lying on their back. Hold and burp your baby together until they become comfortable.
Rendered prompt
[header] How to burp a sleeping baby [title] Hold your baby and burp them. [step] This technique is good for babies who sleep on their stomach or who like to cuddle when they sleep. [substeps] Slowly move your baby next to your body so you do not wake them up.
Model answer
A
Extracted: A · Gold: A
sample 5 · hellaswag:00010031pass · 100.0% · 4399ms · 34999dc38706
Question
[header] How to rent a limousine [title] Decide when you'll rent the limo. [step] Early summer is the peak of the season for hiring limousines. You will probably be able to secure a better price if you hire a limo outside the months of may and june.

A. You're also likely to get a better rate on weekdays. [substeps] Sunday through thursday are typically the least expensive days, while fridays are about 20% less than saturdays.
B. If you're looking for one in july or august, you might want to be the driver. [title] Decide whether to ask friends, relatives, or strangers.
C. [substeps] Many limousine offers large limousine packages for lower prices. However, the driver hiring a limousine for the funeral may not be the first limo you hire.
D. A limousine is usually on the streets or across city traffic and less hectic between the ages of september and july. Saturday taxis are typically used if you live in an area where it is hot.
Rendered prompt
[header] How to rent a limousine [title] Decide when you'll rent the limo. [step] Early summer is the peak of the season for hiring limousines. You will probably be able to secure a better price if you hire a limo outside the months of may and june.
Model answer
A
Extracted: A · Gold: A
sample 6 · hellaswag:00005104pass · 100.0% · 4404ms · ea6b5cd4a03e
Question
[header] How to dip dye dark hair [title] Protect yourself and work area from bleach stains. [step] Cover your counter with newspaper. Wrap a dyeing cape or an old towel around your shoulders and put on a pair of plastic gloves.

A. Lastly, make sure that you have all of your supplies laid out and ready to go. [title] Mix the bleach according to the instructions on the package.
B. [substeps] Deep dye dark hair requires heavy amounts of bleach. For deep dye, i recommend having a shirt with dark sleeves.
C. Wear a sturdy t-shirt and apron to protect yourself while you work. Put bleach bowls or a bucket into your sink so that you can come to an additional solution that's the same color as the dye.
D. Bleach is caustic and it affects your hair and eyes. [substeps] You can also wear a bandanna around your neck.
Rendered prompt
[header] How to dip dye dark hair [title] Protect yourself and work area from bleach stains. [step] Cover your counter with newspaper. Wrap a dyeing cape or an old towel around your shoulders and put on a pair of plastic gloves.
Model answer
A
Extracted: A · Gold: A
sample 7 · hellaswag:00005596pass · 100.0% · 4627ms · fb6cc22c3d0a
Question
[header] How to hire syrian refugees [title] Find your local resettlement office. [step] Refugees admitted into a new country are eligible to work immediately. Government programs assist refugees in finding employment as part of their resettlement programs.

A. If you identify yourself as a refugee, you must work as a refugee in the country. [title] Complete any required occupations (such as a position, career, or order) before applying for a job.
B. [substeps] Use the job search tools at the resource website: https: // www.resettlement-cepus.gov/workspace/guild-acanees. Cfm look on your city or town's website for a list of resettlement department locations.
C. [substeps] In the united states, you can find your local resettlement office by visiting https: // www.acf.hhs.gov/orr/state-programs-annual-overview. Click on your state on the map, or select your state from the drop-down menu below the map to get a list of local offices and contact information.
D. The national guard service reacts positively to refugees seeking work as part of their resettlement program, as they provide a much more productive working environment. [substeps] After visiting your local resettlement office, locate the website by typing " resettlement program " and your city and state into a search engine.
Rendered prompt
[header] How to hire syrian refugees [title] Find your local resettlement office. [step] Refugees admitted into a new country are eligible to work immediately. Government programs assist refugees in finding employment as part of their resettlement programs.
Model answer
C
Extracted: C · Gold: C
Incorrect samples
sample 1 · hellaswag:00001623fail · 0.0% · 4403ms · a061d3a6bb7b
Question
A man has climbed a large ladder outside. he

A. is using trimmers to cut and trim large trees.
B. ropes and lures a cow into the open.
C. is using it to pull himself up onto a platform where a car is parked.
D. is life saving at the bottom of the ladder.
Rendered prompt
A man has climbed a large ladder outside. he
Model answer
C
Extracted: C · Gold: A
sample 4 · hellaswag:00005294fail · 0.0% · 4403ms · 7f5b2215e320
Question
[header] How to use a highlighter stick [title] Choose an ivory or cream highlighter shade if you have fair skin. [step] If you have a pale complexion, cream and ivory highlighters with a pearlescent or icy-silver tinge work really well. While these shades tend to look ghostly and unnatural on other complexions, they add a natural-looking glow to fair skin.

A. [substeps] Look for highlighter shades that include words like " moonbeam, " " ice, " and " crystalline. " avoid shades that are darker than cream and ivory.
B. [title] Try a metallic highlighter shade to go with your outfit. [step] If you have a pale complexion, you may prefer something metallic.
C. [substeps] Ivory highlighters are quite pricey. They cost anywhere from $14-20.
D. They make blending difficult to maintain, so get creative and test out different color highlighters. [substeps] For darker skin colors, choose a similarly color highlighter.
Rendered prompt
[header] How to use a highlighter stick [title] Choose an ivory or cream highlighter shade if you have fair skin. [step] If you have a pale complexion, cream and ivory highlighters with a pearlescent or icy-silver tinge work really well. While these shades tend to look ghostly and unnatural on other complexions, they add a natural-looking glow to fair skin.
Model answer
B
Extracted: B · Gold: A
sample 8 · hellaswag:00001187fail · 0.0% · 4406ms · 7d9da675c4e0
Question
The woman stands still for a short while in her blue/white outfit. Then she lifts the heavy yellow weight and someone who's watching her shows peremptory by yelling. she

A. picks it back up and lifts it again, guzzling the last of the shiraz.
B. drops the weight and the face of female trainer starts into the screen.
C. holds it up for a good second and then drops it back down to the ground.
D. slowly lifts the weight up she long the overweight weight over her head, then drops the weight to the ground.
Rendered prompt
The woman stands still for a short while in her blue/white outfit. Then she lifts the heavy yellow weight and someone who's watching her shows peremptory by yelling. she
Model answer
D
Extracted: D · Gold: C
sample 13 · hellaswag:00000389fail · 0.0% · 4406ms · 2802a406f49a
Question
A young man is seen playing drums in front of an audience. other people

A. are seen speaking to the camera as well as running into the audience.
B. are watching as he continues to play.
C. are walking by back and fourth.
D. are shown playing guitar while the man plays more drums.
Rendered prompt
A young man is seen playing drums in front of an audience. other people
Model answer
B
Extracted: B · Gold: D
sample 15 · hellaswag:00008268fail · 0.0% · 4398ms · d96daca04434
Question
[header] How to discreetly breastfeed your baby on the go [title] Wear nursing clothing. [step] Nursing tops can be purchased at maternity stores and most major clothing retailers. These tops have hidden openings that allow you to easily feed your baby.

A. The lining on these dresses is designed to allow you to access the breast once your baby has grown accustomed to you. To remove the lining, simply slide it off and then gently remove it.
B. [substeps] Wear one that covers the breasts, but does not cover them completely. Many maternity tops are available in mesh or sheer fabrics.
C. You can get a variety of items such as tops, bras, and dresses. [substeps] Nursing bras that are made like sports bras are usually better.
D. When your baby is traveling, these have tiny baby items that you can remove and show your baby. [substeps] You can also move to a place where you will not be bothered by baby items.
Rendered prompt
[header] How to discreetly breastfeed your baby on the go [title] Wear nursing clothing. [step] Nursing tops can be purchased at maternity stores and most major clothing retailers. These tops have hidden openings that allow you to easily feed your baby.
Model answer
A
Extracted: A · Gold: C
Qwen3.6-27B-NVFP4NVFP4unknown protocol · unknown agent
by Stewart_of_Mars · shard 12 · 7/1/2026, 1:17:01 PM · cmr23ny1404cvpi01mpgkrqsz
83.8%
311/371 correct · 5 correct traces · 5 incorrect traces
Correct samples
sample 1 · hellaswag:00007797pass · 100.0% · 3873ms · ca10ca786b1b
Question
[header] How to save money renting a car in the united states [title] Check online travel sites. [step] You can compare car rental prices online through most travel aggregators, such as orbitz, priceline, expedia, travelocity, and kayak. Car rental prices can fluctuate wildly, so check multiple sites regularly, starting at least several weeks before you need the car.

A. [substeps] If you combine your car rental with a flight and/or hotel booking, you can often save significantly more than by reserving them separately. [title] Compare prices directly from the major company websites.
B. [substeps] Paying online usually involves submitting a bill to the online car rental service. Check car websites and discuss customizable payment plans before finalizing the transaction.
C. [title] Sign up with a car sharing company. [step] Certain companies such as airtrain, lyft, lambordia, or banal are also carriers for car rental companies.
D. [title] Add up your driving and parking costs. [step] Be prepared to get a loan or borrow money from a friend or family member.
Rendered prompt
[header] How to save money renting a car in the united states [title] Check online travel sites. [step] You can compare car rental prices online through most travel aggregators, such as orbitz, priceline, expedia, travelocity, and kayak. Car rental prices can fluctuate wildly, so check multiple sites regularly, starting at least several weeks before you need the car.
Model answer
A
Extracted: A · Gold: A
sample 2 · hellaswag:00002368pass · 100.0% · 3884ms · e64052a5f5d3
Question
A woman is shown speaking to the camera while holding a dart. pictures of a man

A. 's arms are shown as well as this dart.
B. with a bowflex in one ear and a dart in her other ear next to a dart board in a room.
C. playing darts appear and he is shaking in distress before returning to talking to the camera.
D. are shown and leads into her throwing darts.
Rendered prompt
A woman is shown speaking to the camera while holding a dart. pictures of a man
Model answer
D
Extracted: D · Gold: D
sample 3 · hellaswag:00003525pass · 100.0% · 3888ms · 2637f8600bc9
Question
[header] How to wear a bomber jacket [title] Consider fit. [step] Because they started as a utilitarian piece for servicemen strapped with equipment, they're naturally oversized. However, designers haven't been afraid to play around with the bomber jacket design.

A. You've got tons of options-from thick american blazers to ¾-1-come-out bomber jackets. You just need to try them on and see if one fits you perfectly.
B. In fact, they've made many custom bomber jackets fit for military situations in the past. [substeps] Bomber jackets are wide-brimmed, functional, and tend to be less oversized.
C. You can wear them as an oversized piece, but you can also find them in slim fitting versions, cropped versions, and so on. [substeps] Getting the proper fit is key to rocking this look.
D. Try a simple button-down and it can work great. You can use a plain button-down instead if you want to wear some seasonal " steampunk " materials.
Rendered prompt
[header] How to wear a bomber jacket [title] Consider fit. [step] Because they started as a utilitarian piece for servicemen strapped with equipment, they're naturally oversized. However, designers haven't been afraid to play around with the bomber jacket design.
Model answer
C
Extracted: C · Gold: C
sample 5 · hellaswag:00003679pass · 100.0% · 3880ms · 1d6c41ac9865
Question
[header] How to clean hairbrushes [title] Remove the hair from the brush. [step] Taking out the old hair is always a good idea, since dirt and skin particles tend to stick to it as it builds up in your brush. Reach in and pinch the hair with your fingers, then pull it out from around the bristles.

A. [substeps] Brushes that are attached to anything (such as the brush) will be clean and free of dirt and bacteria, but brushes with metal can damage them and produce residue. [title] Choose one that is easy to clean.
B. [substeps] These small pinches are usually easy to remove or you can simply take them out. Removing used hair from an old hairbrush, just may take some effort to get it off of the handle.
C. Keep doing this until you've extracted all of the hair from the brush, then throw the hair away. [substeps] If your brush's bristles are made from natural fibers, take care not to yank them out with the hair.
D. [substeps] Most brushes have a teeth crown screw or other attachment. If not, put the brush in to remove the crown.
Rendered prompt
[header] How to clean hairbrushes [title] Remove the hair from the brush. [step] Taking out the old hair is always a good idea, since dirt and skin particles tend to stick to it as it builds up in your brush. Reach in and pinch the hair with your fingers, then pull it out from around the bristles.
Model answer
C
Extracted: C · Gold: C
sample 7 · hellaswag:00005557pass · 100.0% · 3870ms · fd9e38b8c08f
Question
[header] How to tie a doo rag [title] Place the durag on your head. [step] You can choose a color and style of durag that work for you. Many people prefer doo rag that are a stretchy material that can be seen through when stretched.

A. They look like long cargo pants or long jacket pieces. Your crotches will have to be about even or slightly larger than the durag's elastic.
B. This makes them more breathable when they're tied tightly around your head. [substeps] Line the center seam up with the center of your head for symmetry.
C. The durag should be loose enough so that you can loosely wrap the rag around your head. [substeps] Your durag may be hard to place with one hand or with two hands so you need to make sure that it will allow you to handle it.
D. Use a product like dawww.dahoo.com or zieforednwearer if you want a casual cut. Durag fabric is also often called sewn cloth and is woven with hemp or microfiber.
Rendered prompt
[header] How to tie a doo rag [title] Place the durag on your head. [step] You can choose a color and style of durag that work for you. Many people prefer doo rag that are a stretchy material that can be seen through when stretched.
Model answer
B
Extracted: B · Gold: B
Incorrect samples
sample 4 · hellaswag:00001285fail · 0.0% · 6764ms · 30d7069fbc34
Question
The players stand around talking to one another and one shoots a bow and arrow at a target. more people

A. watch while the owners wait.
B. are seen shooting the bows followed by interviewers speaking and team mates reacting.
C. walk by and the players talk to each other in the background.
D. come and go behind the referee who has just come on stage to take picture and record the game.
Rendered prompt
The players stand around talking to one another and one shoots a bow and arrow at a target. more people
Model answer
D
Extracted: D · Gold: B
sample 6 · hellaswag:00002729fail · 0.0% · 3886ms · 25a7613dfe9e
Question
A clip begins and it features people running and the words on the screen say they are novices who are training to run a marathon in tel aviv, israel. a man

A. is seen speaking in a temple.
B. is then shown running and the words say can be seen on the screen again saying he has some good news.
C. appears training them and different marathoners in training take turns being interviewed while they continue to switch back and forth to everyone going through the training.
D. is seen standing in front of a street and he begins to perform an act while holding the statue and flipping back and fourth.
Rendered prompt
A clip begins and it features people running and the words on the screen say they are novices who are training to run a marathon in tel aviv, israel. a man
Model answer
B
Extracted: B · Gold: C
sample 8 · hellaswag:00002795fail · 0.0% · 4311ms · dd25d87c8fb2
Question
A towel and glass are on a table next to a piece of foam and metal. A tool is being shown by a man holding it. the metal

A. is cooled into a liquid and garnishes the fencing material.
B. is separated from the foam and put over a bath tub before being pressed with sponge and rinsed off.
C. is used to sharpen a knife in front of him.
D. is being used on a piece of cloth, cloth, toothbrush and shirt.
Rendered prompt
A towel and glass are on a table next to a piece of foam and metal. A tool is being shown by a man holding it. the metal
Model answer
A
Extracted: A · Gold: C
sample 11 · hellaswag:00001467fail · 0.0% · 3883ms · 5b7790382de9
Question
A girl runs down a track in slow motion. she

A. is throwing a javelin over the bar.
B. smashes into a sand pit several times.
C. takes a huge jump, hurling herself over a bar.
D. jumps into a sand box and begins to play a game of hopscotch.
Rendered prompt
A girl runs down a track in slow motion. she
Model answer
A
Extracted: A · Gold: C
sample 12 · hellaswag:00006499fail · 0.0% · 3870ms · a30d372168a2
Question
[header] How to make your hair stay straight all day [title] Use shampoos and conditioners that help straighten hair. [step] Look for shampoos and conditioners that have lots of vitamins and minerals in them, giving your hair refreshing nutrients that will help smooth it out. These are usually marketed as smoothing, sleek, or straightening products.

A. [substeps] Look for a shampoo that lists " hydrating ingredients ", which are sodium and other all natural ingredients. You can also try going for one that specifically states " hydrating " on the bottle.
B. [substeps] If you have very curly hair, you may need to consider using a straightening shampoo. You can also purchase a hair-straightening powder and conditioners to help moisturize your hair.
C. If you have dry hair, look for shampoos that will keep it soft, wet, and shine. If you have hair that is frizzy, opt for sleek, volumizing, or foam shampoos.
D. [substeps] Look for ingredients such as wheat proteins, pro vitamin b5, or hedera helix extract. Take the time to comb the conditioner into your hair while in the shower, using your fingers or a wide-toothed comb.
Rendered prompt
[header] How to make your hair stay straight all day [title] Use shampoos and conditioners that help straighten hair. [step] Look for shampoos and conditioners that have lots of vitamins and minerals in them, giving your hair refreshing nutrients that will help smooth it out. These are usually marketed as smoothing, sleek, or straightening products.
Model answer
B
Extracted: B · Gold: D