{"question_id":"hellaswag:00000731","item_index":0,"attempt":0,"prompt_hash":"315b3bc53367a9d35e894e7178e16a6b0d082351fa12e46e514f862c3b115bf3","question":"Maria is demonstrating how to make a nyquil cocktail drink. she\n\nA. shows the ingredients required for the drink.\nB. pours it through a straw into a glass and washes it down with a cup of soda water.\nC. shows each scale of the cocktail by adding different colored liquids.\nD. pours the ice in a small glass and squirts lemon juice into the glass.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nMaria is demonstrating how to make a nyquil cocktail drink. she\n\nA. shows the ingredients required for the drink.\nB. pours it through a straw into a glass and washes it down with a cup of soda water.\nC. shows each scale of the cocktail by adding different colored liquids.\nD. pours the ice in a small glass and squirts lemon juice into the glass.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":8972,"error":null,"output":"A"}
{"question_id":"hellaswag:00006203","item_index":1,"attempt":0,"prompt_hash":"510fb1db449c17a776da6920d56778571a6fdced05a948f7d292b02f84e3a156","question":"[header] How to zone agricultural land for use as a wedding venue [title] Talk to your neighbors. [step] Your neighbors can be your strongest supporters or your most bitter rivals if you want to rezone your property. The best way to potentially get them on your side is to keep them informed of your plans every step of the way.\n\nA. [substeps] In the early stages, let them know about your plans and reassure them that you will keep them informed on developments. Give them the chance to ask questions, and do everything you can to assure them that events you host won't cause any disturbances to them or their property.\nB. [substeps] Make sure you know your neighbors' lives and can help them out if they disagree about your plan. Give them written correspondence so they can be helpful to you if needed.\nC. This means that if your neighbors are anything but informed, you should talk to them as well. If you don't know them well enough to keep detailed notes, reach out to them and ask them to help keep a record of your plans.\nD. [substeps] Don't forget that some of your family's acreage may need transplant rights. Always decide ahead of time what kind of liens will be put on what land you are planning on and why.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to zone agricultural land for use as a wedding venue [title] Talk to your neighbors. [step] Your neighbors can be your strongest supporters or your most bitter rivals if you want to rezone your property. The best way to potentially get them on your side is to keep them informed of your plans every step of the way.\n\nA. [substeps] In the early stages, let them know about your plans and reassure them that you will keep them informed on developments. Give them the chance to ask questions, and do everything you can to assure them that events you host won't cause any disturbances to them or their property.\nB. [substeps] Make sure you know your neighbors' lives and can help them out if they disagree about your plan. Give them written correspondence so they can be helpful to you if needed.\nC. This means that if your neighbors are anything but informed, you should talk to them as well. If you don't know them well enough to keep detailed notes, reach out to them and ask them to help keep a record of your plans.\nD. [substeps] Don't forget that some of your family's acreage may need transplant rights. Always decide ahead of time what kind of liens will be put on what land you are planning on and why.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1845,"error":null,"output":"A"}
{"question_id":"hellaswag:00002742","item_index":2,"attempt":0,"prompt_hash":"43625a13ba92496ef94b32cfffac3019361eb2405b3d21f5e49ba46780c2baba","question":". a man with a orange shirt and blue gloves\n\nA. is standing in a snow covered parking lot talking while holding a skateboard and talking.\nB. is doing gymnastics on an elliptical.\nC. is shown operating a fire torch machine.\nD. approaches a green motor vehicle that is in a pool.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n. a man with a orange shirt and blue gloves\n\nA. is standing in a snow covered parking lot talking while holding a skateboard and talking.\nB. is doing gymnastics on an elliptical.\nC. is shown operating a fire torch machine.\nD. approaches a green motor vehicle that is in a pool.","gold":"C","extracted_answer":"D","choices":null,"choice_scores":null,"score_normalization":null,"score":0,"passed":false,"latency_ms":712,"error":null,"output":"D"}
{"question_id":"hellaswag:00009360","item_index":3,"attempt":0,"prompt_hash":"772bf3a3a4f6a6927208ce2e4fa7c07cbbfee0ad799500801f4f1547d544e752","question":"[header] How to get ready for a party [title] Prepare anything you need to bring. [step] If the party is a potluck, you'll want to make and bring a dish to share. If the party is a birthday party or some kind of shower, you'll need to bring a gift.\n\nA. If you are throwing a birthday party or if your parents are fine with it, feel free to bring these items. [substeps] In a party, use sachets or strings to write things in.\nB. If the party is a dinner party, you should bring a bottle of wine or a hostess gift. At more casual parties in high school or college, it's generally expected that guests won't bring anything unless it's been specified on the invitation.\nC. You'll also want to bring a good notebook and a pen so you can write down ideas on how to best decorate the house. A classic approach would be to bring plastic utensils, such as wine glasses.\nD. [title] Know what you're expected to wear. [step] If you're just going to school, you might wear a dress, but if you're doing other things, such as going to a dance, you might wear some pajamas or something a little fancier like a dress.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to get ready for a party [title] Prepare anything you need to bring. [step] If the party is a potluck, you'll want to make and bring a dish to share. If the party is a birthday party or some kind of shower, you'll need to bring a gift.\n\nA. If you are throwing a birthday party or if your parents are fine with it, feel free to bring these items. [substeps] In a party, use sachets or strings to write things in.\nB. If the party is a dinner party, you should bring a bottle of wine or a hostess gift. At more casual parties in high school or college, it's generally expected that guests won't bring anything unless it's been specified on the invitation.\nC. You'll also want to bring a good notebook and a pen so you can write down ideas on how to best decorate the house. A classic approach would be to bring plastic utensils, such as wine glasses.\nD. [title] Know what you're expected to wear. [step] If you're just going to school, you might wear a dress, but if you're doing other things, such as going to a dance, you might wear some pajamas or something a little fancier like a dress.","gold":"B","extracted_answer":"B","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1865,"error":null,"output":"B"}
{"question_id":"hellaswag:00004111","item_index":4,"attempt":0,"prompt_hash":"804eac15909fbdc0b40566a12b0f5f85961ced020d50aac1bab1e3027f292eff","question":"[header] How to style mom jeans [title] Pick a pair of jeans that fit your waist well. [step] Mom jeans are meant to sit high on your waist. When picking out a new pair, try them on and see how the waist fits-you don't want the jeans too tight or too loose.\n\nA. Try on any pair and see if it's even snug. [substeps] If your mom jeans tend to dig into your legs, look for jeans that come up above the knee.\nB. [substeps] Pick out jeans that are at the leg length of your legs. If you have shorter legs, opt for what looks best to you, rather than what might look nice with longer pants.\nC. [substeps] The high waist will show off your waist and elongate your legs. [title] Choose a leg style that's appropriate for your figure.\nD. Try buying pairs that are so low that you can still walk in them but they are a little looser. [substeps] Ask your mom about how to size your jeans.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to style mom jeans [title] Pick a pair of jeans that fit your waist well. [step] Mom jeans are meant to sit high on your waist. When picking out a new pair, try them on and see how the waist fits-you don't want the jeans too tight or too loose.\n\nA. Try on any pair and see if it's even snug. [substeps] If your mom jeans tend to dig into your legs, look for jeans that come up above the knee.\nB. [substeps] Pick out jeans that are at the leg length of your legs. If you have shorter legs, opt for what looks best to you, rather than what might look nice with longer pants.\nC. [substeps] The high waist will show off your waist and elongate your legs. [title] Choose a leg style that's appropriate for your figure.\nD. Try buying pairs that are so low that you can still walk in them but they are a little looser. [substeps] Ask your mom about how to size your jeans.","gold":"C","extracted_answer":"C","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1646,"error":null,"output":"C"}
{"question_id":"hellaswag:00009418","item_index":5,"attempt":0,"prompt_hash":"80921f961a3699061838a59107bfa1e79ffce3ad31d9fb275a8246e438123299","question":"[header] How to diagnose heel spurs [title] Locate the pain. [step] Heel spurs can show up in multiple places on your heel. This can cause the pain to be slightly different depending on exactly where the heel spur is.\n\nA. It can be hard to judge the strength of the spur and whether or not it is painful at all. When you just try to be more specific about where it is, your foot and heel will become increasingly entangled and there will be no wiggle room.\nB. They can be located at the back of the heel or under the heel, near the sole of your foot. If you are experiencing pain along the back of your foot, up through your ankle, you might have a heel spur on the back of your heel.\nC. The first position you identify is the thigh. You will see white or pinkish swelling and inflammation along the arch of the heel.\nD. The most common places you might see the sensation of the pain are your heels and your nail. [substeps] You may discover that the pain is more acute in one heel then it is in another.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to diagnose heel spurs [title] Locate the pain. [step] Heel spurs can show up in multiple places on your heel. This can cause the pain to be slightly different depending on exactly where the heel spur is.\n\nA. It can be hard to judge the strength of the spur and whether or not it is painful at all. When you just try to be more specific about where it is, your foot and heel will become increasingly entangled and there will be no wiggle room.\nB. They can be located at the back of the heel or under the heel, near the sole of your foot. If you are experiencing pain along the back of your foot, up through your ankle, you might have a heel spur on the back of your heel.\nC. The first position you identify is the thigh. You will see white or pinkish swelling and inflammation along the arch of the heel.\nD. The most common places you might see the sensation of the pain are your heels and your nail. [substeps] You may discover that the pain is more acute in one heel then it is in another.","gold":"B","extracted_answer":"B","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1597,"error":null,"output":"B"}
{"question_id":"hellaswag:00004876","item_index":6,"attempt":0,"prompt_hash":"d149686f1c12ffdc1a528922300a1e30d3b235542fbc8a3a935f00d6a7a5863d","question":"[header] How to dye deviled eggs [title] Hard boil the eggs. [step] Place 7 eggs into a saucepan, then cover them with 1 inch (2.5 cm) of water. Add a pinch of salt, then boil them, covered, for 10 minutes.\n\nA. This will help the eggs separate from the dye and flavor them further. [substeps] If the eggs take longer than 10 minutes to boil, wait until they're cool, then run them under cool tap water.\nB. Turn the heat off, and let the eggs sit in the saucepan for another 15 minutes; don't remove the lid. [substeps] If you have a different method for hard boiling eggs that you prefer to you, go ahead and use it.\nC. [substeps] You can use any type of seafood for this recipe, including fish, shrimp, bird, breast, and octopus. If your recipe calls for steamed eggs, you will need to cook on a low heat.\nD. Drain the eggs with a strainer and discard them. [title] Smash the egg shells with a potato masher.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to dye deviled eggs [title] Hard boil the eggs. [step] Place 7 eggs into a saucepan, then cover them with 1 inch (2.5 cm) of water. Add a pinch of salt, then boil them, covered, for 10 minutes.\n\nA. This will help the eggs separate from the dye and flavor them further. [substeps] If the eggs take longer than 10 minutes to boil, wait until they're cool, then run them under cool tap water.\nB. Turn the heat off, and let the eggs sit in the saucepan for another 15 minutes; don't remove the lid. [substeps] If you have a different method for hard boiling eggs that you prefer to you, go ahead and use it.\nC. [substeps] You can use any type of seafood for this recipe, including fish, shrimp, bird, breast, and octopus. If your recipe calls for steamed eggs, you will need to cook on a low heat.\nD. Drain the eggs with a strainer and discard them. [title] Smash the egg shells with a potato masher.","gold":"B","extracted_answer":"B","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1795,"error":null,"output":"B"}
{"question_id":"hellaswag:00009212","item_index":7,"attempt":0,"prompt_hash":"b6bcc300e707b15c2e579c151cc25257001c8bf3a2ca616794cc497d38d2a05d","question":"[header] How to maintain your baby's curly hair [title] Use a gentle, sulfate-free baby shampoo only once or twice per week. [step] Over-washing your baby's hair can cause dryness, resulting in dull curls. When you do use shampoo, avoid rubbing or tangling the curls.\n\nA. Massage the scalp gently to wash the roots only, avoiding the ends. [substeps] Curly hair has less moisture than straight hair, so you'll want to preserve the natural oils in the hair by using shampoo sparingly.\nB. Your baby's hair will end up with tangles and curl marks in it. [title] Dry your baby's hair with a clean towel.\nC. [substeps] Additionally, make sure your baby has a moisturized hair regime such as a straightening shampoo. This will help maintain a smooth, curly hair.\nD. If your hair is already a curl long, you can just squeeze the shampoo out of it. [title] Avoid using drying sprays, sizzles, or heated water as much as possible.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to maintain your baby's curly hair [title] Use a gentle, sulfate-free baby shampoo only once or twice per week. [step] Over-washing your baby's hair can cause dryness, resulting in dull curls. When you do use shampoo, avoid rubbing or tangling the curls.\n\nA. Massage the scalp gently to wash the roots only, avoiding the ends. [substeps] Curly hair has less moisture than straight hair, so you'll want to preserve the natural oils in the hair by using shampoo sparingly.\nB. Your baby's hair will end up with tangles and curl marks in it. [title] Dry your baby's hair with a clean towel.\nC. [substeps] Additionally, make sure your baby has a moisturized hair regime such as a straightening shampoo. This will help maintain a smooth, curly hair.\nD. If your hair is already a curl long, you can just squeeze the shampoo out of it. [title] Avoid using drying sprays, sizzles, or heated water as much as possible.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1589,"error":null,"output":"A"}
{"question_id":"hellaswag:00000155","item_index":8,"attempt":0,"prompt_hash":"165c9772335a366801f1bcc4dad3e247411eb7b64fe68a4eed50b6389885ba7b","question":"The harmonica is held up and shown. a person\n\nA. holds a harmonica between his hands a plays a song.\nB. is flying through the sky.\nC. runs up and grabs the guitar, then begins strumming.\nD. is strumming an electric guitar.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nThe harmonica is held up and shown. a person\n\nA. holds a harmonica between his hands a plays a song.\nB. is flying through the sky.\nC. runs up and grabs the guitar, then begins strumming.\nD. is strumming an electric guitar.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":697,"error":null,"output":"A"}
{"question_id":"hellaswag:00002117","item_index":9,"attempt":0,"prompt_hash":"c3af8bbc4450b92241174425417df40a3cda399bb001ffa6caf4b7c8ddbdd9ad","question":"The woman points to a water splash on the counter next to the sink after turning on the water at the sink. the woman\n\nA. takes out a spray bottle and a contact lens.\nB. talks to the camera and washes her hands before rinsing her hands.\nC. turns on the water with a giant sponge, grabs soap and adds some drops of the water.\nD. then grabs a paper towel and wipes down the sink counter, the inside of the sink and the faucet spout.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nThe woman points to a water splash on the counter next to the sink after turning on the water at the sink. the woman\n\nA. takes out a spray bottle and a contact lens.\nB. talks to the camera and washes her hands before rinsing her hands.\nC. turns on the water with a giant sponge, grabs soap and adds some drops of the water.\nD. then grabs a paper towel and wipes down the sink counter, the inside of the sink and the faucet spout.","gold":"D","extracted_answer":"D","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":911,"error":null,"output":"D"}
{"question_id":"hellaswag:00000009","item_index":10,"attempt":0,"prompt_hash":"e1841a3f7d3c6bf99336c317fc8eacc5fbec6c4ffb6b49681ea6a4f2af31f4e7","question":"Then he takes a small stone from the flowing river and smashes it on another stone. He starts to crush the small stone to smaller pieces. he\n\nA. cuts the center stone in half and blow it on to make it bigger.\nB. grind it hard to make the pieces smaller.\nC. eventually brings it back into view and adds it to the smaller ones to make a small triangular shaped piece.\nD. starts to party with them and throw the pieces by hand while they celebrate.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nThen he takes a small stone from the flowing river and smashes it on another stone. He starts to crush the small stone to smaller pieces. he\n\nA. cuts the center stone in half and blow it on to make it bigger.\nB. grind it hard to make the pieces smaller.\nC. eventually brings it back into view and adds it to the smaller ones to make a small triangular shaped piece.\nD. starts to party with them and throw the pieces by hand while they celebrate.","gold":"B","extracted_answer":"B","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":911,"error":null,"output":"B"}
{"question_id":"hellaswag:00006104","item_index":11,"attempt":0,"prompt_hash":"197e084176dccddd28f5bfb03cb84e5e2df5b27ae98323599b34c85d6299e95d","question":"[header] How to make your own wig out of weave [title] Maintain your hair. [step] Depending on your hair type, you may need special products to condition and treat your hair. From natural oils like coconut oil to specialty products like hair mayonnaise and a protein pack, use the right combinations that work to keep your hair healthy.\n\nA. Always clean your hair before going through your stylist or straightening it. He or she should be able to tell you what styles and tonings work best for you.\nB. A light amount of conditioner or oil is fine as long as you use it regularly. [substeps] Under-washing doesn't provide moisturizer as you don't want your hair to wear down and the texture of your wig may change.\nC. [substeps] Curly hair becomes spikier and will return more easily in the first 48 hours. Loose or wavy hair may need trims or mousse to keep it on and out of the way.\nD. [substeps] Consult a stylist or haircare specialist to find what works best for you. There are natural essential oils and mixtures that can keep your hair moisturized like, lavender and rosemary if you are concerned with using a chemical formula.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to make your own wig out of weave [title] Maintain your hair. [step] Depending on your hair type, you may need special products to condition and treat your hair. From natural oils like coconut oil to specialty products like hair mayonnaise and a protein pack, use the right combinations that work to keep your hair healthy.\n\nA. Always clean your hair before going through your stylist or straightening it. He or she should be able to tell you what styles and tonings work best for you.\nB. A light amount of conditioner or oil is fine as long as you use it regularly. [substeps] Under-washing doesn't provide moisturizer as you don't want your hair to wear down and the texture of your wig may change.\nC. [substeps] Curly hair becomes spikier and will return more easily in the first 48 hours. Loose or wavy hair may need trims or mousse to keep it on and out of the way.\nD. [substeps] Consult a stylist or haircare specialist to find what works best for you. There are natural essential oils and mixtures that can keep your hair moisturized like, lavender and rosemary if you are concerned with using a chemical formula.","gold":"D","extracted_answer":"D","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1926,"error":null,"output":"D"}
{"question_id":"hellaswag:00004337","item_index":12,"attempt":0,"prompt_hash":"33942341c4ffd600065be529b1cca7f9f228214b1dae43c6740ed32c993eb32a","question":"[header] How to advertise for wedding photography [title] Purchase a small newspaper advertisement in your local paper. [substeps] Start out by placing a print ad in your town's sunday paper. Newspaper advertisement returns have been on the decline for some time, but ad space can be relatively inexpensive depending on the paper.\n\nA. If you're looking for outdoor photography, try local paintshops or area recreation areas. They also have online sites to let you advertise on a regular basis, such as forums.\nB. Increase your ad space if you get a number of calls after your first attempt. This larger ad should provide an exemplary photograph and a range of contact information.\nC. Local newspapers are also a great place to post a brief ad--whatever type of photo the print advertisement contains--for your most popular wedding photography locations. [title] Advertise your photography in local newspapers.\nD. Customers walking by today are often drawn to newspapers where images are sold relatively cheaply. If you're a reviewer of the ads on the classified sections of your newspaper, sometimes the word \" wedding \" stands out.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to advertise for wedding photography [title] Purchase a small newspaper advertisement in your local paper. [substeps] Start out by placing a print ad in your town's sunday paper. Newspaper advertisement returns have been on the decline for some time, but ad space can be relatively inexpensive depending on the paper.\n\nA. If you're looking for outdoor photography, try local paintshops or area recreation areas. They also have online sites to let you advertise on a regular basis, such as forums.\nB. Increase your ad space if you get a number of calls after your first attempt. This larger ad should provide an exemplary photograph and a range of contact information.\nC. Local newspapers are also a great place to post a brief ad--whatever type of photo the print advertisement contains--for your most popular wedding photography locations. [title] Advertise your photography in local newspapers.\nD. Customers walking by today are often drawn to newspapers where images are sold relatively cheaply. If you're a reviewer of the ads on the classified sections of your newspaper, sometimes the word \" wedding \" stands out.","gold":"B","extracted_answer":"B","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1634,"error":null,"output":"B"}
{"question_id":"hellaswag:00003466","item_index":13,"attempt":0,"prompt_hash":"dcfbef80cafc22c03088cda59b9392468334c979ab054eb4dfa87b96c1ca3eed","question":"[header] How to use ableton live [title] Install ableton live on your computer. [step] Use the provided disc or other means to install the program and get it fully functional on your computer and operating system. [substeps] Look at all system requirements for ableton live.\n\nA. Like many other modern sound programs, this program may include requirements for an advanced sound card, graphics display card or other piece of hardware, as well as requirements for minimum memory or processing speed. Make sure that your computer has what it takes to effectively run ableton live.\nB. Most operating systems require that you access basic operating systems, but some systems, such as windows or macos, require the installation of the software on a physical cd or dvd through a system drive. This method can take up to two months.\nC. That's just the nature of the program: it should also have 1 or more power requirements. Are you operating the program from a desktop computer? If so, take your computer to the computer store as this will determine software needs and install requirements.\nD. Identifying the relevant expiration date on your system can save time and expense. [title] Access the system application to which you need to install ableton live.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to use ableton live [title] Install ableton live on your computer. [step] Use the provided disc or other means to install the program and get it fully functional on your computer and operating system. [substeps] Look at all system requirements for ableton live.\n\nA. Like many other modern sound programs, this program may include requirements for an advanced sound card, graphics display card or other piece of hardware, as well as requirements for minimum memory or processing speed. Make sure that your computer has what it takes to effectively run ableton live.\nB. Most operating systems require that you access basic operating systems, but some systems, such as windows or macos, require the installation of the software on a physical cd or dvd through a system drive. This method can take up to two months.\nC. That's just the nature of the program: it should also have 1 or more power requirements. Are you operating the program from a desktop computer? If so, take your computer to the computer store as this will determine software needs and install requirements.\nD. Identifying the relevant expiration date on your system can save time and expense. [title] Access the system application to which you need to install ableton live.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1930,"error":null,"output":"A"}
{"question_id":"hellaswag:00000147","item_index":14,"attempt":0,"prompt_hash":"b3d8baba7fc8282238d2ec2cdaa7a81be4c3164edfe70c99e0d6a3b0d63676a3","question":"People are standing on a hill of snow. People are snowboarding down a hill of snow. people\n\nA. are pouring snowballs into tubes as they go down the hill.\nB. are skiing down a roof of a house.\nC. are skiing down a hill.\nD. are standing on a hill holding stick.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nPeople are standing on a hill of snow. People are snowboarding down a hill of snow. people\n\nA. are pouring snowballs into tubes as they go down the hill.\nB. are skiing down a roof of a house.\nC. are skiing down a hill.\nD. are standing on a hill holding stick.","gold":"C","extracted_answer":"C","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":731,"error":null,"output":"C"}
{"question_id":"hellaswag:00001356","item_index":15,"attempt":0,"prompt_hash":"97335ae649b0f4648b6b1828f4103bb79ddc0eff3c7eb2590b19247796f0e85c","question":"A lady licks an ice cream cone and holds a baby. The lady takes the pacifier from the baby and lets the baby eat ice cream. the baby\n\nA. eats pieces of the ice cream and mixes it in a bowl.\nB. makes a face and stare at the camera as she eats.\nC. isn't happy but he feeds it from the ice cream.\nD. is getting more ice cream, and the lady keeps licking his face.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nA lady licks an ice cream cone and holds a baby. The lady takes the pacifier from the baby and lets the baby eat ice cream. the baby\n\nA. eats pieces of the ice cream and mixes it in a bowl.\nB. makes a face and stare at the camera as she eats.\nC. isn't happy but he feeds it from the ice cream.\nD. is getting more ice cream, and the lady keeps licking his face.","gold":"B","extracted_answer":"B","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":842,"error":null,"output":"B"}
{"question_id":"hellaswag:00005743","item_index":16,"attempt":0,"prompt_hash":"0b1a27f0023914786c864e8de829dc9ece91e1de49c768d1885cb5b85dc1f935","question":"[header] How to melt hershey kisses [title] Unwrap the kisses and place them in a heat-safe bowl. [step] Hershey kisses come wrapped individually in thin foil. The standard milk chocolate versions, which are the best for melting, are wrapped in plain silver foil.\n\nA. Use store-bought, homemade kisses if you wish. [substeps] If you like homemade ones, wrap them in foil, though you may wish to crush the chocolate.\nB. [title] Fill a microwave-safe bowl with 4 ounces (200 ml) of water and 1 ounce (15 ml) of sugar. [step] You can microwave the dipped kisses for just a few seconds for extra sweetness.\nC. Warm the kisses gently in the microwave at least 30 seconds before opening and unfolding them or they may melt before melting. When you open and reunwrap your kisses, the chocolate will liquefy and melt.\nD. Unwrap each one and place them into a heat-safe bowl (like glass). If you aren't sure, check your bowl for a \" heat-safe \" label.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to melt hershey kisses [title] Unwrap the kisses and place them in a heat-safe bowl. [step] Hershey kisses come wrapped individually in thin foil. The standard milk chocolate versions, which are the best for melting, are wrapped in plain silver foil.\n\nA. Use store-bought, homemade kisses if you wish. [substeps] If you like homemade ones, wrap them in foil, though you may wish to crush the chocolate.\nB. [title] Fill a microwave-safe bowl with 4 ounces (200 ml) of water and 1 ounce (15 ml) of sugar. [step] You can microwave the dipped kisses for just a few seconds for extra sweetness.\nC. Warm the kisses gently in the microwave at least 30 seconds before opening and unfolding them or they may melt before melting. When you open and reunwrap your kisses, the chocolate will liquefy and melt.\nD. Unwrap each one and place them into a heat-safe bowl (like glass). If you aren't sure, check your bowl for a \" heat-safe \" label.","gold":"D","extracted_answer":"D","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1744,"error":null,"output":"D"}
{"question_id":"hellaswag:00002056","item_index":17,"attempt":0,"prompt_hash":"81e682a6e26a6f53eb1fa6c45078b09f79d0356f8994402bd1af53accd080534","question":"Another person moves with the camera as they watch from a window in an elevated room against a white window sill. the person blowing the leaves\n\nA. gets closer to the camera as well as the person in the street behind him.\nB. gives a final look like he is talking and ends with him dropping the leaves as soon as he makes it to the ground.\nC. leaves are shown from a small angle.\nD. moves around the yard and continues to do lawn maintenance.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nAnother person moves with the camera as they watch from a window in an elevated room against a white window sill. the person blowing the leaves\n\nA. gets closer to the camera as well as the person in the street behind him.\nB. gives a final look like he is talking and ends with him dropping the leaves as soon as he makes it to the ground.\nC. leaves are shown from a small angle.\nD. moves around the yard and continues to do lawn maintenance.","gold":"D","extracted_answer":"B","choices":null,"choice_scores":null,"score_normalization":null,"score":0,"passed":false,"latency_ms":938,"error":null,"output":"B"}
{"question_id":"hellaswag:00005641","item_index":18,"attempt":0,"prompt_hash":"dc01383e99587b85b56a776785b4275141fedca1813f34f14d83b4fa23905bfe","question":"[header] How to go outside naked [title] Get comfortable with your body. [step] Most people aren't used to seeing themselves naked. In order to work up the courage to go uncovered outdoors, you first have to be confident in the way you look and feel.\n\nA. Additionally, you'll be aware of the physical aspect of being naked, so you can make an educated decision in the way that your skin feels. [substeps] Being naked isn't a punishment.\nB. When you are out naked, you will be known to be an alpha and you will show others that you can be just as confident. You will also be attracted to your surroundings because of this confidence.\nC. Make time for a few hours each day to watch yourself naked. [substeps] Even the most confident people always get naked after a workout or before bed.\nD. Take some time to get reacquainted with yourself in your most natural state, without judgment or self-consciousness. [substeps] Learn to love and accept your body.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to go outside naked [title] Get comfortable with your body. [step] Most people aren't used to seeing themselves naked. In order to work up the courage to go uncovered outdoors, you first have to be confident in the way you look and feel.\n\nA. Additionally, you'll be aware of the physical aspect of being naked, so you can make an educated decision in the way that your skin feels. [substeps] Being naked isn't a punishment.\nB. When you are out naked, you will be known to be an alpha and you will show others that you can be just as confident. You will also be attracted to your surroundings because of this confidence.\nC. Make time for a few hours each day to watch yourself naked. [substeps] Even the most confident people always get naked after a workout or before bed.\nD. Take some time to get reacquainted with yourself in your most natural state, without judgment or self-consciousness. [substeps] Learn to love and accept your body.","gold":"D","extracted_answer":"D","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1674,"error":null,"output":"D"}
{"question_id":"hellaswag:00004314","item_index":19,"attempt":0,"prompt_hash":"f425a28a7aaca14c58953c663d60eab2dec528d120d5dc18153352aa9d391125","question":"[header] How to get over a girl that left you for another guy (teens ) [title] Take her off the pedestal. [step] Best girl in the world? Never met, and never will meet another like her? It's common to feel this way, but trust us, it probably is not true. So let your friends talk trash about her.\n\nA. Ask your friends what went wrong, and vent the frustration. We love you for who you are, and there's no one better than you.\nB. There are a lot of teens who would at least look at where she is on a radar to make sure she hasn't left. Start with your friends, if you have the ball.\nC. Remind yourself of the annoying things she did. Do what you need to do to remind yourself that nobody's perfect, especially her.\nD. To get over her, you need to be prepared for the worst, because she is the one. Sometimes all you can do is make your own mistake.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to get over a girl that left you for another guy (teens ) [title] Take her off the pedestal. [step] Best girl in the world? Never met, and never will meet another like her? It's common to feel this way, but trust us, it probably is not true. So let your friends talk trash about her.\n\nA. Ask your friends what went wrong, and vent the frustration. We love you for who you are, and there's no one better than you.\nB. There are a lot of teens who would at least look at where she is on a radar to make sure she hasn't left. Start with your friends, if you have the ball.\nC. Remind yourself of the annoying things she did. Do what you need to do to remind yourself that nobody's perfect, especially her.\nD. To get over her, you need to be prepared for the worst, because she is the one. Sometimes all you can do is make your own mistake.","gold":"C","extracted_answer":"C","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1663,"error":null,"output":"C"}
{"question_id":"hellaswag:00009328","item_index":20,"attempt":0,"prompt_hash":"9d376828ee8867598f86f563356d2b67da65c33465e5b44ab96b7aba447c32de","question":"[header] How to make a rope braid with only two strands [title] Brush your hair. [step] Your hair should be smooth and free of tangles when you begin to make the rope braid. Use a brush or comb to thoroughly brush any knots or tangles out of your hair.\n\nA. You can use a moisturizing cream to smooth your hair out if it is not complying. [substeps] You can also dampen your hair with water to make it easier to put back in a ponytail and braid.\nB. If you have fine, fine hair, you can skip the brush or comb and just use the palm of your hand to tousle your hair into the braid. [title] Wrap a piece of cord in a cheesecloth.\nC. Your hair should also be relaxed and even. [title] Tie a few strands of hair in two bows.\nD. [substeps] Use a texturizing hairspray or mousse, which has been approved by the american academy of hairstylists. To make the rope braid, rub a water conditioner into your hair and smooth it down using a plastic comb.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to make a rope braid with only two strands [title] Brush your hair. [step] Your hair should be smooth and free of tangles when you begin to make the rope braid. Use a brush or comb to thoroughly brush any knots or tangles out of your hair.\n\nA. You can use a moisturizing cream to smooth your hair out if it is not complying. [substeps] You can also dampen your hair with water to make it easier to put back in a ponytail and braid.\nB. If you have fine, fine hair, you can skip the brush or comb and just use the palm of your hand to tousle your hair into the braid. [title] Wrap a piece of cord in a cheesecloth.\nC. Your hair should also be relaxed and even. [title] Tie a few strands of hair in two bows.\nD. [substeps] Use a texturizing hairspray or mousse, which has been approved by the american academy of hairstylists. To make the rope braid, rub a water conditioner into your hair and smooth it down using a plastic comb.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1781,"error":null,"output":"A"}
{"question_id":"hellaswag:00004965","item_index":21,"attempt":0,"prompt_hash":"806d1b4e379442ec73857acf4ee96d1089a66e17832a74771add3545c13a8747","question":"[header] How to report vandalism [title] Call your local emergency services to report active vandalism. [step] Do this whether the vandalism is being committed against public or private property. If you catch someone in the act of spray painting a building, stealing a street sign, or otherwise destroying or vandalizing property, call your local emergency services number immediately.\n\nA. [title] Report the vandalism by reporting the vandalism. [step] This method will require no explanation of the vandalism.\nB. The us department of buildings (ftc) has a map at https: // www.ftc.gov/free-contractor/disputes /. [substeps] If you see an officer or official police report vandalism, don't get upset or panic.\nC. Destruction of property is illegal, and the authorities will want to catch the person committing the crime if at all possible. [substeps] In the united states you should call 9-1-1 to report vandalism in progress.\nD. [substeps] A 911 call is a fast, easy way to report vandalism. If you have been on the scene for a year or more, it may be worth calling your local report center.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to report vandalism [title] Call your local emergency services to report active vandalism. [step] Do this whether the vandalism is being committed against public or private property. If you catch someone in the act of spray painting a building, stealing a street sign, or otherwise destroying or vandalizing property, call your local emergency services number immediately.\n\nA. [title] Report the vandalism by reporting the vandalism. [step] This method will require no explanation of the vandalism.\nB. The us department of buildings (ftc) has a map at https: // www.ftc.gov/free-contractor/disputes /. [substeps] If you see an officer or official police report vandalism, don't get upset or panic.\nC. Destruction of property is illegal, and the authorities will want to catch the person committing the crime if at all possible. [substeps] In the united states you should call 9-1-1 to report vandalism in progress.\nD. [substeps] A 911 call is a fast, easy way to report vandalism. If you have been on the scene for a year or more, it may be worth calling your local report center.","gold":"C","extracted_answer":"C","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":2167,"error":null,"output":"C"}
{"question_id":"hellaswag:00003513","item_index":22,"attempt":0,"prompt_hash":"3ec0150cf69c497c651e2c98de316b7256d6dc2c4c1a6b602107f99a044c7fbf","question":"[header] How to cope with your parents' custody battle (for kids ) [title] Recognize that it isn't your fault that your parents are having problems. [step] You are not the cause. They may not love each other anymore, but they love you, and that's why they are in this custody battle.\n\nA. No matter what you think, your parents are fighting to uphold their rights and responsibilities. [title] Understand that actions can affect your consequences.\nB. If you are in your parents' custody battle, your parents are not the reason why they are fighting. [substeps] They are fighting because they are angry with you.\nC. Your friends could provide you a little moral support and that could be helpful. [title] Your parents feelings are not your responsibility.\nD. However, appreciate that by recognizing what your parents are arguing about, you know that your perspective is the only way you can do better in this situation. Try your best to understand their perspective so that you have a better chance of changing your mind about custody.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to cope with your parents' custody battle (for kids ) [title] Recognize that it isn't your fault that your parents are having problems. [step] You are not the cause. They may not love each other anymore, but they love you, and that's why they are in this custody battle.\n\nA. No matter what you think, your parents are fighting to uphold their rights and responsibilities. [title] Understand that actions can affect your consequences.\nB. If you are in your parents' custody battle, your parents are not the reason why they are fighting. [substeps] They are fighting because they are angry with you.\nC. Your friends could provide you a little moral support and that could be helpful. [title] Your parents feelings are not your responsibility.\nD. However, appreciate that by recognizing what your parents are arguing about, you know that your perspective is the only way you can do better in this situation. Try your best to understand their perspective so that you have a better chance of changing your mind about custody.","gold":"C","extracted_answer":"B","choices":null,"choice_scores":null,"score_normalization":null,"score":0,"passed":false,"latency_ms":1739,"error":null,"output":"B"}
{"question_id":"hellaswag:00002538","item_index":23,"attempt":0,"prompt_hash":"64330a5261d73854b6f54b1b08f961d90d78679e9fac79b4cb437bbf354c1401","question":"Various pictures of javelin athletes are shown that lead into videos of the athletes throwing the pole. a man\n\nA. then steps up and demonstrates how to properly throw the pole while bouncing up and down.\nB. talks to the camera while holding a metal ball and talking to the camera and other athletes.\nC. is watching the entire performance while people are playing the game.\nD. is then shown attempting hand stands on the rope and runs to another location and is shown again over and over throwing the pole.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nVarious pictures of javelin athletes are shown that lead into videos of the athletes throwing the pole. a man\n\nA. then steps up and demonstrates how to properly throw the pole while bouncing up and down.\nB. talks to the camera while holding a metal ball and talking to the camera and other athletes.\nC. is watching the entire performance while people are playing the game.\nD. is then shown attempting hand stands on the rope and runs to another location and is shown again over and over throwing the pole.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":932,"error":null,"output":"A"}
{"question_id":"hellaswag:00009494","item_index":24,"attempt":0,"prompt_hash":"60faec525c6ba17a6bbd95b8d30b36b49b512ba5c1a5d308f21c8d6b8f81e3b1","question":"[header] How to apply eyeshadow that lasts [title] Wash your face. [step] Starting with a clean and fresh face will make your makeup stand out more, help it last longer, and help prevent your makeup from clumping and flaking. Use your favorite gel or cream facial cleanser and lukewarm water.\n\nA. Rub the cleanser in circular motions for 20 minutes before rinsing with warm water. [title] Apply your eyeliner to the crease of your eye.\nB. [substeps] Wet your face and pour some cleanser into your hands. Work the cleanser into a lather, and then gently massage it into your face for about 30 seconds using circular motions.\nC. Tap your face with your fingers to get the excess off, then pat your face dry with a clean towel. [substeps] Do not splash water onto your face.\nD. Pat your face dry with a clean towel. [substeps] An eyeshadow primer also works just as well.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to apply eyeshadow that lasts [title] Wash your face. [step] Starting with a clean and fresh face will make your makeup stand out more, help it last longer, and help prevent your makeup from clumping and flaking. Use your favorite gel or cream facial cleanser and lukewarm water.\n\nA. Rub the cleanser in circular motions for 20 minutes before rinsing with warm water. [title] Apply your eyeliner to the crease of your eye.\nB. [substeps] Wet your face and pour some cleanser into your hands. Work the cleanser into a lather, and then gently massage it into your face for about 30 seconds using circular motions.\nC. Tap your face with your fingers to get the excess off, then pat your face dry with a clean towel. [substeps] Do not splash water onto your face.\nD. Pat your face dry with a clean towel. [substeps] An eyeshadow primer also works just as well.","gold":"B","extracted_answer":"D","choices":null,"choice_scores":null,"score_normalization":null,"score":0,"passed":false,"latency_ms":1739,"error":null,"output":"D"}
{"question_id":"hellaswag:00004289","item_index":25,"attempt":0,"prompt_hash":"5cd2545060b43084fd3d586e5ed2e37794a2a6f4f10b79183f705c212e017779","question":"[header] How to make a louisiana seafood boil [title] Cut the head off your shrimp with a sharp knife. [title] Devein the shrimp. [step] Your shrimp will have a natural arch to it so cut along the middle of the inner arch up to the tip of the tail where the intestinal tract is.\n\nA. [title] Place the shrimp in boiling water for 3-5 minutes to loosen them. [title] Place the shrimp into a pot of boiling water.\nB. From the cut, a black line (the shrimp's vein) will become visible. Remove the vein with your hands or a fork and wash the shrimp under running water.\nC. Don't cut on the gills yet. [title] Separate the insides of the shrimp by grasping the fleshy side with fingers that are pointing away from you.\nD. [title] Carefully cut out the intestines. [step] You may have to hack at the shrimp as you cut it.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to make a louisiana seafood boil [title] Cut the head off your shrimp with a sharp knife. [title] Devein the shrimp. [step] Your shrimp will have a natural arch to it so cut along the middle of the inner arch up to the tip of the tail where the intestinal tract is.\n\nA. [title] Place the shrimp in boiling water for 3-5 minutes to loosen them. [title] Place the shrimp into a pot of boiling water.\nB. From the cut, a black line (the shrimp's vein) will become visible. Remove the vein with your hands or a fork and wash the shrimp under running water.\nC. Don't cut on the gills yet. [title] Separate the insides of the shrimp by grasping the fleshy side with fingers that are pointing away from you.\nD. [title] Carefully cut out the intestines. [step] You may have to hack at the shrimp as you cut it.","gold":"B","extracted_answer":"B","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1712,"error":null,"output":"B"}
{"question_id":"hellaswag:00007481","item_index":26,"attempt":0,"prompt_hash":"cfe61ac5fa023bf03a54431b1cee9d2baa877781886171242b848db826f498a2","question":"[header] How to replace ball joints and wheel bearings on a 1996 f150 [title] Jack up vehicle and place securely on jackstands. [title] Remove front tires. [title] Remove brake caliper.\n\nA. [substeps] Do not let the caliper hang by the rubber hose. Always either tie it to something that's not coming off, or find a nice place for it to rest that it won't drop off.\nB. [title] Dump jack stands and throw them out (if it was available) in a trash can. [title] Inspect ball joints and wheel bearings for possible damage or cracks.\nC. [title] Remove alternator and module before applying fluid. [title] Remove tire, wiring, brake pads and radiator backing.\nD. [title] Replace wheel spring and brake support with new tire springs. [title] Replace wheel suspension spring and transmission plate spring with new spring springs.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to replace ball joints and wheel bearings on a 1996 f150 [title] Jack up vehicle and place securely on jackstands. [title] Remove front tires. [title] Remove brake caliper.\n\nA. [substeps] Do not let the caliper hang by the rubber hose. Always either tie it to something that's not coming off, or find a nice place for it to rest that it won't drop off.\nB. [title] Dump jack stands and throw them out (if it was available) in a trash can. [title] Inspect ball joints and wheel bearings for possible damage or cracks.\nC. [title] Remove alternator and module before applying fluid. [title] Remove tire, wiring, brake pads and radiator backing.\nD. [title] Replace wheel spring and brake support with new tire springs. [title] Replace wheel suspension spring and transmission plate spring with new spring springs.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1596,"error":null,"output":"A"}
{"question_id":"hellaswag:00003354","item_index":27,"attempt":0,"prompt_hash":"c001ad8200136596e8b10351a0141966705f13e360be50e208b1516ed40196b7","question":"[header] How to adopt a child in missouri [title] Assess your situation and preferences. [step] You have many options, and the type of child you want will drive who you work with during the adoption process. Spend some time considering the following : [substeps] Do you want to adopt a child in the u.s.? This is called a domestic adoption.\n\nA. You can adopt a child in the u.s. you can adopt the child in the u.s.\nB. However, you might have to wait a long time for a child to become available. Do you want to adopt a child internationally? You might have a better chance at finding a child, but international adoptions can be complicated and more expensive.\nC. You can adopt a child at the start of an adoption process, where the adopting party adopts a minor child in a different state. They provide a temporary home for the minor child, then return the child to their home country and select a permanent home from there.\nD. You are responsible for raising the child and supporting him or her through the adoption process. You would want to adopt a child under the age of two.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to adopt a child in missouri [title] Assess your situation and preferences. [step] You have many options, and the type of child you want will drive who you work with during the adoption process. Spend some time considering the following : [substeps] Do you want to adopt a child in the u.s.? This is called a domestic adoption.\n\nA. You can adopt a child in the u.s. you can adopt the child in the u.s.\nB. However, you might have to wait a long time for a child to become available. Do you want to adopt a child internationally? You might have a better chance at finding a child, but international adoptions can be complicated and more expensive.\nC. You can adopt a child at the start of an adoption process, where the adopting party adopts a minor child in a different state. They provide a temporary home for the minor child, then return the child to their home country and select a permanent home from there.\nD. You are responsible for raising the child and supporting him or her through the adoption process. You would want to adopt a child under the age of two.","gold":"B","extracted_answer":"B","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1850,"error":null,"output":"B"}
{"question_id":"hellaswag:00001371","item_index":28,"attempt":0,"prompt_hash":"dec8afd01bbe450168d2194872123038ed3751e5bd9648c687de0754551ac7e5","question":"He moves a can and touches two other cans. He picks up a can and sprays it on the woman's arm and rubs it. he\n\nA. picks up another can and talks about it then sprays her arm again and rubs the cream in.\nB. rubs foam on her arm in a circular motion that includes her eyes.\nC. leans over with his arm out and the knife enters the sleeve of her gown.\nD. then moves the can and wipe it off a row of cans.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nHe moves a can and touches two other cans. He picks up a can and sprays it on the woman's arm and rubs it. he\n\nA. picks up another can and talks about it then sprays her arm again and rubs the cream in.\nB. rubs foam on her arm in a circular motion that includes her eyes.\nC. leans over with his arm out and the knife enters the sleeve of her gown.\nD. then moves the can and wipe it off a row of cans.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1058,"error":null,"output":"A"}
{"question_id":"hellaswag:00004689","item_index":29,"attempt":0,"prompt_hash":"3d949ab877a60ae581fb5e8a42d686d560d2b935f4774970a3d74596063154e3","question":"[header] How to make gelato [title] Fill a shallow pan with water and bring it to a simmer. [step] The pan should be large enough to hold a smaller heatproof bowl. [title] Separate the yolks from the whites.\n\nA. [step] This will divide the yolks in half. [title] Add the flour , 1 tablespoon (15 ml) at a time; these should be separated from one another.\nB. [step] Sift them into a small container. [title] Add the other ingredients (optional).\nC. [step] Place 2 bowls or other containers on a flat, clean surface. One container is for the egg whites and the other is for the yolks.\nD. [step] While each piece of yolk can go in small batches, there are two basic ways to separate them. [substeps] For instance, you could break each yolk into two separate bowls, so you can place the yolks into one large bowl and the whites into another.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to make gelato [title] Fill a shallow pan with water and bring it to a simmer. [step] The pan should be large enough to hold a smaller heatproof bowl. [title] Separate the yolks from the whites.\n\nA. [step] This will divide the yolks in half. [title] Add the flour , 1 tablespoon (15 ml) at a time; these should be separated from one another.\nB. [step] Sift them into a small container. [title] Add the other ingredients (optional).\nC. [step] Place 2 bowls or other containers on a flat, clean surface. One container is for the egg whites and the other is for the yolks.\nD. [step] While each piece of yolk can go in small batches, there are two basic ways to separate them. [substeps] For instance, you could break each yolk into two separate bowls, so you can place the yolks into one large bowl and the whites into another.","gold":"C","extracted_answer":"C","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1816,"error":null,"output":"C"}
{"question_id":"hellaswag:00008127","item_index":30,"attempt":0,"prompt_hash":"f28ad4113759abe2ad784b54f0cfec92a3461ab0fba1138670e8715fae21d9b2","question":"[header] How to host an english style high tea [title] Pick a time. [step] Consider your guests when choosing a time. People often host tea during the afternoon.\n\nA. Work and/or school may be a problem during the afternoon on weekdays, however. A weekend afternoon tea could be ideal if you let your guests know about the date you choose in enough time.\nB. The most popular time is slightly below 10pm so it would work to host the group for one hour. However, if, at the end of that time, you want your group to be very close, then a mid afternoon tea can also work well.\nC. Choose a time that is convenient for you and your guests. [substeps] For example, you might invite your friends and family over for tea at 8 pm.\nD. If people you are inviting are already well entrenched in local politics, religious issues, or other matters, pick the right time to host the tea. [substeps] Consider setting up an informational meeting or meeting in the afternoon.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to host an english style high tea [title] Pick a time. [step] Consider your guests when choosing a time. People often host tea during the afternoon.\n\nA. Work and/or school may be a problem during the afternoon on weekdays, however. A weekend afternoon tea could be ideal if you let your guests know about the date you choose in enough time.\nB. The most popular time is slightly below 10pm so it would work to host the group for one hour. However, if, at the end of that time, you want your group to be very close, then a mid afternoon tea can also work well.\nC. Choose a time that is convenient for you and your guests. [substeps] For example, you might invite your friends and family over for tea at 8 pm.\nD. If people you are inviting are already well entrenched in local politics, religious issues, or other matters, pick the right time to host the tea. [substeps] Consider setting up an informational meeting or meeting in the afternoon.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1746,"error":null,"output":"A"}
{"question_id":"hellaswag:00004936","item_index":31,"attempt":0,"prompt_hash":"4c5512327130c2bd41b5fbd7453d8b0b3b8903bc96d3b5c3346ffd8182209cfa","question":"[header] How to get in shape for volleyball [title] Perform lunges. [step] Start from a standing position. First, straighten your back and roll your shoulders back and down to keep your core engaged.\n\nA. Breathe in and out slowly while in the lunge position, keeping your abs tight to your body. [substeps] Brace your feet and place your hand on your hips with your fingers pointing in front of you.\nB. Raise your chin and fix your eyes at a spot directly in front of you, at eye-level. Then : [substeps] Take a step forward with one leg so that ankle is directly below your knee, with your calf and upper thigh forming a right angle.\nC. After recovering, raise up one leg with your heel slightly facing forward. Once you are standing, point your toes so that your heels are pointing towards a ball and continue working on getting the muscles in your core engaged.\nD. After you've got the hang of this, try to do two sets of lunges each on each side of your body. [substeps] To extend your lunges, jump slowly and take one lunge while keeping your feet squarely on the ground.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to get in shape for volleyball [title] Perform lunges. [step] Start from a standing position. First, straighten your back and roll your shoulders back and down to keep your core engaged.\n\nA. Breathe in and out slowly while in the lunge position, keeping your abs tight to your body. [substeps] Brace your feet and place your hand on your hips with your fingers pointing in front of you.\nB. Raise your chin and fix your eyes at a spot directly in front of you, at eye-level. Then : [substeps] Take a step forward with one leg so that ankle is directly below your knee, with your calf and upper thigh forming a right angle.\nC. After recovering, raise up one leg with your heel slightly facing forward. Once you are standing, point your toes so that your heels are pointing towards a ball and continue working on getting the muscles in your core engaged.\nD. After you've got the hang of this, try to do two sets of lunges each on each side of your body. [substeps] To extend your lunges, jump slowly and take one lunge while keeping your feet squarely on the ground.","gold":"B","extracted_answer":"B","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":2042,"error":null,"output":"B"}
{"question_id":"hellaswag:00007509","item_index":32,"attempt":0,"prompt_hash":"b9e139f5ff460012b24b1b34ec71a11fc5612cfcca3e3304c075147381f7245c","question":"[header] How to start dreads [title] Section your hair into small squares. [step] You'll need a friend to help with this method: they should use their hands or a comb to part your hair into multiple small squares. It's up to you how many squares your friend makes.\n\nA. Each square will become a single dreadlock, and smaller squares make thinner dreads. Generally , 1-or 2-inch (2.5 or 5 cm) squares are preferable, though it depends on what you want.\nB. [title] Use your hands to create food-sized strips from the hair you're affixing to dreads. [title] Feed the strips at medium frequency until your hair is disconnected from the dreads.\nC. A small square will fit around the core but large squares will fit on top. [substeps] Some people prefer this method because you have to supervise the center of the hair, tuck at an even rate, and it also gives them a bit more control with how often they use hair as they work.\nD. After a few minutes of doing this, switch it down to your roll curling method. [substeps] If you're struggling with how to determine how many squares you should curl, consult your friend's instructions.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to start dreads [title] Section your hair into small squares. [step] You'll need a friend to help with this method: they should use their hands or a comb to part your hair into multiple small squares. It's up to you how many squares your friend makes.\n\nA. Each square will become a single dreadlock, and smaller squares make thinner dreads. Generally , 1-or 2-inch (2.5 or 5 cm) squares are preferable, though it depends on what you want.\nB. [title] Use your hands to create food-sized strips from the hair you're affixing to dreads. [title] Feed the strips at medium frequency until your hair is disconnected from the dreads.\nC. A small square will fit around the core but large squares will fit on top. [substeps] Some people prefer this method because you have to supervise the center of the hair, tuck at an even rate, and it also gives them a bit more control with how often they use hair as they work.\nD. After a few minutes of doing this, switch it down to your roll curling method. [substeps] If you're struggling with how to determine how many squares you should curl, consult your friend's instructions.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":2148,"error":null,"output":"A"}
{"question_id":"hellaswag:00002037","item_index":33,"attempt":0,"prompt_hash":"0529d743a9f9f1673db3d64949e9bb91bec965491d51c4fe8e33f35266cb0137","question":"A person is seen running onto a court and shooting a basketball. the man\n\nA. tosses the shot put to another player while the score of the game is shown.\nB. keeps throwing the balls as well as from behind while the camera takes pictures.\nC. is shown in several clips running down and scoring a basket.\nD. runs down a track and catches the ball and throws it off in the distance.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nA person is seen running onto a court and shooting a basketball. the man\n\nA. tosses the shot put to another player while the score of the game is shown.\nB. keeps throwing the balls as well as from behind while the camera takes pictures.\nC. is shown in several clips running down and scoring a basket.\nD. runs down a track and catches the ball and throws it off in the distance.","gold":"C","extracted_answer":"C","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":871,"error":null,"output":"C"}
{"question_id":"hellaswag:00007831","item_index":34,"attempt":0,"prompt_hash":"26ea288bfdcaa4afe6c3925a90a947ef4e1e9f15155008788bb802d5cdfebf05","question":"[header] How to apply eye black for baseball [title] Remove the eye black stick from its package. [title] Take off the lid to the eye black stick. [step] A lid to eye black is removed much in the same way you remove a lid from a stick of lip balm.\n\nA. [title] Apply the eye black to your face. [substeps] Place the stick on the outside of either eye.\nB. [substeps] Lipstick is removed from the mouth prior to applying makeup. If you don't want to remove the lid, use either a q-tip, toothbrush or mini bottle.\nC. Go with the universal method: simply remove the lid. [title] File the top of the stick onto the lid.\nD. The lid may fit in the palm of your hand, but it will slide across your eye to one side. [substeps] If your lid comes in a bottle or small round plastic container, make sure you grab that first.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to apply eye black for baseball [title] Remove the eye black stick from its package. [title] Take off the lid to the eye black stick. [step] A lid to eye black is removed much in the same way you remove a lid from a stick of lip balm.\n\nA. [title] Apply the eye black to your face. [substeps] Place the stick on the outside of either eye.\nB. [substeps] Lipstick is removed from the mouth prior to applying makeup. If you don't want to remove the lid, use either a q-tip, toothbrush or mini bottle.\nC. Go with the universal method: simply remove the lid. [title] File the top of the stick onto the lid.\nD. The lid may fit in the palm of your hand, but it will slide across your eye to one side. [substeps] If your lid comes in a bottle or small round plastic container, make sure you grab that first.","gold":"A","extracted_answer":"C","choices":null,"choice_scores":null,"score_normalization":null,"score":0,"passed":false,"latency_ms":1838,"error":null,"output":"C"}
{"question_id":"hellaswag:00009218","item_index":35,"attempt":0,"prompt_hash":"8f1c7eecf3d77dcfc345c0f371792836390fc97405d9b857d8835dcb9f8c8509","question":"[header] How to touch up car paint [title] Buy a matching touch-up paint color. [substeps] Check the firewall (bulkhead) to find the paint code number. The firewall is the piece of sheet metal that separates the engine under the hood from the passengers inside the vehicle.\n\nA. The number will also designate the name of the engine you will be touching up. [title] Call your locksmith to make a custom primer for your car.\nB. If you don't have a matching touch-up paint code, observe the color coding in the firewall. This is usually indicated with a red and white square at the bottom of the panel.\nC. You will need to open your hood to find this number. Purchase a primer along with the touch-up paint unless the particular touch-up paint you've chosen states otherwise.\nD. The code is imprinted along the bottom of the firewall, beginning at the fully hot brights. [title] Look for the manufacturer's matching logo under the engine compartment (backside).","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to touch up car paint [title] Buy a matching touch-up paint color. [substeps] Check the firewall (bulkhead) to find the paint code number. The firewall is the piece of sheet metal that separates the engine under the hood from the passengers inside the vehicle.\n\nA. The number will also designate the name of the engine you will be touching up. [title] Call your locksmith to make a custom primer for your car.\nB. If you don't have a matching touch-up paint code, observe the color coding in the firewall. This is usually indicated with a red and white square at the bottom of the panel.\nC. You will need to open your hood to find this number. Purchase a primer along with the touch-up paint unless the particular touch-up paint you've chosen states otherwise.\nD. The code is imprinted along the bottom of the firewall, beginning at the fully hot brights. [title] Look for the manufacturer's matching logo under the engine compartment (backside).","gold":"C","extracted_answer":"B","choices":null,"choice_scores":null,"score_normalization":null,"score":0,"passed":false,"latency_ms":1785,"error":null,"output":"B"}
{"question_id":"hellaswag:00003483","item_index":36,"attempt":0,"prompt_hash":"b85bea1f6f6c91f3bef4cab653a9a497bb2f06e0b652e85df20d5bd48e54781b","question":"[header] How to recognize obsessive compulsive disorder in children [title] Don't jump to conclusions. [step] Remember that children have quirks and often go through phases that may make you wonder whether they are normal. If you're concerned that your child may have any kind of mental disorder, it's best to talk to a pediatrician or a child psychologist before you try to diagnose a disorder on your own.\n\nA. [substeps] Your child may have borderline obsessive disorder. This may include a learning disorder, an inability to think and be articulate, or some sort of cognitive impairment.\nB. Your first instinct is to freak out when your child does something that is completely out of character. Your children may be pansensitive, compulsive, or make mistakes.\nC. If you have had your child evaluated and are still unsure, don't be afraid to get a second opinion. [title] Look out for signs of obsessions.\nD. [title] Consider the symptoms and reactions of your child. [step] Many children with ocd have behaviors that others find unappealing.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to recognize obsessive compulsive disorder in children [title] Don't jump to conclusions. [step] Remember that children have quirks and often go through phases that may make you wonder whether they are normal. If you're concerned that your child may have any kind of mental disorder, it's best to talk to a pediatrician or a child psychologist before you try to diagnose a disorder on your own.\n\nA. [substeps] Your child may have borderline obsessive disorder. This may include a learning disorder, an inability to think and be articulate, or some sort of cognitive impairment.\nB. Your first instinct is to freak out when your child does something that is completely out of character. Your children may be pansensitive, compulsive, or make mistakes.\nC. If you have had your child evaluated and are still unsure, don't be afraid to get a second opinion. [title] Look out for signs of obsessions.\nD. [title] Consider the symptoms and reactions of your child. [step] Many children with ocd have behaviors that others find unappealing.","gold":"C","extracted_answer":"D","choices":null,"choice_scores":null,"score_normalization":null,"score":0,"passed":false,"latency_ms":1858,"error":null,"output":"D"}
{"question_id":"hellaswag:00001966","item_index":37,"attempt":0,"prompt_hash":"03277f8021053b5f5d8cbc104909ff81bc6c01fff2b0395be79f30f144576cc9","question":"A man is sitting in a barber chair. Another man is giving him a hair cut. he\n\nA. takes the hair over the scissors.\nB. then has him cut and styled.\nC. shows off his trimmed hair from a man sitting behind him.\nD. then brushes off the mans neck.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nA man is sitting in a barber chair. Another man is giving him a hair cut. he\n\nA. takes the hair over the scissors.\nB. then has him cut and styled.\nC. shows off his trimmed hair from a man sitting behind him.\nD. then brushes off the mans neck.","gold":"D","extracted_answer":"B","choices":null,"choice_scores":null,"score_normalization":null,"score":0,"passed":false,"latency_ms":690,"error":null,"output":"B"}
{"question_id":"hellaswag:00008370","item_index":38,"attempt":0,"prompt_hash":"e639be82a796fe1684fe91023b04c4395245bca66cef2c004edfa6ddc9adf1fc","question":"[header] How to start a vending route [title] Choose the type and size of the locations you want for your vending route. [step] The places you put your machines will determine which vending machines you need to purchase. You should know : [substeps] Whether your machines will be inside or outside; the type of products people at the location would be interested in vending; and approximately how many people will use your machines.\n\nA. This will help you decide what type of vending machine to have. For example, if you have gas carts with vending machines in them, you would want your machines to be able to use them.\nB. How many machines can you build on your location. Generally , 20-30 machines, so models may be nearby, larger includes retail, craft, stationary, bridge, stationery, and colored plastic/wooden buildings.\nC. [title] Choose the type and size of machines that best meet your location needs. [step] Guesstimate the size of the equipment you need by assuming a daily average of one vend, per machine, per person.\nD. Alternatively, you can purchase multiple vending machines. These might include : [substeps] Grocery stores and large grocery stores anywhere outdoors during cooler months.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to start a vending route [title] Choose the type and size of the locations you want for your vending route. [step] The places you put your machines will determine which vending machines you need to purchase. You should know : [substeps] Whether your machines will be inside or outside; the type of products people at the location would be interested in vending; and approximately how many people will use your machines.\n\nA. This will help you decide what type of vending machine to have. For example, if you have gas carts with vending machines in them, you would want your machines to be able to use them.\nB. How many machines can you build on your location. Generally , 20-30 machines, so models may be nearby, larger includes retail, craft, stationary, bridge, stationery, and colored plastic/wooden buildings.\nC. [title] Choose the type and size of machines that best meet your location needs. [step] Guesstimate the size of the equipment you need by assuming a daily average of one vend, per machine, per person.\nD. Alternatively, you can purchase multiple vending machines. These might include : [substeps] Grocery stores and large grocery stores anywhere outdoors during cooler months.","gold":"C","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":0,"passed":false,"latency_ms":2156,"error":null,"output":"A"}
{"question_id":"hellaswag:00006054","item_index":39,"attempt":0,"prompt_hash":"d07bd32ddbc4ee221a9f5229458755d5eb3cfc6a36eccc00d1e7f5d1ad6dee90","question":"[header] How to move heavy furniture [title] Invest in furniture sliders. [step] You can buy the sliders of appropriate size at your local hardware or home goods store. National chains like home depot or lowes will definitely sell furniture sliders.\n\nA. The sliders will help to ensure that the furniture doesn't sag or strain when you load it. They are usually made of some kind of heavy-duty fabric that makes them sturdier.\nB. Just make sure to check the packaging before you buy one. [substeps] Avoid chinchillas, trombone meats, and porgie foods.\nC. If you are planning to move your furniture over carpet or grass you should buy sliders that specialize in that movement. [substeps] If you don't have any sliders you can also try using frisbees.\nD. [substeps] If you're not familiar with how they are designed, try home depot or lowes. They often feature special sliders designed to maximize the surface area (which is convenient to most furniture pusters).","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to move heavy furniture [title] Invest in furniture sliders. [step] You can buy the sliders of appropriate size at your local hardware or home goods store. National chains like home depot or lowes will definitely sell furniture sliders.\n\nA. The sliders will help to ensure that the furniture doesn't sag or strain when you load it. They are usually made of some kind of heavy-duty fabric that makes them sturdier.\nB. Just make sure to check the packaging before you buy one. [substeps] Avoid chinchillas, trombone meats, and porgie foods.\nC. If you are planning to move your furniture over carpet or grass you should buy sliders that specialize in that movement. [substeps] If you don't have any sliders you can also try using frisbees.\nD. [substeps] If you're not familiar with how they are designed, try home depot or lowes. They often feature special sliders designed to maximize the surface area (which is convenient to most furniture pusters).","gold":"C","extracted_answer":"D","choices":null,"choice_scores":null,"score_normalization":null,"score":0,"passed":false,"latency_ms":1828,"error":null,"output":"D"}
{"question_id":"hellaswag:00003019","item_index":40,"attempt":0,"prompt_hash":"ca45e135edc72389c5ec8cb8a6b1e409e155e80e4f981b8e708122c3c90489d1","question":"A man is shown running down a track. he\n\nA. throws open his arms, spinning around on a dirt track.\nB. does a series of back flips while going down the track.\nC. pulls a stick as he runs.\nD. then catapults into the air over a bar.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nA man is shown running down a track. he\n\nA. throws open his arms, spinning around on a dirt track.\nB. does a series of back flips while going down the track.\nC. pulls a stick as he runs.\nD. then catapults into the air over a bar.","gold":"D","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":0,"passed":false,"latency_ms":686,"error":null,"output":"A"}
{"question_id":"hellaswag:00008631","item_index":41,"attempt":0,"prompt_hash":"dc030f9df14161212953f2ddfa2fd3ca9eabddcaee1cfc2c1680e2f882894ac0","question":"[header] How to wear a camel coat [title] Choose a trench coat style for a formal look. [step] Camel trench coats are one of the most popular styles, because they can make any outfit look immediately classier. Fortunately, you can find ones in a range of materials from light cotton to heavy wool, depending on what you need for the fall weather where you live.\n\nA. [substeps] Camel jackets are typically black-gray, velvet, or beige. They have multiple sleeves in a row, and extend from the upper arms to the lower legs.\nB. You'll just need to choose one or both for all seasons. [substeps] Coat coats in light cotton are most commonly worn in the summertime, while coats in heavy wool are most often worn during the winter or during hot, sunny months.\nC. [substeps] The main reason camel trench coats are common was that they are far more casual than the traditionally worn more formal coats. Therefore, they take a great deal of the emphasis out of the outfit.\nD. [substeps] Choose a coat with a belt for the more traditional trench coat style, or find a design that doesn't use a belt, giving the overall effect a less busy finish. [title] Buy a peacoat for versatility.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to wear a camel coat [title] Choose a trench coat style for a formal look. [step] Camel trench coats are one of the most popular styles, because they can make any outfit look immediately classier. Fortunately, you can find ones in a range of materials from light cotton to heavy wool, depending on what you need for the fall weather where you live.\n\nA. [substeps] Camel jackets are typically black-gray, velvet, or beige. They have multiple sleeves in a row, and extend from the upper arms to the lower legs.\nB. You'll just need to choose one or both for all seasons. [substeps] Coat coats in light cotton are most commonly worn in the summertime, while coats in heavy wool are most often worn during the winter or during hot, sunny months.\nC. [substeps] The main reason camel trench coats are common was that they are far more casual than the traditionally worn more formal coats. Therefore, they take a great deal of the emphasis out of the outfit.\nD. [substeps] Choose a coat with a belt for the more traditional trench coat style, or find a design that doesn't use a belt, giving the overall effect a less busy finish. [title] Buy a peacoat for versatility.","gold":"D","extracted_answer":"D","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":2219,"error":null,"output":"D"}
{"question_id":"hellaswag:00004355","item_index":42,"attempt":0,"prompt_hash":"c373cd461121335361689f8012fb796c65f0559e7d3cb49caca719b72cc685bb","question":"[header] How to make your hair stand up [title] Wash your hair. [step] You'll have an easier time adding volume to your hairstyle if you're working with clean hair. Gently dry off with a towel when you're finished.\n\nA. [title] Blow dry your hair from head to tail. [step] Use flat dry hair is what all the adults used to say.\nB. You can also skip this step if your hair is very clean, but you can't mess with the part you just washed. You can separate your hair into four pieces and wash them all in the same washing cycle.\nC. Make sure you use products specially designed for working with hair. [substeps] If you have light hair, it's a good idea to skip products like blow drying it straight, rubbing alcohol into it, or straightening it a bit (as well).\nD. [title] Massage 1 tablespoon (15 ml) of pre-styling mousse into your damp hair. [step] Rub a palm-sized amount of mousse between your hands, and then run your fingers from the front to the back of your scalp.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to make your hair stand up [title] Wash your hair. [step] You'll have an easier time adding volume to your hairstyle if you're working with clean hair. Gently dry off with a towel when you're finished.\n\nA. [title] Blow dry your hair from head to tail. [step] Use flat dry hair is what all the adults used to say.\nB. You can also skip this step if your hair is very clean, but you can't mess with the part you just washed. You can separate your hair into four pieces and wash them all in the same washing cycle.\nC. Make sure you use products specially designed for working with hair. [substeps] If you have light hair, it's a good idea to skip products like blow drying it straight, rubbing alcohol into it, or straightening it a bit (as well).\nD. [title] Massage 1 tablespoon (15 ml) of pre-styling mousse into your damp hair. [step] Rub a palm-sized amount of mousse between your hands, and then run your fingers from the front to the back of your scalp.","gold":"D","extracted_answer":"D","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":2050,"error":null,"output":"D"}
{"question_id":"hellaswag:00008445","item_index":43,"attempt":0,"prompt_hash":"c1d20c8d22297da8d455df2d87ed8d9a9b451e9a80b06e950b033e9888578d12","question":"[header] How to do eye makeup for blue eyes [title] Use eyeshadow primer under your eyeshadow. [step] This will help enhance the color of your eyeshadow, making it appear brighter, which in turn will also make your eyes appear brighter. [title] Choose eyeshadow with contrasting colors.\n\nA. [step] One of the best ways to bring out the blue in your eyes is to use contrasting colors. This means choosing eyeshadows in neutral and earth tones, warm colors, and colors opposite of blue and green on the color wheel.\nB. [step] Avoid black, or pink, however. This will only enhance the orange/black look without adding any contrast.\nC. [step] You may find it easier to use contrasting colors if your eyes are slightly different. Then, when applying eyeshadow primer for blue eyes you may choose a less neutral shade, such as beige or brown.\nD. [step] Go to the store, if you have one, and figure out what color eyeshadow you'd like. It should look one color but not several different colors, as this will make your eyes look noticeably brighter.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to do eye makeup for blue eyes [title] Use eyeshadow primer under your eyeshadow. [step] This will help enhance the color of your eyeshadow, making it appear brighter, which in turn will also make your eyes appear brighter. [title] Choose eyeshadow with contrasting colors.\n\nA. [step] One of the best ways to bring out the blue in your eyes is to use contrasting colors. This means choosing eyeshadows in neutral and earth tones, warm colors, and colors opposite of blue and green on the color wheel.\nB. [step] Avoid black, or pink, however. This will only enhance the orange/black look without adding any contrast.\nC. [step] You may find it easier to use contrasting colors if your eyes are slightly different. Then, when applying eyeshadow primer for blue eyes you may choose a less neutral shade, such as beige or brown.\nD. [step] Go to the store, if you have one, and figure out what color eyeshadow you'd like. It should look one color but not several different colors, as this will make your eyes look noticeably brighter.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":2136,"error":null,"output":"A"}
{"question_id":"hellaswag:00008759","item_index":44,"attempt":0,"prompt_hash":"c8362841288618abe66c6960ae4d4fa488232412ec64f082ca44949a924bbefe","question":"[header] How to balance work and family [title] Decide what your values are for yourself and your family. [step] A value is a principle, standard, or quality considered worthwhile or desired. Values guide our actions and structure our lives.\n\nA. Make a list of values you feel have value to you. [substeps] Look at how often you feel valued or desired.\nB. Decide what your values are and draw inspiration to your personal goals : [substeps] Don't immediately become addicted to a book or tv series immediately. Don't give up faith in nature, visual or scientific intent of mind, decade-long solitary time despite out growing money.\nC. [substeps] Areas where we often have strong values include housework, meal times, child care, car and house maintenance, relationships between spouses and parents and children, education, money, politics, religion, etc. Specifying your values is key to managing work and family demands.\nD. Grandparents are often taught a value is something that each parent doesn't think otherwise, like \" because we are doing the best job. \" in turns, values dictate what we do and the way we live.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to balance work and family [title] Decide what your values are for yourself and your family. [step] A value is a principle, standard, or quality considered worthwhile or desired. Values guide our actions and structure our lives.\n\nA. Make a list of values you feel have value to you. [substeps] Look at how often you feel valued or desired.\nB. Decide what your values are and draw inspiration to your personal goals : [substeps] Don't immediately become addicted to a book or tv series immediately. Don't give up faith in nature, visual or scientific intent of mind, decade-long solitary time despite out growing money.\nC. [substeps] Areas where we often have strong values include housework, meal times, child care, car and house maintenance, relationships between spouses and parents and children, education, money, politics, religion, etc. Specifying your values is key to managing work and family demands.\nD. Grandparents are often taught a value is something that each parent doesn't think otherwise, like \" because we are doing the best job. \" in turns, values dictate what we do and the way we live.","gold":"C","extracted_answer":"C","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":2115,"error":null,"output":"C"}
{"question_id":"hellaswag:00002185","item_index":45,"attempt":0,"prompt_hash":"f1e2d97a3ba249785ab3126aaec4fad1d0d79a0894066f0b7202172221e0343f","question":"Two men stand on the far left of a beer pong table. one of the men\n\nA. throws a beer pong ball across the table, into a cup.\nB. begins hitting the ball into the other dealer.\nC. throws a shot put, then two men play beer pong.\nD. hits the ball.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nTwo men stand on the far left of a beer pong table. one of the men\n\nA. throws a beer pong ball across the table, into a cup.\nB. begins hitting the ball into the other dealer.\nC. throws a shot put, then two men play beer pong.\nD. hits the ball.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":695,"error":null,"output":"A"}
{"question_id":"hellaswag:00001496","item_index":46,"attempt":0,"prompt_hash":"9c2c0752b9feaabbd365747d3f412376d8e50433198f808b0dccbaf0f1e45404","question":"Children swings a broom handle at a pinata in a backyard. the boy\n\nA. breaks the pinata open and candy spills on the ground.\nB. smiles, then hops on the broom handle.\nC. sits on a swing set and mows the patch of grass where the boy sits.\nD. swings a mop handle at the pinata too then goes to get a different mop.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nChildren swings a broom handle at a pinata in a backyard. the boy\n\nA. breaks the pinata open and candy spills on the ground.\nB. smiles, then hops on the broom handle.\nC. sits on a swing set and mows the patch of grass where the boy sits.\nD. swings a mop handle at the pinata too then goes to get a different mop.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":886,"error":null,"output":"A"}
{"question_id":"hellaswag:00002109","item_index":47,"attempt":0,"prompt_hash":"758a6e7be49547c0d2d72ed3c1758a4e53c85273ae43ca59e62341d224670b10","question":"A camera shows a person riding around a dirt track on a bike moving at a quick speed. several more shots\n\nA. are shown of people riding the dirt track on bikes while the camera captures them from several angles.\nB. are shown of people riding around on bikes and speaking to one another.\nC. are shown of people riding in different speeds and going over hills passing where there is much higher grass.\nD. are shown of people riding around on the dirt track as well as riding on a stationary bike.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\nA camera shows a person riding around a dirt track on a bike moving at a quick speed. several more shots\n\nA. are shown of people riding the dirt track on bikes while the camera captures them from several angles.\nB. are shown of people riding around on bikes and speaking to one another.\nC. are shown of people riding in different speeds and going over hills passing where there is much higher grass.\nD. are shown of people riding around on the dirt track as well as riding on a stationary bike.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":968,"error":null,"output":"A"}
{"question_id":"hellaswag:00007658","item_index":48,"attempt":0,"prompt_hash":"39b764ac980e0e82d4b0751f4369360dc6b923e7b52dbe76b54cef9eb64706a3","question":"[header] How to clean white vans shoes [title] Mix up a cleaning solution. [step] Using a mixture of laundry detergent and warm water is an easy way to clean vans. Into a large bowl, mix 1/4 cup laundry detergent and 2 cups warm water.\n\nA. Dump in 1/4 cup fabric softener and 1 cup bleach. Mix together with a spoon or a rubber scrub brush.\nB. If your vans are extremely dirty, you may need to mix up a clean batch for the second shoe. If you're low on laundry detergent, here are other cleaning solutions you can try.\nC. Pour the cleaning solution into the shoe, scrub the inside of the shoe, and rub in the rinse with a cloth or rag. The rinse will return the shoe to its original clean state.\nD. Swirl the solution on a rag or cloth to get suds into the shoes. As you rub the solution in, notice if the color of your vans shoes gets darker or lighter.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to clean white vans shoes [title] Mix up a cleaning solution. [step] Using a mixture of laundry detergent and warm water is an easy way to clean vans. Into a large bowl, mix 1/4 cup laundry detergent and 2 cups warm water.\n\nA. Dump in 1/4 cup fabric softener and 1 cup bleach. Mix together with a spoon or a rubber scrub brush.\nB. If your vans are extremely dirty, you may need to mix up a clean batch for the second shoe. If you're low on laundry detergent, here are other cleaning solutions you can try.\nC. Pour the cleaning solution into the shoe, scrub the inside of the shoe, and rub in the rinse with a cloth or rag. The rinse will return the shoe to its original clean state.\nD. Swirl the solution on a rag or cloth to get suds into the shoes. As you rub the solution in, notice if the color of your vans shoes gets darker or lighter.","gold":"B","extracted_answer":"B","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1903,"error":null,"output":"B"}
{"question_id":"hellaswag:00007241","item_index":49,"attempt":0,"prompt_hash":"89b678b8bc13e6a6c85a0047ad474ef58d3228ceea44c59b0eea523ffd7b2a51","question":"[header] How to make fried smashed potatoes [title] Place the potatoes in a pot and cover with water. [step] Add 4 pounds (1.8 kg) of small red or yukon gold potatoes to a large pot. Fill the pot with enough cold water to cover the potatoes by at least 2-inches (5-cm).\n\nA. [substeps] Fingerling potatoes also work well for the recipe. [title] Season the water with salt and bring to a boil.\nB. Add salt to the water so the potatoes are submerged. The potatoes should smell fresh and fragrant as they are infused with freshly cooked onions and garlic.\nC. Let the potatoes soak for 10 to 15 minutes. [substeps] Do not use 1 pound (0.45 kg) of frozen potatoes in this recipe.\nD. [substeps] Alternatively, if you prefer to cook the potatoes on their own, you can simply boil the water for the oven to bake them in. This should be enough time so that the potatoes can brown thoroughly and cook evenly.","prompt":"Answer the question. Reply with only the letter of the correct choice.\n\n[header] How to make fried smashed potatoes [title] Place the potatoes in a pot and cover with water. [step] Add 4 pounds (1.8 kg) of small red or yukon gold potatoes to a large pot. Fill the pot with enough cold water to cover the potatoes by at least 2-inches (5-cm).\n\nA. [substeps] Fingerling potatoes also work well for the recipe. [title] Season the water with salt and bring to a boil.\nB. Add salt to the water so the potatoes are submerged. The potatoes should smell fresh and fragrant as they are infused with freshly cooked onions and garlic.\nC. Let the potatoes soak for 10 to 15 minutes. [substeps] Do not use 1 pound (0.45 kg) of frozen potatoes in this recipe.\nD. [substeps] Alternatively, if you prefer to cook the potatoes on their own, you can simply boil the water for the oven to bake them in. This should be enough time so that the potatoes can brown thoroughly and cook evenly.","gold":"A","extracted_answer":"A","choices":null,"choice_scores":null,"score_normalization":null,"score":1,"passed":true,"latency_ms":1987,"error":null,"output":"A"}