The mathematics of UNFAIR exam questions
Audience:
Tags: satactexamsitem-analysisexam-designsigmoid
Why do big exams like the SAT, GRE, ACT, LSAT, or GMAT often feel UNFAIR? Trick questions, ambiguity, and senseless difficulty make these tests frustrating. But to an extent, these tests HAVE to be this way.
This video looks at 4 example SAT or ACT questions that might be considered unfair. Through these questions, we’ll discover a simple formula that’s behind every exam question you’ve ever seen. The formula explains why its necessary to write hard questions that feel tricky or unfair to students. These questions often must have tricky but convincing answer choices (called distractors) or be brutally difficult.
Specifically, the video presents the 3-item characteristic curve for item analysis. This formula, and others similar to it, is used at large companies like College Board to design exams. The video is mostly targeted to high-school students who may be mathematically inclined (or angry about their exam scores), but educators should find value in understanding how to analyze test items in more detail.
Thanks for watching! -John-
Analytics
Comments
Truly outstanding video. Very hooking
FWIW I don’t agree that 16 is the answer, because you did a sneaky notational change in explaining the 16 answer from 2(4) to 2 x 4. That’s a huge difference IMO and worth acknowleding rather than dodging.
Very nice video. Enticing motivation and memorable insights. The animation and styling are a bit bare-bones but effective. My main complaint would be pacing: the “3,2,1 go“‘s make the video stressful to watch (“pause here and think about it” would be gentler) and also the introduction of your probability graph was too fast for me to take in without re-watching, which is bad for an element so critical to your explanation. Last, naming the parameters a, b, and c, rather than more descriptive names, meant I had to keep reminding myself what each of those meant once the equation left the screen.
But really, the content was great and the explanation was very clear, and I learned a lot.
Gotta say, great and enjoyable video with memorable concepts. Really shows to quantify how some exam might be ‘tougher’ than other one. 9/9
Absolute cinema. Well-motivated. Really clear explanation. It’s clear that bro is a seasoned educator.
It was quite an engaging video owing to the creator asking the audience to participate in the questions while on screen, and observe if it feels “unfair”, to then supply it with an explanation for what worked and what did not. The introduction seemed a bit abrupt to me for I did not get if I’m supposed to find these questions ‘unfair’ which is the device that builds up the narrative of the video. But maybe I’m the minority.
What worked well & What I especially liked:
The framing of the video is brilliant. Connecting the emotional frustration of “unfair” exam questions directly to Item Response Theory (IRT) and the 3-parameter logistic model gives immediate motivation to care about the math. The animations were top-notch—clean, intuitive, and extremely helpful in visualizing how varying discrimination (a), difficulty (b), and pseudo-guessing (c) shape the characteristic curve.
What I learned & Key Connection:
I really enjoyed seeing the formalization of the guessing parameter alongside the sigmoid function. It directly explains why standardized tests need clever distractors to keep test items statistically valid.
What could be improved & Actionable suggestion:
-
Pacing: For a high-school and undergraduate audience with a mathematical inclination, the breakdown of the formula could have been paced slightly faster without losing clarity.
-
Topic Expansion (Data Fitting): The video builds great intuition for how the parameters dictate the curve, but it leaves out how test makers actually infer these parameters from real student response data. Including a brief overview of how sample responses are used to fit the sigmoid curve (such as maximum likelihood estimation) would have given the mathematical arc a complete and deeply satisfying conclusion.
Overall, an outstanding and beautifully animated submission!
Very informative. Nicely animated.
I really enjoyed this video. The presenter was really good at explaining tricky math problems and the formula used to create them. I stopped, did the problems, and did not get one bit bored or perplexed.
This video has one major flaw. If it is intended for a general audience (ie teachers, students etc…) the presnter needs to define what e is. HE DID NOT DO THAT. Defining variables and constants is very important.
Great applied statistics explainer. Would greatly benefit from a non-American perspective
I guess it’s a good clickbait title but the actual content of this video is just not that interesting to me personally.
Very interesting. I never put that much thought into test design, and this really made me appreciate how carefully standardized tests are put together.
Really great video! I was kind of skeptical at first because of the way the video is marketed (and I think that would be my only real complaint here). It feels a bit clickbait-y, like it’s targeted at people struggling with SAT or those who find it to be unfair. You highlighted your intention as “math-inclined students” which I agree with. However, the marketing could reflect that a bit better than a core focus on struggling students which might deter some students. It’s a really well done math explainer that I think people would benefit from seeing!
The visuals were all done very well. I liked all the graphs throughout the video, the boxes, etc. They contributed a lot and really made sense. The algebra was also easy to follow when you got to the sigmoid function.
Real examples of problems also helped a lot with engagement and made the video feel fun and gamified in a way.
One last thing, I think you should have mentioned where they get the ability scores from. I assume it’s from other questions, but I can’t tell for sure. Plus, what happens if the other questions are bad then? Would the graph be skewed?
Setting aside my feelings about standardized testing, this is a good video. It justifies the sigmoid for question difficulty well, the examples motivate the use of the parameters, and you explain why each question works or doesn’t.
The pace feels a bit slow and repetitive. It might make more sense to build the model as you go, rather than explaining all the questions and then revisiting them as you build the model.
I really enjoyed this video—the setup of comparing 4 different types of questions, showing how the model was built and then showing which questions are useful for differentiating ability groups was great. I honestly don’t anything to suggest to improve this video… the visuals were quite nice and helped illustrate the math. I guess you could talk about the historical context of test companies using this equation? How it’s made the tests more fair since being implemented? But that’s not necessary…
It’s an interesting topic, I liked the concept of specifically trying to measure whether a question is good at separating students who understand from those that don’t. One thing I would have liked is an explanation why adding the a parameter results in a steeper drop (e.g., it’s scaling the x, and hence sort of squeezing the axis). Other than that the visuals are great, and the examples also demonstrate the different cases quite nicely.
Very well constructed. The subject matter of the multiple-choice questions is interesting in itself, and the description of the mathematical tools—even though they remain relatively elementary—is well executed; it is truly an engineering approach. The animation is understated and clear. The commentary is well-structured and delivered at a perfect pace. If one wanted to delve deeper into the subject, perhaps a multidimensional aspect could be introduced, with questions designed not merely to rank students but to classify them.
For me personally the video moved a bit slowly and I ended up putting it at a higher speed, but I don’t really struggle with exam questions, so I probably wasn’t the target audience.
How is ‘ability’ measured? Is it just the outcome of the whole test? In such case it would make sense that in general there is some correlation between who did well and ability even if the question was random. This however leads to a small effect if there are a lot of questions. It makes sense that it works but still I would have liked some clarifications.
I don’t relate a lot with the whole “unfair” narrative. You are explaining why hard or tricky questions are useful and I never found them unfair. What does unfair even mean? that is favoring towards more attentive or straight-up better students?
You should have included one of the easy questions with a low both a and b values to see the counterpart of hard questions and what good easy questions are, in order to compare it to the bad easy question presented.
Finally, the video is a bit slower than I would have liked it and some elements are repeated multiple times, making it a bit more boring to follow, like the presentation of the bad question with the divide sign.
What I liked a lot is the topic (sat tests) and thesis (there must be hard questions).
Interesting video! I do wish you’d spent some time reading the questions, it felt a bit rushed at times. However I can also recognize that the exact questions weren’t the point of this video, but still, I feel like it would’ve helped pace it a bit better!
I also feel like the premise of question 3 is a bit silly. Marking the wrong answer in the answer sheet is not a design decision, but a mistake, and it doesn’t really fit like the others do. The video without it is honestly a great length, so I’d probably just drop it entirely!
Really nice video! I like the way you naturally set up the topic with the examples, it felt natural, well motivated and nicely demonstrated the ideas. My main thing I think could be improved iss the question of “where is this data coming from?” aka “but who are these people?”. A quick line like “this is just made up data for illustration” would also be fine, but I found that confusing (maybe you said this and I missed it, if so sorry!). Even better would be to show a real data with it too. Similarly, a I know you mentioned ability was on an arbtary scale of -3 to 3, but a bit more detail here would be nice too (e.g. “this is a normalized score on how they did on the rest of the test” or something like that?). Overall great job though, its a nice little video!
Good motivating question. “Are SAT’s unfair?” It does get a bit technical with the equation derivation and I’m not sure if the equation is a way of identifying a good question for the test before or after it is given on the test. This work does show that the point of view of the test writers is sometimes(often) different from the students’ point of view and explains why ( in part) the question and test may be viewed differently. Not so much “Can students get this right?” But “Does this differentiate between the good and poorer students?”
Interesting!
I think the pacing was a little off - could have gone faster for the algebra/parameterization part and spent more time on how it’s useful