The Geometry of Inference: Ordinary Least Squares
Audience:
Tags: statisticslinear-algebradata-scienceregressionordinary-least-squaresprojectionscolumn-spacesample-mean
What if the average wasn’t just an arithmetic recipe, but a geometric object?
This video starts with a familiar calculation, the sample average, and reinterprets it using linear algebra. If we stack data values into a vector and project that vector onto the span of the all-ones vector, the projection has the form
The stretch factor is exactly the sample mean.
From there, the video builds toward ordinary least squares regression. Instead of viewing regression only as “drawing the best line through points,” we reinterpret it as a projection problem: the data vector is projected onto the column space of the design matrix . The fitted values are the projection, the residual vector is the perpendicular leftover, and the normal equations come from that right angle.
The goal is to show that the sample mean, intercept-only regression, residual orthogonality, and least squares are all part of the same geometric story.
Analytics
Comments
Good use of animation and humour. It’s hard to make formulas entertaining. I would suggest not using the variable names directly from the textbook and using something more visually striking/memorable to really utilise the visual medium.
Very creative. I like the square animation. I like the beginning how you bring humor to it. Even your subscribe was nicely done!
As a general impression, I very much like the tone of the video. Having some jokes in there makes it a lot more entertaining and you did a very good job with this. Furthermore, your little cube is especially adorable (although it seems a little unreasonably angry sometimes [still cute]). When it comes to novelty, this interpretation is not new to me and probably neither to the audience. I’ve seen this interpretation in lectures (with a lot less depth of course). It remains a little unclear who the target audience for this video is, since (usually) people who are familiar with matrix multiplication are just as familiar with the best approximation and orthogonal spaces, but I may be wrong there. Overall, I think this is a good score for a good explainer. Keep it up. PS: What did Inception do to you??
I really like the topic of this video, and overall I think the creator did a nice job. But there were a number of things that made me not love the video. For one, I just didn’t like the little cube character always on screen talking to me. And while I really appreciated the “meta” scenes, I must admit that it did detract from the overall flow. I also noticed an error in the video, where the formula was written to minimize the sum of u-hat sub-i, instead of (u-hat sub-i) squared.
Another thing I thought detracted from the video was that the creator was often just “reading” what was on the screen. And unfortunately, this led to the video starting to feel an awful lot like simply reading a book, and this continued for a long time.
Again, I want to reiterate that I really like the topic and the creator did a nice job with this video. But in the end, I felt that it was simply too long and felt too much like a book was being read to me. So, unfortunately, I didn’t feel like the video was all that useful. It’s too bad, because there are some big ideas here, but the video just wasn’t quite up to snuff. But I feel like there’s a lot of potential here, so I hope the creator continues making videos.
Cool video. Mind blown.
Maths is very intuitive and clearly explained. I must especially comment on how novel this connection is, especially in a video with such well executed jokes. Moreover, your visual style is very clean, and while your avatar can be slightly obstructive and distracting, I don’t think most viewers would complain, and it adds a personal touch. I would also recommend explaining why you/convention chose certain pronumerals briefly (~2 sentences) but this is nitpicky at best.
The idea of the video is nice and interesting; the structure is logical and engaging. But there’s an excessive concentration of AI-like idioms throughout the script, some of which are a bit strange: e.g.
-
at 4:51, “Now every point contributes one square residual: not just whether the line missed, but how badly it missed”, the contrast “not just…but…” is strange here since the video has already just been talking for some time about these errors.
-
at 7:34, “So now we’re going to take this exact same problem, and turn it sideways: not metaphorically - geometrically”, I don’t see what the geometric sideways turn is. I see that there is a beautiful shift in geometric perspective, but the language of “turn it sideways” here seems metaphorical and not geometrical.
-
perhaps the worst offender is at 16:21-26, “That is the fitted vector: not approximately, but geometrically.” The contrast between “approximately” and “geometrically” makes no sense; this seems like an AI trying to force a “not…but…” contrast structure and hallucinating a like-category contrast between “approximately” and “geometrically” because of the fact that the concepts of approximation and of geometric representation of statistics have occurred in the video up to that point.
And in connection with this last point: At 16:08-16:34, in the explaination that is meant to “make the projection idea impossible to miss”, showing a finite-height sun dispelling a spread of rays that are clearly not orthogonal to the plane [you’ve drawn your outermost rays at about 30 degrees to the plane rather than 90 degrees] while stating that the shadow is the projection is extremely confusing, and makes something that it actually very simple seem complicated. Was this also an AI’s decision?
Some aspects of the descriptions were also a bit confusing: e.g. when you explain (10:39-10:51) that we can’t just choose equal to , it would be good if you made it explicit that this is because we are looking for a linear-fit model rather than an overfitted model that perfectly fits all the given data while totally lacking useful predictive power as a result. Instead, you explain that has “rules” in terms of some complicated matrix algebra. And concerning this matrix algebra, your description at 11:10-11:55 is difficult to understand and follow (e.g. how does the column of 1s “create the intercept” and how does the column of -values “create the slope”?).
Also, at 21:53 (“Now we multiply…”), it’s not clear why further manipulations are being done, since we already got to the result we were after (as stated at 21:31), namely the line of best fit as encoded in the vector \hat{\beta}}.
On a more minor note, the facial expressions of the animated narrator seem, for much of the time, pretty random and disconnected from the content of what is being said. Again, I guess AI might perhaps be responsible for this? Also, some of the symbols appear to have not rendered properly, e.g. the bar in at 25:32-27:35.
SUMMARY: Nice content, but it looks like you’ve tried to get AI to help make your script sound snazzy (and perhaps also to help with your visuals), but AI is not yet good enough to do that well.
Great use of ai generated images :-)
A 45° angle hurt me once, but to be fair, I did bang my leg into it.
This is a good topic. The video grounds an algebraic formula into a geometric and visual space - very satisfying. I found myself thinking ‘Oh of course! That makes a lot of sense!’ The cube is adorable. The jokes were fine. The mathematical rigor seems sound. The sun’s shadow onto the prediction space was an excellent visual choice, and it allowed for good understanding of the perpendicular residual. Much of this video was matrix formulas on a blank background, and since matrix multiplication expresses a transformation, that would have made for excellent visualizations. Particularly the part that involved y_hat=y_bar*[1 1 … 1], that formula is asking “what is the length of the vector along [1 1 .. 1] that equals the length of y projected onto the same line” — that would have made a great 2D or 3D visualization. The final derivation of y_bar was very satisfying, I was waiting for that punchline and I got it.
The algebra was pretty dense, and I didn’t have the time to think through every single step of it, but I think I understood the core concept in the end. This is actually a pretty fascinating concept. I think it was a good idea to start with the “projection equals average times vector of ones” thing to build interest, then explore this idea of projections further in the rest of the video. It’s pretty obvious why using linear regressions to predict variables is an important thing, and it’s also obvious why you would want a formula for that, so I think that this video was well-motivated.
The explanations seem pretty clear for someone with a decent understanding of linear algebra and statistics, though putting the entire video on hold to explain what a projection is (at 1:39) is a strange decision if you expect your audience to know linear algebra already. I’ve never seen a presentation of this specific way of thinking geometrically about linear regression, so it does seem unique in that regard, though I’m not sure if other materials about this topic are common. I personally have had some questions in the back of my mind about linear regression for a while, so I’ll probably remember this video for helping answer some of them.
There is one elephant in the room to address, which is the AI usage. Within the video itself, the only outright confirmation is at 15:26. The description additionally clarifies, “Some non-mathematical visual assets were generated using AI,” of which the images at 0:27 are probably one example. This possibly falls under a prohibited use of AI as outlined in the AI Policy: “Generating diagrams, illustrations, animations, or visual assets that form a significant creative component of the exposition.”
Even aside from that, aspects of the script itself raise suspicion from me, and AI use there would be a significantly worse problem. There are some of the potential indicators, such as abundant use of negative parallelisms and the rule of threes. In addition, this video contains many jokes that feel particularly shoehorned in, as if they were inserted by someone who had already received a script written by someone else. Admittedly, this one can potentially be attributed to an awkward sense of humor. In any case, I see enough red flags here that I would find it worthwhile for the SoME organizers to investigate.