3blue1brownHow a Transformer Plays Tic-Tac-Toe
Audience:
An interactive article that explains how Transformers work by training a very small model to play fading Tic-Tac-Toe.
Instead of using text prediction, the article uses a simple game where the model can still learn real strategies. This makes it possible to clearly show how the main parts of a Transformer work together to produce a prediction.
Each visualization can optionally show the exact math behind the model, so readers can follow the matrix steps as the Transformer processes a sequence.
The article also includes experiments showing what happens when different parts of the model are removed.
The intended audience is students, programmers, and anyone who wants a simple and visual introduction to how Transformers work internally.
Analytics
Comments
Visualization is nice.
Transformers are very topical, so good choice of subject matter to present. Playing the game was fun. I had never heard of fading tic-tac-toe. I found the animations of the matrix operations visually interesting but not very enlightening from a mathematical point of view. By the end of the presentation I realized that you were showing how to use a previously trained model, not how to train it. As such, the animations were just visualizing a fairly complex function evaluation. Have you considered focusing more on the training phase? For example, show how the weights change as training progresses. Also maybe show how the performance of the model improves as training proceeds. There were a few mathematical expressions but their connection with the animations could be made clearer. I think there is a lot of scope for developing this explainer further.
Really nice to see someone talk about transformers with a concrete, simple and yet meaningful application as a goal. The website is very well made. (Maybe vibe coded? If yes, at least with an understanding of design and UX which is a very nice change compared to other stuff.) I just got lost in the sauce after a while being battered by too many red and blue pixels. The text is very terse, and should put much more focus on the fact that solid linear algebra knowledge (and even basic ML knowledge TBH) is necessary to follow along.
Incredible job! Very clear and interactive illustration.
I really enjoyed this. I think the main feedback here is that this is less accessible than you would think. What you’ve done here is really to unpack your code and mathematics, but you’ve not placed too much focus on explaining the concepts and ideas