Coding Theory: A Playful Introduction
Audience:
Tags: discrete-mathematicscoding-theorydigital-communicationsource-codingline-coding
Coding theory forms the backbone of modern digital communication. This article is a playful introduction to some of the most important and impactful ideas in its history. While many of us are familiar with topics like error-correction and cryptography, I wanted to focus this post on a relatively less discussed topic, at least in my experience: source coding, which we will explore through an interesting problem setup from Charles Petzold.
Coding theory is usually taught at the undergraduate level, but I have tried to keep the jargon to a minimum and build things up using mostly elementary algebra. My hope is to make this accessible to everyone, from undergraduates and high schoolers to curious middle schoolers. More than anything, I want this to feel like a mathematical journey where you get to discover the ideas for yourself, step by step.
Analytics
Comments
a few missed words here and there is totally fine, i enjoyed the read
Most of your topic is, in my experience, already widely-explained. Unary, frequency-sorting, Morse code, and Huffman coding have been explained many times by many presenters. The new, exciting part is Line Coding. You could assume Morse, Huffman, etc codes to be already known and expand Line Coding into an article of its own. I like how you compared how a one-bit error affects various codes. I haven’t seen that detail as much in other explanations of the topic. You could improve the “Frequency Analysis” section with a second diagram, showing the relative frequencies of each letter, sorted. The x-axis there would be not in alphabetical order but e, t, a, o, etc, to show how you pick the encoding of each letter. In many parts, the sloppy grammar makes it harder to read. E.g.
Punctuation adds delay into our communication, adding to the time spent in spending a message when we do nothing, it would be much better if we could just flash dots and dashes consecutively. This delay is essentially acting as a third symbol (like ) for our codewords. And so, Morse code can’t be used for storing into today’s memories as they strictly support only two symbols (which we denote by and ). could be improved to These various levels of punctuation slow down our communication for relatively little information. It’d be nice if we could just flash dots and dashes consecutively, without long pauses. The punctuation between letters acts like a third symbol, besides the dot and dash already used. That means Morse code can’t be stored in typical computer memory, as they only support two symbols, often called 0 and 1. If English isn’t your first language, you might write carefully in what is, then see how a machine translator words it in English.
This is a nice article overall! It’s well structured, easy-to understand and hints at future directions to explore. Some interactive visualizations would be great to make it more engaging for the reader.
Nice introduction to coding theory; I liked your post, although here are some details I would improve:
Firstly, I think too much time was devoted to the first section: The problems and ideas exposed there are (I believe) widely known, so it could have been covered briefly and as a review.
Following this same line, perhaps your article could have gone a little more in depth, as in showing some fundamental theorems in the literature as well as possibly their proofs. Most of what was shown (except for Huffman codes) is standard even to someone not familiar with the topic. Hamming and Solomon codes would have been a nice inclusion, although I would understand if the request is a bit far-fetched.
Finally, since your publication is contained in a website, you could add some interactivity to it, especially when explaining a topic so related to computers. For instance, a widget which encodes any given message into one of the described codes.
Nevertheless, as I have already mentioned, your post is a good reading; thanks for sharing.
I thought the progression from the initial blinking scheme through frequency analysis, Morse code, fixed-length coding, prefix-free codes, and finally Huffman coding was really well done. Each new scheme was motivated by a limitation of the previous one.
I liked the tree visualizations. Seeing the Morse decoding tree made the prefix problem very concrete. The observation of internal nodes in the tree causing the code to not be prefix free was cool, and made it kinda trivial to see why huffman is prefix-free.
The main place I would have liked more detail was the Huffman section itself. Since Huffman coding is really the culmination of the preceding progression, I think the tree-generation example would benefit from being walked through step by step rather than mainly being presented as an image. The written algorithm is understandable, but notions like merging the least frequent symbols into a partial tree and reinserting it with their combined frequency may not be immediately intuitive to someone seeing Huffman coding for the first time.
I also think the claim that Huffman coding is “optimal” could be stated more precisely and justified a little more. Optimal with respect to what objective and among what class of codes? I’d have loved to see a proof sketch or an intuitive explanation on why this is “optimal” and would give this section more mathematical depth.
Overall, though, I thought this was a solid introduction. Someone encountering these ideas for the first time would probably get a lot out of the progression and the visuals.
