Breaking Unbreakable Codes: The Evolution of Cryptography

Transcript

[01:00:00.07]
Ned: The front legs will switch positions on the body. The front right leg will suddenly become the front left leg.

[01:00:11.01]
Chris: So it's like quap, but worse.

[01:00:14.20]
Ned: Yeah. So I think we're all right for a while unless you really want something surreal. And then maybe that's the thing for you. That's your jam.

[01:00:25.07]
Chris: That's true. We're not here to harsh on anybody's... What's the expression? Buzz?

[01:00:31.09]
Ned: Kink? We'll workshop it.

[01:00:34.15]
Chris: Jumbalaya.

[01:00:46.19]
Ned: Hello, alleged human. Welcome to the Chaos Lever podcast. My name is Ned, and I'm definitely not a robot. I have four limbs that do not swap positions for no apparent reason. For that reason, you know that I am real and not generated by AI. With me is Chris, who's also not generated with AI. Hi, Chris.

[01:01:10.25]
Chris: Who am I?

[01:01:13.11]
Ned: I didn't say anything about Alzheimer's. That's a different capital A.

[01:01:18.29]
Chris: Who am I and who am I?

[01:01:21.22]
Ned: Now you're just an existential bot.

[01:01:24.17]
Chris: No, that's M-I.

[01:01:27.27]
Ned: Think about it. Don't.

[01:01:30.20]
Chris: Don't think about it.

[01:01:31.23]
Ned: Please don't think about it. Ai is going to burn down the world, and all we're going to get out of it is really terrible video content.

[01:01:43.15]
Chris: Well, that's a relief. Last thing on that topic, did you see the Coca-Cola commercial that was 100% generated by AI?

[01:01:53.07]
Ned: I did not. Did it have something to do with a polar bear? Can I stick in with that Christmas theme?

[01:01:58.10]
Chris: Sort of. Okay. I also don't want to say more about it because I want you to experience it in its mastery just without any preconceived notions.

[01:02:09.05]
Ned: You don't want to ruin the surprise is what you're saying.

[01:02:11.05]
Chris: Then I'm going to need you to read the comments because That's where the magic happens.

[01:02:17.15]
Ned: Isn't that always the case? Okay, well, that is my task after we finish today's recording is to go watch that and then maybe just pour one out.

[01:02:29.26]
Chris: I expect you to live tweet the experience.

[01:02:33.29]
Ned: Live skeet? I'm on Blue Sky now, so I'm skeeting.

[01:02:37.29]
Chris: Now, we got to come up with a better verb.

[01:02:40.24]
Ned: We have already lost that opportunity, and it has been decided for us that that is the verb. Honestly, it could have been worse.

[01:02:51.06]
Chris: So you know young people, some of whom you keep locked up in your house, which is a weird way to talk about parenting.

[01:03:00.01]
Ned: But frighteningly accurate.

[01:03:01.27]
Chris: So if something's a vibe, that's good, right? According to the young people?

[01:03:10.28]
Ned: It could be either.

[01:03:12.21]
Chris: Okay, so that's what I remember. This person's giving off bad vibes. This person's giving off good vibes. But now something is a vibe. Right. And as I speak, I realize that I am, in fact, 100 years old.

[01:03:26.29]
Ned: And you are a vibe. You're both. The circle has been completed, sir.

[01:03:34.08]
Chris: Okay. I forget what my question was, but we should probably get going because I got a lot of words.

[01:03:40.16]
Ned: I did notice that the word count was excessively high, and you promised this was going to be a two-part series, and I don't believe you.

[01:03:48.08]
Chris: I have this long history of lying to both myself and others.

[01:03:53.17]
Ned: Yes. Well, I'm glad that you're keeping up with it. Being true to yourself, if not anybody else.

[01:04:01.07]
Chris: Consistency, just like we just talked about, I think.

[01:04:05.01]
Ned: I wasn't doing that. I don't know. I wasn't paying attention. Go ahead.

[01:04:10.11]
Chris: Anyway, we are going to continue our discussion of the history of cryptography, this being part two. Part Two. Part One, if we did this correctly, should have been released last week, I think?

[01:04:23.19]
Ned: It was. I did things. Neat.

[01:04:26.15]
Chris: So if listeners want to catch up on all of the history that predates what we're going to talk about today, go listen to that one. That one goes first. In that one, we talked about... Well, we started with prehistory, which in retrospect, might have been a mistake.

[01:04:46.20]
Ned: I don't think so. It's not on.

[01:04:50.11]
Chris: We hit some of the oldest hieroglyphs that are available to us in Egypt that showed modifications that intended to share secret with people that were only in the know. Ancient wars, like in Greece and Rome, were the inspiration for things like the Sightail Cipher. That's the one where you wrap the message around a stick. And the Caesar Cipher, also known today as Rot 13. That's the one that Ned had absolutely no idea about and had never heard of before.

[01:05:21.07]
Ned: Sure.

[01:05:22.02]
Chris: And then we talked about the political intrigues that utilize ever more sophisticated types of codes, culminating in one of the most well-known code-breaking events of the time, that being the story of Mary, Queen of Scots.

[01:05:34.26]
Ned: A thing that I did not know about. And now I think there's a whole Netflix series I have to watch.

[01:05:39.27]
Chris: Probably many.

[01:05:41.11]
Ned: Keep giving me homework. Hell.

[01:05:44.11]
Chris: Also, I do want to take a second before we get started to note that I do realize that this has been and will continue to be a very Western breakdown of what is unquestionably a universal topic. There have been codes, encryptions, ciphers, et in all languages and in all countries all throughout history. Couple of quick examples. In India, in 400 BC, we have evidence of them using substitution ciphers. Ancient China uses complex codes similar to the Egyptian example where utilizing the complexity of the written language, they can manipulate the characters to tell secret tales.

[01:06:22.25]
Ned: Okay.

[01:06:23.17]
Chris: And as a matter of fact, it was in 19th... No, not 19th, ninth. It was ninth century Arabia that saw Alkindi Pioneer Frequency Analysis, about which much has been said, as well as the polyalphabetic cipher, about which you will hear more very soon.

[01:06:42.04]
Ned: All right.

[01:06:43.28]
Chris: I don't want to dwell on it, but I definitely wanted to point out whole wide world out there and everything.

[01:06:50.07]
Ned: If you happen to be an expert on nonwestern cryptography and you want to come on and talk about it, let us know. We love that.

[01:06:58.21]
Chris: Even if If you just want to wave your finger at me for 45 minutes, I'm completely fine with it. I think, you know.

[01:07:07.01]
Ned: That's bonus content right there.

[01:07:08.22]
Chris: Plenty of opportunity. Maybe that'll go on Patreon.

[01:07:11.14]
Ned: We have a Patreon? Sure. No. That was it.

[01:07:16.22]
Chris: Anyway, so let's pick back up as codes and code-breaking go ever more profoundly into the world of nation-state intrigue and world politics. We are going to fast forward to the 17th and 18th centuries. First thing to remember, just like there have always been code makers, there have always been code breakers. It's a bit of a race, which helps explain why codes continue to get complicated. In the 17th and 18th centuries, across Europe, that's exactly what happened. In the first part, we talked about code-breaking ideas like frequency analysis, where For example, you could identify the letter E. It's the historically the most common letter in English. In order to break a code, just count up all the times the different letters show up. If E was substituted by something else, you could make a reasonable estimation of which letter was E in the new code. Now, something that came about to counter this is called a polyalphabetic cipher, wherein you use multiple substitution alphabets or mechanisms during one encryption. This way, E might be coded as X the first time it comes around, and then later on, it might be coded as W. Various keys and known secret contrivances dictated when the different alphabets would be used or shifted to, making it that much harder to break.

[01:08:52.18]
Chris: The most famous version of this from the time is called the Vignère Cipher. This was created in France, and the cipher was time consuming to encrypt and decrypt, but it was considered unbreakable for something like 250 years.

[01:09:07.18]
Ned: That's pretty good, if true.

[01:09:09.27]
Chris: Yeah, and it turns out that it is true, but more on that later. So I'm going to try to explain how these codes work and why they're so different than the simple substitution ciphers that we've talked about before. But I will note that it's really hard to explain without visuals.

[01:09:29.27]
Ned: In order to- For those who are watching this on YouTube, I will put up visuals. By the way, did you know that we have this on YouTube as well? If you're watching on YouTube, you'll have visuals. If not, just use your mind's eye.

[01:09:43.12]
Chris: I should have put my hair on. No, that's cool. I didn't know we could do that. Neat.

[01:09:52.13]
Ned: More work for me, but I'll do it in the service of good content, Chris.

[01:09:58.28]
Chris: So what I'm going to do is I'm going to simplify the concept and talk about a polyalphabetic code, but I'm going to use the example based on ROT 13. Now, like I said, this is way more complicated in real life, but this gets to the idea. So with a rotational cipher, you simply substitute each letter one for one based on an agreed-upon number with a letter further down the alphabet. So if your number is three, A's become D's, B's become E's, et cetera. Pretty simple. So if you use that a code using the key number three, a set of letters, A, A, A, A, Now this is where you can see why frequency analysis is so powerful, because that letter is the same thing every single time. That gives you an idea of what you're trying to decode. With a polyalphabetic cipher, though, we can mock this up something serious, and that is a technical term. Yes. Okay, now this is the part where it gets weird, so try to stick with me. Let's say instead of a single key of three, we decide to use three separate keys, one after another in a repeating fashion.

[01:11:17.26]
Chris: Okay, so let's use the key one, two, three.

[01:11:20.23]
Ned: Okay.

[01:11:21.15]
Chris: Now, the first character, A, in our A, A, A, A, A example, the first character, A becomes B because one. The second character, we move on to the second part of the key, which is the number two. So the second A becomes C. Then we have the third character. Third number is three in the key. So that becomes D. Then you start over. So doing this with a polyalphabetic cipher in this ROT 13 style, using the key one, two, three on that same set of letters, A, A, A, A, becomes BCDBC.

[01:12:00.08]
Ned: Right. Okay. Does that make sense? I follow. Yeah.

[01:12:03.08]
Chris: So this, if you just gave that to somebody to try to decrypt, is a whole hell of a lot harder to understand.

[01:12:11.14]
Ned: Yeah, just a little bit.

[01:12:14.04]
Chris: So I do have an example that I put together that hopefully Ned will show on the screen that shows the power of this. So let's take a simple message like, the attack begins at three, aim east. If you simply rotate it using the key three, what you can highlight is all of the E's in the message show up as Hs. Right. And this is also the longer the code, the better it is for the code breaker because they have more information to work with. A short message like A, A, A, A, A, whatever. It could just be go. Who knows? But when you have more letters, you can start to do frequency analysis. Now, when you do a polyalphabetic cipher like this, and you can see this on the screen, I hope, the E's are not just H's. It's an H, it's a G, it's an F, depending on where they are and depending when the key rolled back around again. Thus, if somebody tried to do frequency analysis on this, they would get nowhere. You would end up with gibberish, which is, from a code maker's perspective, It's good.

[01:13:30.21]
Ned: Yeah, that's what they're looking for, is to make it much, much harder to crack.

[01:13:36.02]
Chris: Right. And again, this is a crazy simplified version. In real life, first off, the alphabets were more complex. The keys were way longer than three characters, and there's a whole lot of other complications that could be thrown in there. Links in the show notes to show you in great detail how the cipher acted in real life, and also a website where you can create your own, which is actually pretty fun.

[01:14:02.20]
Ned: All right.

[01:14:04.02]
Chris: Let's get back to practical matters. So Napoleon. Heard of him?

[01:14:11.15]
Ned: Something about Ziggy Piggy?

[01:14:13.11]
Chris: The guy with the hat? A big deal for a few years in the early 1800s?

[01:14:18.23]
Ned: Yeah, yeah, yeah. That sounds familiar. Bill and Ted helped me understand where he fits into history.

[01:14:24.26]
Chris: One of the best documentaries of our time.

[01:14:27.18]
Ned: Absolutely.

[01:14:29.17]
Chris: He We, Napoleon, had an army that was pretty huge, and it needed to communicate. Now, remember, we're talking about some of the historical encryptions that existed in the time. There was one called the Great Cipher, or the Grand Chiffre, which you think is how it's pronounced in French. But these other ciphers, the problem with them was they were complicated, and it took forever to make the codes. So Napoleon did the only thing that he thought was and just went ahead and created his own.

[01:15:03.26]
Ned: Sure.

[01:15:04.26]
Chris: It was not of this style. It was not polyalphabetic because, again, that took too long. But Napoleon had his armies use it on the battlefield because they thought that it was unbreakable. There's a certain amount of optimism that goes through a lot of Napoleon's life, doesn't always work out for him.

[01:15:25.22]
Ned: In this case- It's like the story ends poorly for him, if I remember correctly.

[01:15:30.03]
Chris: Yeah, a lot of arsenic. A lot of arsenic. But anyway, long story short, too late. I'm sure you know where I'm going with this. The code that he created, totally broken. It was an iterative process, but it was like a period of months, if not sooner.

[01:15:49.02]
Ned: That is too quick.

[01:15:50.13]
Chris: Too quick. Yeah. They broke the code in or around 1811. A guy who was on Wellington staff named George Scovel, esoteric character. He wasn't even like an official codebreaker. He was just a guy that liked puzzles, and he broke it. Faster than the French expected, and the breaking of this code allowed the British to anticipate French movement, including at the Battle of Salamanca, where decrypted messages played a crucial role in securing the British a crushing victory, basically ending every aspect of the idea of having the war in Spain, which was stupid in the first place, but I'm getting ahead of myself.

[01:16:33.08]
Ned: Not the last time that cracking codes for the British led to victory.

[01:16:39.25]
Chris: Correct. So funny story about Scoville. He was present at that battle. Like I said, he was on Wellington staff.

[01:16:49.21]
Ned: Okay.

[01:16:50.08]
Chris: He was breaking codes as always. Yet they found him just wandering around the front lines of the battlefield with the decoded French cipher in his pocket. So this, I'm sure all intelligence officers can tell you, this is a bad idea. You shouldn't do that.

[01:17:12.13]
Ned: Right.

[01:17:13.02]
Chris: Generally, you don't want the enemy to know that the code has been broken, nor do you want to expose the primary codebreaker to the risk of death.

[01:17:24.13]
Ned: Yeah, it seems bad.

[01:17:26.22]
Chris: In any event, he didn't get captured or deathed, but just an interesting character. I had more about him, but I'm going to carry on.

[01:17:37.26]
Ned: All right.

[01:17:40.05]
Chris: Let's fast forward 50-ish years and move 10,000 miles left, I think. I don't have a map.

[01:17:50.22]
Ned: All right.

[01:17:51.24]
Chris: For one of the last pre-machine code examples, and we're talking about the US Army in the Civil War. And what's interesting here is rather than dramatically changing the what of the encryption and the usage of it, what the war did was change the how.

[01:18:12.17]
Ned: Okay.

[01:18:13.10]
Chris: Now, remember how we talked about those codes earlier being complicated and really hard to encrypt and decrypt?

[01:18:20.07]
Ned: Yeah.

[01:18:20.20]
Chris: What the US wanted to do was still use a complicated code, but make it faster to decrypt. And what they came up with was their version of the cipher disk. Now, I say their version because cipher disks had been invented in Italy way back in the 15th century. This was a known idea. It's a simple device. You've got a circle, and on the outside of the circle, you've got letters. Then you have a smaller disk that sits on top of it with other letters. The outside disk stays static. It doesn't move. The inside one, you can turn. So think about the ROT 13 example. If the outside is the alphabet, the inside is the alphabet, you turn it three notches. Now you can see with a glance, A becomes D, B becomes E, et cetera. In your head, you can probably do that for a lot of letters. If I go D plus three, you're going to figure it out right away. But can you do that with K or F? No, not off the top of your head. That's what these disks did, was you were able to turn it to the right place and just look at a glance and see what the decoded next step was.

[01:19:44.17]
Chris: No cables, no paper that you had to carry around. These were small dials that you could stick in your vest pocket. If you were afraid of being captured, you could just spin it to something random, so nobody would know what code or what key you were looking at previously.

[01:20:05.15]
Ned: Sure. I mean, still, if it's just a straight alphabetic cipher, it's still got all the problems with just being a straight mapping of a to three letters ahead. But yeah, I mean, at the very least, you wouldn't know that this is the code we're using today or this week. Right.

[01:20:26.05]
Chris: And again, I'm simplifying. What I explained was the easy version. The hard version, which was the one that was actually used by the Union in the Civil War, used the outside that was a numeric four-digit code, and the inside was a wildly scrambled alphabet with different characters.

[01:20:44.11]
Ned: Okay.

[01:20:45.21]
Chris: Now, the four-digit codes were actually only the numbers either one or two. So 1111, 1112, 1121. You get the idea.

[01:20:56.21]
Ned: I do.

[01:20:57.05]
Chris: The reason that they did this was Now the code was greatly simplified, so you could use it extremely quickly on things like the telegraph or on the battlefield using flags.

[01:21:12.13]
Ned: Okay.

[01:21:13.13]
Chris: Before you ask why that would even come up, I would politely remind you that battlefields are big, and the battles, especially in this war, lasted a long time. The Battle of Gettysburg took place over three days, and it was not the only multi-day conflict. So what you would end up with is parts of your army scattered all over the place. They need to communicate with each other, but you can't trust that a messenger is going to get there. Thus, using the disk and the numbered code with flags, you can make signs that they easily be seen across the battlefield. And since it was very quickly encrypted and decrypted, it gets across securely.

[01:21:59.26]
Ned: Right. Because the enemy can also see the flags, so they can see what you're saying, but they can't understand what you're saying.

[01:22:08.05]
Chris: Correct. Which is always, and we talked about this before, a lot of this innovation comes from either political intrigue or war, and it almost always has to do with how much encryption is good enough to get the message across securely before that message stops being useful.

[01:22:26.19]
Ned: Right.

[01:22:27.23]
Chris: If somebody decrypts this message three days from now, we don't really care.

[01:22:32.02]
Ned: Yeah, what mattered was speed. How fast can you encrypt the message and transmit it via the flags and then decrypt it? If it can be broken in two hours, it doesn't matter because that message is not relevant in two hours anymore.

[01:22:45.14]
Chris: Exactly. That's why in these types of codes were used in war and things like the Vigneer Cipher were used in things like diplomacy because you could decrypt that message in as much time as you wanted to. It wasn't time sensitive in the same way. Now, since I mentioned the Vigneer Cipher again, I want to close with talking about something that was happening back in England. Namely, Charles Babbage was happening.

[01:23:12.11]
Ned: He was happening.

[01:23:14.20]
Chris: He, of analytical engine fame, in ways that people probably don't know about, also helped cryptography. Because, of course, he did.

[01:23:26.20]
Ned: Well, they're all interrelated, right? Computing, cryptography, cryptography and math. So if you were the person who really liked math, you were probably involved in all of these things.

[01:23:37.17]
Chris: True. And also, like the story we're telling before, some people get into cryptography because it's a fun puzzle to unpack. They don't necessarily care about war or political intrig. They just want to solve the thing.

[01:23:49.29]
Ned: A bunch of weirdos.

[01:23:51.23]
Chris: It's like Werdle in almost no ways.

[01:23:54.03]
Ned: Not like Werdle at all, but okay.

[01:23:59.15]
Chris: So Remember, back in the Napoleon section, we said that the Veneer cipher was considered unbreakable. Babbage broke it, which is crazy because it was about 250 years of that code being impregnable.

[01:24:15.02]
Ned: Right.

[01:24:16.16]
Chris: So how did he do that? So in short, he figured out that when you have a key, the one, two, three code from example before, it repeats. The key itself eventually repeats. You get to the end, you start from the beginning again.

[01:24:32.24]
Ned: Sure.

[01:24:33.13]
Chris: That's the weakness. So you find places where the plain text aligns with the same set of letters over and over again. Then you can deduce the length of the key. So in order to explain this a little bit better, I'm going to go back to our veneer example. The attack begins at dawn, aim east. When it's put through the very simple veneer confusion, it turns into gibberish. But if you look very carefully, the letters A and T in the word attack and the letters A and T in the word at, they repeat. And in this case, it's just luck that it hits in exactly the right place. But if you look at the encoded version, A, T, is B, V in both places. That's the thing that Bab which figured out, obviously in way more complicated fashion. But he was able to figure out, and then you can do some mathematical experimentation once you have a hypothesis. You run that through this math based on probabilities, testing the various key lengths, eventually discovering the key length with 100% probability. So that's the short version of what he did. Does that make any sense at all?

[01:26:00.23]
Ned: It does. I'm almost envisioning you could take the lines of text and just keep shifting them until you end up with these ways that they line up in terms of translation. And then you're going, well, I know the is just a very, very common word. And so if I see the represented a bunch of times with the same group of letters, I now can figure out how long the key is for it to rotate back so the would be the same combination of letters again. And because it's so common, it's probably going to come up if the message is sufficiently long.

[01:26:35.11]
Chris: Yeah. And that became an interesting problem with... We talked about with the Mary, Queen of Scot's example, in the substitution ciphers, they would use a single letter for words like the to try to avoid exactly what you're talking about. But again, it's code makers versus code breakers. And here's the thing about Babbage and what he did. First off, side note, his accomplishments didn't even become public knowledge for like 100 years after his death.

[01:27:09.13]
Ned: That is not surprising. I'm sure whatever the agency was in England at the time was not about to make any of this public.

[01:27:17.22]
Chris: Yeah, we're talking 1850s, so it was like early generations. It's probably MI1.

[01:27:23.04]
Ned: Mi0.

[01:27:25.14]
Chris: Mi not because British. Yes. Because British. You get it.

[01:27:32.10]
Ned: I get things.

[01:27:33.21]
Chris: What Babbage was able to do, and his accomplishment, it went beyond just breaking a single cipher in a vacuum. It was a systematic mathematical method that foreshadowed all modern crypto analysis to come. The way that he did it could be applied to any code. It wasn't a one-off situation. It wasn't boutique crypto analysis. It was something that any veneer cipher could be broken using this method, which was a game changer. It also set the stage for allowing machines to do encryption and decryption. Because, again, it's not bespoke. It's based on you figure out the code and then you run all these things against it, and eventually you get the correct answer. Sounds like the thing that a computer can do.

[01:28:24.10]
Ned: Sounds like something that the bomb machines were doing.

[01:28:28.24]
Chris: And that is where we'll pick back up in part three.

[01:28:36.00]
Ned: You set this table.

[01:28:40.23]
Chris: We got to this point. There was so much else that I wanted to put in.

[01:28:46.02]
Ned: I know.

[01:28:46.23]
Chris: This could very easily have been a three-hour episode.

[01:28:49.26]
Ned: Maybe it shall be at some point. I'll put them all together as a playlist or something.

[01:28:55.27]
Chris: There we go. Okay.

[01:28:58.14]
Ned: Hey, thanks for listening or something. I guess you found it worthwhile enough if you made it all the way to the end. So congratulations to you, friend. You accomplished something today. Now you can go sit on the couch, fire up a veneer, sorry for, and send an encrypted message to your loved ones. Maybe it'll be Merry Christmas. Who knows? You've earned it. You can find more about the show by visiting our LinkedIn page. Just go to Chaos Lever or go to our website, chaoslever. Com. By the way, we have a listener survey currently going on that we're going to close out at the end of the year. We want to know your thoughts, your feelings, and your praise about what a great job we do. Just go to chaoslever. Com/survey and take the survey. It's only eight questions. It is very quick, and we would appreciate it. We'll be back next week to see what fresh hell is upon us. Tata for now.

[01:29:59.11]
Chris: I'll just let everybody know that the original version was 25 questions. Ned was asking about your favorite peanut butter. It got wildly out of control.

[01:30:08.05]
Ned: The only answer is crunchy.

[01:30:10.09]
Chris: On that, we agree.