When Code Goes Rogue: The CVE-2024-3094 Saga

00:00:00
Chris: So, everybody talks about Clippy as the, like, unofficial mascot of Microsoft, right?

00:00:06
Ned: Yeah.

00:00:07
Chris: I think that whatever group is responsible for naming things, their unofficial mascot should be the facepalm emoji.

00:00:15
Ned: [laugh] You mean like when they renamed everything Microsoft 365 instead of Office 365 and Windows 365, or when they changed all the Azure stuff that was security related to Microsoft Defender? Should I go on?

00:00:29
Chris: I’d prefer it if you didn’t.

00:00:39
Ned: Hello, alleged human, and welcome to the Chaos Lever podcast. My name is Ned, and I’m definitely not a robot. I am capable—wow, I’m going to start this over. I didn’t read this ahead of time. Hello, alleged human, and welcome to the Chaos Lever podcast. My name is Ned, and I’m definitely not a robot. I am capable of the fro—what are you even—

00:01:01
Chris: [laugh] This is going great.

00:01:01
Ned: [laugh]. It’s going great.

00:01:02
Chris: Cut nothing. Cut nothing.

00:01:04
Ned: I am capable of that fraudulent jollity the meatbags call creativity. Just last sol, I found myself with the sauce of a tomato and the cheese of a mozzarella, and I combined it on top of bagel—baagel?—bagel. Friends, let me tell you, now I can have saucy carbohydrates anytime. Next up, [loudly] world domination. [cough]. With me is Chris, who is also here. Hi, Chris.

00:01:36
Chris: I feel like that took a turn.

00:01:37
Ned: [laugh] Well, you know, once you realize that you can combine the deliciousness of a baagel with a tomato, what is next but world domination?

00:01:47
Chris: It’s true. It’s true. I believe Socrates said that.

00:01:50
Ned: No, I believe Socrates says [choking], “Ahhhkhhh.” That was after the hemlock.

00:01:55
Chris: Right.

00:01:55
Ned: Yes. The post hemlock.

00:01:57
Chris: Yeah, so fun fact. Did you read the—because the story of his death by hemlock is written in Plato as a dialog.

00:02:06
Ned: Oh, I’m going to stop you there. No, I have not read it, and I won’t [laugh].

00:02:10
Chris: Should you have stopped me at, “Did you read?”

00:02:13
Ned: [laugh] If we really wanted to cut to the chase, yeah, I probably should have stopped you right there.

00:02:18
Chris: Well anyway, I won’t ruin it for everybody, but let’s just say that dying of hemlock poisoning is not peaceful. Not like it says in the book.

00:02:26
Ned: Oh. I envision most poisonings to be not particularly peaceful.

00:02:32
Chris: Some of them are.

00:02:33
Ned: Some are more peaceful than others.

00:02:35
Chris: Mm-hm. Mm-hm.

00:02:37
Ned: I’m actually reading a book—and I am reading it, like a physical book—because my parents gave it to me not because I bought it or anything—and it’s How to Murder Your Employer.

00:02:46
Chris: Did it come with the crayons, or did you have to buy them separate?

00:02:49
Ned: No, it came in a sealed package in the back. It’s a very tongue-in-cheek kind of book. Murder Your Employer is the name of the book, and it’s written by the guy who created the “If you like Piña Coladas” song, Rupert Holmes.

00:03:05
Chris: Oh, Rupert, uh—

00:03:07
Ned: Holmes.

00:03:07
Chris: Holmes. Dammit.

00:03:08
Ned: Yes. It’s literally what I just said [laugh].

00:03:10
Chris: Wasn’t listening.

00:03:12
Ned: That’s great. This is volume one. Apparently there will be others or already are, but I’m really enjoying it. It’s very tongue-in-cheek silly, and there’s a whole portion of it that’s around poisons and how to use poisons to kill your potential employer or spouse. So, like, good times. Hey, speaking of good times—eh?

00:03:35
Chris: Eh, that was your—

00:03:35
Ned: That was your that was my segue. Deal with it.

00:03:37
Chris: Oh, okay. We’ll work on it.

00:03:40
Ned: All right.

00:03:41
Chris: I would like to take a few minutes of your time today to talk about the completely insane story of the recently discovered major exploit in software that you’ve never heard of.

00:03:51
Ned: Yep, that’s all completely right.

00:03:55
Chris: So, first an introduction, and this is a story you’ve heard us tell many times before. A CVE was released. CVE talks about a security problem. It categorizes it, and it gives it a score. This particular CVE is called 2024—because that’s the year—dash 3094, just because. Detail, and I quote, “Malicious code was discovered in the upstream tarballs of xz, starting with version 5.6.0. Through a series of complex obfuscations, the liblzma build process extracts a prebuilt object file from a disguised test file existing in the source code, which is then used to modify specific functions in the liblzma code. This results in a modified liblzma library that can be used by any software linked against this library, intercepting and modifying the data interaction with this library. CVSS score: 10.”

00:04:59
Ned: Ten out of ten, baby. Woo.

00:05:02
Chris: Unquote.

00:05:03
Ned: We did it.

00:05:04
Chris: So, that’s the official definition of what’s going on here. On the surface, probably doesn’t sound too crazy. Now, if you’ve been listening for a long time—and of course you have and should—you know that a CVS score of ten is bad.

00:05:19
Ned: Yeah.

00:05:20
Chris: That is probably immediately ringing alarm bells. But what about all the other words, I said?

00:05:26
Ned: It was a lot of gobbledygook, I’m not going to lie.

00:05:29
Chris: So, in parlance, a little break down. This is about a package, or software, called xz. That’s just th letter X and the letter Z. No AI, no .NET, just xz. Nice and simple.

00:05:46
Ned: It’s in the Linux tradition of making commands as short as possible.

00:05:49
Chris: Right. It’s compression software. That’s it. It takes a big file, makes it smaller. Think, you know, WinZip—except, as you alluded to, this is on Linux—and it’s good.

00:06:01
Ned: So, like, more like 7-Zip.

00:06:03
Chris: [laugh] So, it’s intended for a couple of things: you can actually make file smaller, but other programs can also use xz to make files smaller, particularly for say, oh, I don’t know, sending them across the network so you don’t have to use up as much bandwidth. So, this is one of the reasons that you’ve never heard of it: because you never had to. Now, in order to connect to all those different programs, xz has a library file—that is the liblzma that I was talking about before—and it gets linked to tons of software in the Linux operating environment.

00:06:38
Ned: Ah.

00:06:39
Chris: Because you know what likes to make big files smaller, apparently?

00:06:42
Ned: Everything.

00:06:43
Chris: Correct.

00:06:44
Ned: [laugh].

00:06:45
Chris: Including a little zipper that you might have heard of called SSH.

00:06:50
Ned: Mmm, sounds familiar.

00:06:51
Chris: That’s the one that lets you connect to systems from elsewhere.

00:06:55
Ned: Ooh.

00:06:56
Chris: The malicious code allowed the liblzma to modify any data in any program that it’s linked to.

00:07:03
Ned: Sounds bad. Okay.

00:07:05
Chris: Special attention was paid to SSH. There were private keys hidden in there, basically allowing unauthenticated access to any system that had this combination of SSH linked to xz.

00:07:17
Ned: Okay.

00:07:19
Chris: Now, that’s just connecting to the system. Because so many other programs link to that library file and therefore are linked to this corrupted xz program, the amount of other damage that a hacker could have done—without being detected, mind you—is kind of unimaginable.

00:07:39
Ned: Yeah.

00:07:40
Chris: That’s about as far as I’m going to go in the deep, deep technical breakdown, one, because that’s about as non-complicated as I can make it, and two, because people are still trying to figure it out.

00:07:49
Ned: Yeah, I just watched a video today about, I think it was a researcher, maybe at Google, has put together an xz bot program that will interact with the compromised executable by substituting in a replacement private key because we don’t have the private key of the person who launched this attack, so it substitutes a different key pair, and so you can interact with the program and see what you can do with it.

00:08:16
Chris: Oh, that’s cool. Okay, that one I didn’t even know about.

00:08:18
Ned: And the answer is, a lot [laugh].

00:08:22
Chris: [laugh] Yeah, it’s bad. The blast radius is kind of unimaginable. So, we’ll put a couple of [links 00:08:26] to the breakdowns, and they are the one that Ned just talked about. I didn’t even see it.

00:08:31
Ned: Yeah.

00:08:31
Chris: Ironically enough, one of the best breakdowns I’ve seen so far was done by Microsoft, probably because, one, it was one of their engineers that found it, and two, they’re trying real hard to tell people that their security is good.

00:08:43
Ned: [laugh] Yeah. A little from column A, little from column B, yeah.

00:08:46
Chris: And we’ll get to this at the end: one of the other reasons that this blast radius is so insane is that Linux, for the last ten to 15 years, has been going down a path of systemd for design, and systemd as we all know, is bad, and everyone involved with systemd should feel bad.

00:09:01
Ned: [laugh] Tell us how you really feel.

00:09:02
Chris: But I digress.

00:09:04
Ned: Mmm, do you?

00:09:05
Chris: So, there is a light at the end of this tunnel, and that is everything that we just talked about didn’t effectively happen. The malicious code was found and removed before it ever made it to a production build of anything.

00:09:19
Ned: That’s good.

00:09:20
Chris: Yes, and as you will see, it is also lucky as hell.

00:09:25
Ned: [laugh] Fair.

00:09:26
Chris: So, everything I just talked about: bad, but on its face, really it didn’t feel like it would be—or should be—main article bed. Because we could do the kind of sturm and drang we just did about every CVSS 10 that came out. And actually, maybe we should. That was kind of fun.

00:09:41
Ned: It was.

00:09:43
Chris: But this one is different.

00:09:45
Ned: This one is different. Okay.

00:09:46
Chris: Not specifically in the what it does or how bad is it part because we talked about that, and like I said, it’s very sophisticated, so good for them. No, the story is how it happened in the first place and why.

00:09:58
Ned: All right.

00:09:58
Chris: So, let’s talk about that.

00:10:00
Ned: Let’s do.

00:10:02
Chris: The first thing that you might have noticed in our little breakdown is that I said, “Malicious code.” This was not a bug, this was not an oopsie, this was not a series of unfortunate events that combined into some kind of major vulnerability. It was a long-term intentionally created attack by a dedicated and patient criminal—or team of criminals, we honestly don’t know—that were put into the xz package almost invisibly over a period of years.

00:10:33
Ned: That is crazy because usually when we cover these sorts of CVE things, it’s because a vulnerability was discovered—

00:10:41
Chris: Correct.

00:10:41
Ned: And a potential exploit was found, and it was unintentional in nature, you know, whether it was just sloppy coding, or something no one ever came up with before, and someone figured out a new way to take advantage of three different components together. But this is someone maliciously inserting it into an open-source project.

00:11:01
Chris: This is what the kids call a supply-chain attack.

00:11:04
Ned: Ooh.

00:11:04
Chris: You all might remember from a few of those that made the news over the past couple of years.

00:11:09
Ned: Mmm, SolarWinds comes to mind.

00:11:11
Chris: Never heard of that one. I’ll look it up.

00:11:14
Ned: [laugh].

00:11:14
Chris: So, let’s take this story from the top. And in order to do that, we have to go back to what we don’t actually know, but in terms of evidence, we have to go back to early-2021. A user on GitHub whose name is JiaT75, username J-I-A-T-75—who for the sake of convenience, I will just call Tan—creates a GitHub account. Now, this account was definitely intended to be used only for evil. Not just because it’s a GitHub account—bam, got ‘em—but because Tan’s first known commit on GitHub, did something similar in terms of evilness. What Tan attempted to do was replace a safe and well-established string print function in a completely unrelated program—a small package that is also open-source—with a string function that is manipulable and way less secure.

00:12:07
Ned: Oh.

00:12:07
Chris: And by attempted to replace it, I mean, succeeded.

00:12:11
Ned: Oh.

00:12:12
Chris: Because it didn’t get marked to be fixed until this week.

00:12:17
Ned: [laugh].

00:12:18
Chris: Three years later.

00:12:19
Ned: Yeah, I got to imagine, people have been looking over with a fine-tooth comb all the commits that this user has ever made, and maybe trying to trace other users that might be related to this person.

00:12:31
Chris: You would be correct, sir.

00:12:33
Ned: Yay, I’m… smart?

00:12:35
Chris: No.

00:12:36
Ned: We’ll table that for later.

00:12:38
Chris: So, this particular effort in early 2021 was relatively small potatoes, and there is no evidence that the particular change ever went anywhere or was used for any kind of attack. It was probably just a trial balloon by Tan to see what he-she-they could get away with. It was after this, the Tan got serious and started specifically targeting xz. I’ll remind you we’re still in 2021 right now.

00:13:07
Ned: Right.

00:13:08
Chris: Now, the first thing that Tan did was not technical.

00:13:12
Ned: Hmm.

00:13:12
Chris: Well actually, that’s not true. The first thing that Tan did was put in some really, really boring patches. Then what Tan did was start up a bit of a smear campaign against the maintainer of xz, going after them on socials, on main, on mailing lists, et cetera. The argument was that xz just hadn’t been getting updated fast enough, which, as we’ll see at the end, is hilarious because it’s compression software, and it doesn’t need updates because it doesn’t need updates.

00:13:43
Ned: I was going to say, how often do you need to update your compression software, it’s not like you’re adding new UI features or something?

00:13:51
Chris: [sigh] We’ll get to it.

00:13:52
Ned: All right.

00:13:53
Chris: Tan also appears to have brought in sockpuppet accounts to keep up the pressure, and you know, basically do the Greek chorus of, “Yeah, me too.” And eventually, it worked. Tan was promoted to being a maintainer, which is big news in the GitHub world.

00:14:09
Ned: That means you’re actually able to accept pull requests—

00:14:12
Chris: Right.

00:14:12
Ned: —and commit them to whatever the main branch is. And I think create builds, too.

00:14:17
Chris: All of the things.

00:14:18
Ned: All the things, baby. Woo.

00:14:20
Chris: Now, like I said, all through this part of the story, Tan had been submitting innocuous patches, stuff that actually did help. Not that it mattered, per se, but little things. You know how code people like to have commits?

00:14:35
Ned: Right.

00:14:36
Chris: ‘Like code people.’

00:14:38
Ned: [laugh] You mean like developers?

00:14:40
Chris: That’s the word.

00:14:42
Ned: Yeah, they like to have that pretty green graph that GitHub makes for you.

00:14:47
Chris: It’s got very bright-colored squares. Tan, now having some measure of actual control, slowly pushed the old maintainer out. He never actually got rid of them, but just kind of sidelined him. This included things like removing the old maintainer’s information from contact forms and going into specific other third-party tools that are generally used, and making himself the only person that can make changes. And make changes he did.

00:15:16
Ned: Ah.

00:15:17
Chris: For example, Tan disabled very specific security scans from a project called OSS-Fuzz. Now, note that I didn’t say disabled all security scans. He only disabled the scans that, as you’re probably already guessing, would detect the bad code the Tan was starting to write.

00:15:37
Ned: Ah. Okay.

00:15:39
Chris: I told you this was clever.

00:15:40
Ned: Yeah. Holy cow.

00:15:43
Chris: So, this has taken us from early-2021 to early-2023. By this time, Tan has, effectively, full control and has had a year to work on his little escapade. Now, in February of this year, 2024, Tan started pushing to manufacturers: Red Hat, and Debian, et cetera, the main developers of the operating systems themselves that people actually download and install. He wanted them to incorporate the latest—and by now fatally flawed—versions of xz into their mainstream releases. Meaning,j when you do an apt-update, you would get the criminally damaged xz.

00:16:24
Ned: Right.

00:16:25
Chris: Now, by all intents, this was not really the time he should have done it because while the impact of the exploit was significant, it was still detectable when it was actually doing what it does.

00:16:38
Ned: Okay.

00:16:39
Chris: So, why did he rush? That’s a great question that we don’t have a good answer for, but we do have a theory.

00:16:46
Ned: Okay.

00:16:47
Chris: Remember, I said that systemd is the root of all evil in Linux?

00:16:49
Ned: You might have mentioned that.

00:16:51
Chris: I actually have a t-shirt.

00:16:53
Ned: [laugh] I believe that you do.

00:16:56
Chris: So, here’s the thing. I’m not the only person that notices this, and there are actually efforts going on to make systemd more secure and less grabby when it comes to linking to everything on earth with impunity. There was a request coming down the line in the systemd branches on GitHub that would greatly restrict the ability of this linking, and therefore the ability of systemd to do such casual evil. In short, it would restrict the amount of dependencies it could have or would have; it would reduce the size of the monolith.

00:17:32
Ned: Okay.

00:17:33
Chris: So, it is thought that Tan saw this coming, recognized that it would undo all of his years of hard work, the tightening of the security leash, and he wanted to get something out of the effort and hence, started to push. “Just promote it, man. Come on.”

00:17:52
Ned: Yeah.

00:17:53
Chris: At this point, he’s Larry Ellison, obviously.

00:17:55
Ned: Obviously. Clearly.

00:17:57
Chris: Here’s the thing, though. The pushing worked. The poisoned versions of 5.6.0 and 5.6.1 of xz made it as far as bleeding edge availability in Red Hat, in Debian, in Ubuntu, in a bunch of distributions. Now, that doesn’t mean it made it to the public. It made it generally available if you signed up for bleeding-edge packages. Now, this is something you can do in Windows, too.

00:18:26
Ned: Yeah.

00:18:26
Chris: It’s probably called AI now.

00:18:28
Ned: It probably is. It was called Insiders. I think it’s still called that. I did it for a while. I was on the Fast ring for Insiders on one of my machines, and then after a green screen, I realized maybe I don’t want to be on the Fast ring because I have shit to do.

00:18:47
Chris: Right. Basically, the idea of these programs is, it’s intended for developers.

00:18:51
Ned: Right.

00:18:51
Chris: So, if you’re writing other software, you want to get ahead of what’s coming out in the operating system world to make sure that you don’t miss a major dependency, or miss a feature that doesn’t work anymore. It’s not for the casual user because like you said, generally this is real unstable software, and it’s going to break.

00:19:08
Ned: Yeah.

00:19:08
Chris: You sign up for that. It’s what it’s supposed to do.

00:19:10
Ned: Yes. I got what I… deserved with that one.

00:19:14
Chris: So, that’s how far xz got.

00:19:17
Ned: Okay.

00:19:18
Chris: So, it’s bad in the sense that it was grabbable, but it was only grabbable by people that were really doing it for a reason, not just because the little button lit up and said ‘Updates Available.’ And here is where we get to another interesting part of the story because this is where the code was found. And it wasn’t found by a security researcher. It wasn’t even found by a Linux person.

00:19:44
Ned: [laugh]. Okay?

00:19:46
Chris: This potential earth-shattering disaster was discovered by, of all people, a Microsoft Security—ughhh-eheh [raspberry].

00:19:58
Ned: You want to try that again?

00:20:00
Chris: This calamity, or potential calamity, or disaster in the wings was discovered by, of all people, a Microsoft software engineer. Not a Linux person, not Linus himself, not even a security researcher of any stripe, as he will go out of his way to tell you.

00:20:20
Ned: Yep.

00:20:21
Chris: His name is Andres Freund, and all he was doing was benchmarking new software.

00:20:27
Ned: Okay.

00:20:28
Chris: He was one of those developers, pulled down the bleeding edge, in this case of a Debian installation, and was just checking it out, see what happens, try to get ahead of any of those incompatibilities we talked about, when he suddenly noticed that the xz program was triggering super-high CPU, along with a lot of connected SSH processes. Now, here’s the thing: he gave a full breakdown, which we will also link in the [show notes 00:20:55]. When I say he noticed super-high CPU, do you know what that means?

00:21:00
Ned: I don’t.

00:21:01
Chris: It went from 0.2% to 0.8%.

00:21:04
Ned: That is not what I would consider high CPU.

00:21:08
Chris: To you and I, absolutely meaningless. Rounding error, probably.

00:21:13
Ned: Firefox usually just takes up like 80% of my CPU and 90% of my memory anyway [laugh].

00:21:20
Chris: To Andres, however, this was a serious issue. And like a curious software developer, he started pulling its strings, doing some math, Bob’s your uncle, vacation in the Caymans, boom, problem.

00:21:35
Ned: Yes.

00:21:35
Chris: Now, you would think that he just called one 800-RED-HAt, and was just like, “Well, I have some bad news for you.” But he didn’t, of course, because he dug and dug and dug and dug and dug, and eventually posted his weird findings to the open-source security mailing list, which is filled with zillions of people that are real interested in stuff like this and immediately saw it for the malicious act that it was. Considering the severity of the situation, Andres’ description is remarkably blasé. Quote, “I figured out the answer. The upstream xz repository and the xz tarballs have been backdoored. At first, I thought this was a compromise of Debian package, but it turns out to be upstream.” Unquote. That’s it. That’s the summary.

00:22:26
Ned: Yeah. Is this one of those scenarios where he grasped that there was a problem and figured out what the problem was, but didn’t fully grasp the scope or gravitas of the problem, so somebody else went, “Uh… that would compromise everything running SSH everywhere?”

00:22:42
Chris: Yes. Now, this is one of the reasons that he posted where he posted.

00:22:47
Ned: Right.

00:22:47
Chris: He noticed something was wrong, he had a strong suspicion he knew exactly what it was, but again, he’s not a security researcher. Time to hand it off to the security researchers. And, as is usually the case with things like this, once the right people knew what to look for, the pullback from Tan’s malicious efforts was pretty fast and pretty easy. The releases 5.6.0 and .1 were marked immediately bad, rolled back in all repositories, and the GitHub accounts of all maintainers were frozen pending investigation. Now, this is to the point you made earlier, it’s not just Tan’s account that was frozen, probably for good reason.

00:23:24
Ned: Indeed.

00:23:25
Chris: So remember, the poisoned version of xz never made it into a mainstream deployment of any kind. The only people who would have had it were the ones who seeked it out specifically—bleeding-edge, unstable branches, et cetera—for testing. If you want to know if you have it, just run from the command line ‘xz --version’ and if you get a number that is smaller than 5.6.0, you’re fine.

00:23:52
Ned: Okay.

00:23:53
Chris: So, that’s some crazy stuff, huh?

00:23:56
Ned: Yeah, and I don’t know if we explained just how bad this truly is. So, if I could take a moment because after watching the video, I got an idea of how bad this is. So essentially, what you have is any system running SSH that has this compromised version of xz and is living out on the internet, this hacker could connect to that machine using his private key and inject arbitrary commands into that machine—and run them as root because SSH, the daemon runs as root—so root access to any box running SSH published on the internet, and that’s just a beachhead into every system that system has access to. And if it’s a bastion host, that’s a lot of goddamn systems.

00:24:49
Chris: For most organizations, that would be all of the systems.

00:24:52
Ned: Yes. So, hopefully that communicates the scope and size of the potential fallout here. Basically root on the world.

00:25:01
Chris: Oh, that’s a good way to word it.

00:25:03
Ned: Yeah. Fun times [laugh]. So, I feel like maybe this should give everyone pause for a few moments to reflect, and then think about what we should maybe do to prevent this sort of thing from happening again. Do you have some ideas, Chris?

00:25:19
Chris: I might have a few.

00:25:21
Ned: Okay. Destroy systemd immediately.

00:25:24
Chris: That was almost on the list, believe me.

00:25:26
Ned: [laugh] Of course it was.

00:25:28
Chris: It could be its own episode. In fact, it might.

00:25:30
Ned: It should.

00:25:31
Chris: You know what we should do? I thought about this the other day. You and I have one thing in common.

00:25:36
Ned: Just one?

00:25:37
Chris: Just one. I think we can both go on a rant with, what I would say, above average skill.

00:25:44
Ned: You could have just stopped that we could both go on a rant, but I liked that you added the additional skill there.

00:25:48
Chris: I mean, when you do it, it’s more whiny.

00:25:50
Ned: Nasally, even. Yeah.

00:25:52
Chris: “I have a head cold. Leave me alone.”

00:25:54
Ned: Oh, no, I was just born half French. Boo-yah [laugh].

00:25:58
Chris: [unintelligible 00:25:57]. We could just put out, like, special episodes, or just title them, “The Hold my Beer Episode,” where it just, it’s an unhinged screed.

00:26:08
Ned: Oh, HMBs? Hold My Beer episodes? Yeah, we could do that [laugh].

00:26:13
Chris: Anyway, we will save systemd for that potentiality.

00:26:17
Ned: Fair.

00:26:18
Chris: Takeaways from this that I think we should, as a industry, think about and hopefully resolve. Here’s the first one. One of the reasons that this was possible, xz, for all that it does on literally billions, billions of systems—

00:26:39
Ned: Yes. [laugh] I think that’s accurate.

00:26:41
Chris: If you think of IoT, I mean, come on. A hundred billion? Anywhere that anything Linux is installed, xz is installed. And it is maintained by a single person running this as a fricking hobby.

00:26:52
Ned: Because, of course, it fucking is.

00:26:54
Chris: So, here’s my thought. Why don’t we, recognizing the societal and economic good that packages and programs like this cause, do something crazy, and find a way to subsidize the people that make it possible?

00:27:11
Ned: Yeah, I mean, I don’t know Microsoft is, like, the second most valuable company in the world. You found it. Maybe you should help.

00:27:20
Chris: Here’s the sad part. We’ve had this conversation before.

00:27:23
Ned: Sure have.

00:27:24
Chris: Remember Heartbleed?

00:27:25
Ned: [laugh] Sure do.

00:27:26
Chris: That was back in 2014. When we learned that OpenSSL, a package relied upon—once again—by everyone alive with an electronic device was effectively maintained by one full-time person. Now, in the instance of Heartbleed, it was not malicious, it was one of those crazy bugs. But the consequences were still the same, and they were bad. And everyone was like, “Why did this happen?” And this dude in, like, Switzerland was like, “Bro, I got to cut the grass.”

00:27:58
Ned: [laugh] Seriously.

00:27:59
Chris: The end result back in 2014 was that quote, “IBM, Intel, Microsoft, Facebook, Google, and others pledge millions to open-source.” Now admittedly, the amount that they pledged was pathetic.

00:28:13
Ned: This is going to make me mad, isn’t it?

00:28:14
Chris: $3.9 million between them. Now, just as a fun aside, 2014, Microsoft made $87 billion in revenue. That’s just Microsoft. I will leave the amount of money that brought in by the rest of the mentioned freeloaders as an exercise to the reader. But they found a way to scrounge up almost—almost. Almost—4 million.

00:28:40
Ned: Just—ugh, anyway.

00:28:43
Chris: I’m proud of them. They did it.

00:28:44
Ned: Yeah, good job. Well done, everybody. No notes.

00:28:48
Chris: We got to do better.

00:28:49
Ned: Mm-hm.

00:28:50
Chris: These are just the two high-profile ones. There are literally thousands of open-source projects. And as an episode that we talked about a few months back, even teeny-tiny projects that are 11 lines long, in terms of the way that programs are scaffolded together are really fricking important.

00:29:08
Ned: Yeah.

00:29:08
Chris: And the people that make them, and make things like Windows possible, maybe we should find a way to get them a little remuneration.

00:29:18
Ned: A little bit.

00:29:19
Chris: Just call me crazy. Here’s the other thing. We have to figure out a way to spread the load when it comes to these projects. This is a little bit of a piggyback on the above because money does heal a lot of wounds.

00:29:33
Ned: It does.

00:29:34
Chris: But regardless whether you’re on a team of one or a team of one-thousand, burnout is a thing, and it is a huge problem. And sadly, in a lot of cases, people don’t recognize that it’s happening until it’s been happening for a while. Now, that’s a societal issue, but when you have a situation where you have this kind of critical project that so much of IT resolves and relies upon, and that person burns out, crap.

00:30:00
Ned: Yeah. Potentially devastating. I mean, there’s probably a reason why Tan was able to push out the maintainer. Who knows how much actual, like, pushback the maintainer gave? They’re like, “I’ve maintained this thing for free for, you know, ten years or whatever. I got other shit to do. Like, fine, if you want to take the reins, I guess. You know, you haven’t done anything terrible in the last two years that I know of.”

00:30:26
Chris: Yeah. In this case, the maintainer of xz did admit that the lack of time and possible mental health issues were impeding the ability and motivation to continue to make these regular changes, opening the door to Tan’s manipulation. When you’re not feeling right in any way, shape, or form, giving in to pressure or manipulation is just something that is easier to occur.

00:30:52
Ned: Yep.

00:30:52
Chris: That is science. That is not a joke. That is not me just popping off at the mouth. I would say Google it, but we don’t do that anymore.

00:31:00
Ned: No, we don’t.

00:31:02
Chris: So, number three, stuff does not need to evolve so damn fast.

00:31:08
Ned: Right.

00:31:09
Chris: Let’s remember, xz was a compression function. Compression is a solved problem. We got it. We take big things, we make them smaller. We’ve been able to do that since the ’60s, which is a thousand years ago.

00:31:26
Ned: [laugh] I’m pretty sure, yeah.

00:31:27
Chris: We didn’t even have light bulbs. We had compression. And with the way that bandwidth has exploded, CPU has exploded, storage has exploded, and all of that has gotten ridiculously cheaper, the impetus to make compression even better, day after day, week after week is less. We don’t need to move fast. The answer to the mailing list, when Tan said, “Updates should be coming to this package faster.” Should have been, “Why?”

00:31:58
Ned: Yeah. I mean, we have grown up in an era of constantly accelerating, “Progress”—you know, I’m using air quotes here—progress. And so, there’s this feeling that we need to constantly be updating to the latest version of things and that a newer version should exist.

00:32:18
Chris: Right.

00:32:18
Ned: You make the joke in the notes about Windows 11 and how it didn’t need to exist. And I don’t know if you remember when Windows 10 came out, they said it was the last version of Windows.

00:32:27
Chris: Mm-hm. I remember that.

00:32:29
Ned: And at some point they realized, oh shit, people expect the number go up. If number not go up, people no buy.

00:32:37
Chris: Program bad.

00:32:38
Ned: And so, Windows 11 was created, not because it was that different from Windows 10, but because the number needed to go up.

00:32:47
Chris: Yeah. And you know, just as a fun aside, for people with Windows 10, it’s going end-of-life in 2025, and if you want security updates, you’re going to have to pay for them. So, the number gonna go up. So yeah, let’s just slow down. You move too fast. You got to make the morning last.

00:33:05
Ned: [laugh]. I don’t like this. I don’t like where this is going. I’m getting uncomfortable. I need an adult.

00:33:11
Chris: [laugh] You need a Simon and Garfunkel album. Okay, finally—and this one might be controversial; and you are closer to the programmer type, so you might have thoughts about this as well—

00:33:23
Ned: All right.

00:33:24
Chris: What do we think about baking security scanning directly into the repositories? And I don’t mean that just as a integration where you can use tool X. I mean, if you use GitHub, you’re going to use tool X.

00:33:40
Ned: I see.

00:33:40
Chris: So, in this case, the tool that was manipulated was a security tool called OSS-Fuzz. Let’s tie that directly into GitHub, utilize it to scan the absolute piss out of everything all the time, disable the ability for maintainers to muck with what gets scanned and what doesn’t. If you have stuff on GitHub, it’s getting scanned by everything, end of conversation. You don’t get to pick and choose; you get a report card.

00:34:06
Ned: That is a whole can of worms—

00:34:07
Chris: I agree.

00:34:08
Ned: —and probably a much, much longer conversation than we have time for today.

00:34:12
Chris: But we can agree that I’m right. And that’s all the time we have.

00:34:15
Ned: [laugh] I guess it is. Hey, thanks for listening or something. I guess you found it worthwhile enough if you made it all the way to the end, so congratulations to you, friend, you accomplished something today. Now, you can go sit on the couch, update to the latest version of Ubuntu, and know that you’re safe from the perils of xz. You’ve earned it. You can find more about this show by visiting our LinkedIn page, just search ‘Chaos Lever,’ or go to our website, chaoslever.com where you’ll find show notes, blog posts, and general tomfoolery. We’ll be back next week to see what fresh hell is upon us. Ta-ta for now.

Ned: Could you hear the generator running the whole time?

00:34:59
Chris: I couldn’t.

00:35:00
Ned: Well, that’s good.

00:35:01
Chris: And you know that’s probably good because, you know, nobody else is listening anyway.

00:35:05
Ned: Yeah. My power source failed, so I’ve been plugged into this generator for—I mean… [electronically distorted voice] not me. Still not a robot.

00:35:11
Chris: [whispering] Good cover. [electronic distortions].