Unraveling GitHub's Tech Mysteries [CL90]

[00:00:00] Ned: My kids are very entrepreneurial, and they have each started their own store in their rooms. James has the Artemis Armory, where he will make you weapons. Those weapons are made out of popsicle sticks and duct tape—yes, I’m serious—but he will make them.

[00:00:20] Chris: It’s a start.

[00:00:21] Ned: And Tess has Stuff-N-Things. Guess what they sell?

[00:00:26] Chris: Uh… cheese steaks.

[00:00:29] Ned: [laugh] That would be a thing. That’s stuff in it, so sure, why not [laugh] . And I think Genevieve has something to do with… stuffed animal sitting or something. I think that’s what she does.

[00:00:42] Chris: Like babysitting.

[00:00:43] Ned: Yeah, but you know, she’ll watch your stuffed animals for you. And not give them back because [laugh] that would be ridiculous.

[00:00:50] Chris: I don’t think that’s completely how it works.

[00:00:53] Ned: It’s more like stuffy kidnapping.

[00:00:56]

[00:00:56] Chris: Right.

[00:00:58] Ned: Yeah. Except she’s not holding it for ransom. It’s just hers now.

[00:01:12] Hello, alleged human, and welcome to the Chaos Lever podcast. My name is Ned, and I’m definitely not a robot. I am a real human person. There’s nothing special about me. I have the usual 16 senses, and can only hold my breath for like, I don’t know, six hours before I get a little lightheaded. Because, of course, I breathe. Without breath, you can’t laugh. Ha ha ha. And after our arsine gas and silicon tetrachloride, isn’t laughter the best human medicine? With me is Chris, who is blissfully far away. Hi, Chris.

[00:01:47] Chris: Safety first and all that.

[00:01:49] Ned: Well, you know, the problem is not so much holding my breath. It’s when I let it back out. That’s when bad things seem to happen.

[00:01:56] Chris: I have been telling you for years now that toothpaste is a thing.

[00:02:01] Ned: [laugh] I don’t think it’s going to solve the arsenic problem, but you know, we could give it a try. I don’t know what would happen with all that baking soda and arsenic.

[00:02:09] Chris: It’s a good question. There’s only one way to find out.

[00:02:13]

[00:02:13] Ned: Tune into Mr. Wizard next week, where we find out what happens when Timmy tries arsenic.

[00:02:18] Chris: You thought it was exciting when he ate an apple upside down.

[00:02:21]

[00:02:21] Ned: [laugh] How come it doesn’t fall out of my mouth? Mr. Wizard? Oh. That got dark.

[00:02:31] Chris: Just a little bit.

[00:02:32]

[00:02:32] Ned: Yeah, a bit. Do you think people know Mr. Wizard, like—

[00:02:36] Chris: no.

[00:02:37] Ned: —watch him firsthand anymore?

[00:02:38] Chris: Not even a little bit.

[00:02:39] Ned: Okay.

[00:02:40] Chris: No.

[00:02:40] Ned: Because really, that’s our generation, and—maybe mine. Did you watch Mr. Wizard as a child?

[00:02:47] Chris: Of course.

[00:02:48] Ned: Okay. I don’t know. We’re like three years off from each other. So.

[00:02:53]

[00:02:53] Chris: Yeah, it only feels like you’re 30 years older

[00:02:55] than me.

[00:02:56] Ned: And I look 30 years younger. So strange.

[00:02:59] Chris: Well…

[00:03:01] Ned: That’s more on the account of you looking like you’re 70 and have since you were 20.

[00:03:05] Chris: I have no response to that.

[00:03:07] Ned: No [laugh] . At least you dress the part, I guess. And that’s what’s important.

[00:03:13] Chris: That’s true.

[00:03:13] Ned: Let’s talk about some tech garbage [laugh] .

[00:03:15] Chris: Okay.

[00:03:16] Ned: Okay.

[00:03:18] Chris: Let’s talk about people hacking GitHub Actions and Workflows for fun and profit.

[00:03:23] Ned: Woo, yay.

[00:03:25] Chris: So, full warning before we start. This entire topic comes directly from the ‘I understand what some of those words meant’ department. I am only writing it because Ned insisted that I do, probably specifically to hurt me. Little did he know how much it would hurt him to have to hear me read it. So, ha ha.

[00:03:48] Ned: Oh.

[00:03:49] Chris: Take that.

[00:03:49] Ned: Oh.

[00:03:51] Chris: So TL;DR, I am definitely not a programmer, this is definitely not my lane, but I did stay at a Holiday Inn Express last night.

[00:04:00] Ned: Ooh, did you get the cinnamon rolls?

[00:04:03] Chris: So, hacking GitHub Actions and Workflows for fun and profit. Let’s start with some definitions.

[00:04:09] Ned: Oh, that’s my favorite place to start.

[00:04:12] Chris: What is Git?

[00:04:14] Ned: In programming land, or just like in general?

[00:04:17]

[00:04:17] Chris: How long do you want this podcast to be?

[00:04:20] Ned: [laugh] I was very confused when I first started hearing the term Git being thrown around because I thought of it as, sort of, a derogatory term. You know, some useless git,

[00:04:31] you know?

[00:04:31] Chris: Yeah, which is like British slang from the 1910s.

[00:04:35] Ned: Yeah, when you were born.

[00:04:38] Chris: [laugh] Oh, if only I were that young.

[00:04:40] [laugh] .

[00:04:42] In computer terms, Git is a version control system intended for software development. It was invented by Linux legend Linus Torvalds because, shocker of all shockers, there was a version control system that predated it that he was using for Linux development. That program was not open-source. This is similar to how Linux happened.

[00:05:09] Ned: Indeed, yeah.

[00:05:10] Chris: Just to keep you up to date. So, what does version control software do? It allows a lot of control and auditability over code stored in a file/directory structure, whether that file directory structure is real or not doesn’t matter. Any number of developers can be granted access to add files, to remove files, to make changes to existing files, promote or demote versions of these files into various levels of working environments. The whole goal here being asynchronous development of Project X on its way to production.

[00:05:50] Ned: Right. And Git was by no means—and still isn’t—the only game in town when it comes to VCS. It’s certainly the dominant one, especially since that’s what GitHub chose to use for their source control, but there were several other ones before it, like Subversion, Team—what is it called? Trying to remember the name of the Microsoft one. It was something awful. SourceSafe is what it was originally called.

[00:06:16] Chris: Oh, really?

[00:06:17] Ned: Yeah. SourceSafe. And then that turned into Team Foundation Version Control or something like that, which then still exists as a thing on Azure DevOps but, like, no one picks it. It’s like the saddest radio button ever.

[00:06:34] Chris: So, previous to any of these systems, any version control, what you would have while you were developing software is just basically a buttload of crazily named text files that people edited willy-nilly, hoped that they all worked, and that people would remember who edited what, when, and why. With a version control system like Git, you have an organized buttload of text files with historical information that you can look back on, who did what, you can make comments about what changes were made and why, and you can make commits or revert commits—as in changes—as needed. So, like you said, Git was not the first version control system; it is not the only version control system. What made it popular was that it was free.

[00:07:24] Ned: Free goes a long way.

[00:07:26] Chris: Now, side point. Why in the hell is it called Git?

[00:07:31] Ned: That’s a good question. I’ve never thought to ask.

[00:07:34] Chris: Nobody does. Everybody assumes that it means something. Long story short, it doesn’t. Linus Torvalds wanted a short, pronounceable word for his new program. That’s it. In the first code comment for the Git project, he wrote, quote, “Git, the stupid content tracker. Git can mean anything depending on your mood. It is a random three-letter combination that is pronounceable and not actually used by any common Unix command. The fact that it is a mispronunciation of ‘get’ may or may not be relevant.” Unquote.

[00:08:17] Ned: Okay.

[00:08:18] Chris: So, there you have it.

[00:08:20] Ned: I mean, very on brand for him.

[00:08:24] Chris: Yes. I think that’s something that, you know, in terms of programmers, Linus is weird. And if you know programmers, you know, the intent behind that statement.

[00:08:37] Ned: Yeah.

[00:08:39] Chris: Anyway, so that’s Git.

[00:08:40] Ned: Okay.

[00:08:41] Chris: What is GitHub? Now, this is a much simpler conversation. GitHub is just where the files go.

[00:08:48] Ned: Yeah.

[00:08:49] Chris: Git is the version control system. That is the brains, that is the logic, that is the magic. GitHub is where all your files go after you run the Git commands, if you’re running in a GitHub environment. Now, you can have any other target you would like. You can have Git store things on your hard drive, if you want to. You can use other commercial products like GitLab.

[00:09:10] All the second part of that is, is the repository itself. Where are these files stored? They’re stored at GitHub. Okay, GitHub is now sort of still its own giant project, but it’s two separate pieces. The first part—keeping track of the version control system stuff—is Git. Where the files go and all this other additional add-on stuff, which we’re going to get to in just a second, is the company GitHub and the repository where your files are stored.

[00:09:38] Ned: Right. And just to put a finer point on it, when you have a repository that you’re working with actively on your local workstation, the files are located on your local storage. Part of the dance you do with GitHub is you set it as the origin for a remote. You add a remote to the local repository, and you call it origin, and you point it at the GitHub repository. And now Git kind of understands that these two things are copies of the same repository, so when you push a commit up to GitHub, it writes the updated files to that GitHub repository, and you can pull branches and commits down from GitHub. So, there’s a… there’s a bit more to it than that, and—there’s actually more to it than that, than that, but that’s as deep as I think we need to go for this conversation.

[00:10:32]

[00:10:32] Chris: There’s a bit more to it than that, than that.

[00:10:35] Ned: Yep.

[00:10:35] Chris: I mean, I couldn’t have put it better myself.

[00:10:38] Ned: [laugh] .

[00:10:39] Chris: Okay, so Git, pretty straightforward in its purpose, when it was created. GitHub: where your files go. Over time, people use a product, and they think to themselves, “Self,”—or selves—“we should probably figure out better ways to do parts of what we’re doing.” So, extensions to the original goals of the project always end up happening. And in the case of GitHub, we have things like Actions and Workflows.

[00:11:10] Now, we’re going to talk about this stuff only in the context of GitHub. If you use a different system, you probably have something equivalent. It is probably called something else. It is likely that the issues still exist, but they have different names, they have different solutions, et cetera. All right? So, we’re going from the realm of the general, which is Git, into the realm of the specific, which is the environment made, curated, and managed by GitHub.

[00:11:42] So, GitHub has extended the functionality of its product over time, and two of the most interesting things that they have, one is called Actions, one is called Workflows. And they work very closely together with each other, to the point that it is a pain in the ass to pick them apart. I am going to try to explain them, I am not going to do a great job.

[00:12:04] Ned: Excellent.

[00:12:06] Chris: So, a GitHub Action is an automation of a repeatable task, simple job can be custom by you, there can be ones that exist that you have by default, or you can get ones that were written by somebody else, pulled from the helpful GitHub Marketplace. GitHub Actions are written in YAML, and thus I hate them—

[00:12:32] Ned: indeed.

[00:12:33] Chris: And they are stored in a special directory in your repository called ‘.github/workflows.’ Already you can see why I’m annoyed and confused. Actions and Workflows are both stored in a directory called Workflows.

[00:12:47] Ned: Indeed, yeah.

[00:12:49] Chris: Awesome. Now, Actions are intended to do repeatable tasks in multiple environments, as all good modular code does. So, the idea here is you’re writing code that helps you write code. Actions and Workflows are like meta code that help you do the programming that becomes whatever it is that you’re putting out into the world.

[00:13:13] Ned: Right, and more importantly, and this is probably going to be germane to later in the conversation, the meta code, as you’re calling it, is stored in the same repository as the rest of the code, and it is checked in with the rest of that code when you want to push a commit up to GitHub.

[00:13:34] Chris: Right. But it’s in its own directory, so it’s fine.

[00:13:38] Ned: [laugh] It’s not fine.

[00:13:41]

[00:13:41] Chris: Okay, so those are Actions. Actions are kind of more atomic, self-contained, very much like the idea of the original goal behind Linux commands: do one thing, do it extremely well. One individual Action. A Workflow is also a YAML file, and thus I hate it, but it is set to run one or more jobs automatically. The jobs that they run through the Workflow can be: built-in Actions, in terms of things that just exist in GitHub; they can be GitHub Actions that we just discussed. And again, those can be custom or premade.

[00:14:21] What causes a Workflow to run? Well, they are all tuned to what are called GitHub Events. A GitHub Event is a trackable Action, all of the things that can possibly happen inside of GitHub. And there’s a lot of them. I looked at the list, I got scared, I closed the list.

[00:14:44] Ned: Yeah [laugh] .

[00:14:45] Chris: But let’s keep it simple. So, things that you do in GitHub, you publish new code, you push code up. That’s an Action. You pull code, you download it, you pull code down. That’s an Action. You can also do things in even other meta areas like, open an issue, which is aka complain.

[00:15:04] Ned: Yeah.

[00:15:05] Chris: That’s an Event. All of these things are baked into GitHub. You don’t have to do anything on your own; they just are there, and they can kick off a workflow.

[00:15:14] Ned: You can even schedule an Event that just runs at the same time every day. It uses the crontab format, so however you want to schedule it, you can use that crontab format to schedule it to run at this regular interval.

[00:15:29] Chris: Right. And in that case, if you want to keep it in the same context of Events, the Event that you’re talking about is ‘what time is it?’ So, the way Workflows work is simple. When an Event is identified as part of a Workflow—aka it’s kicked off—a Docker container—aka a Runner—spins up in the background, executes the Workflow, hits all the automations, all the Actions, et cetera, and then turns off. That’s it. Pretty straightforward.

[00:16:00] Now, these things are not free if you use them in the cloud. You get charged like functions, which effectively is what they are. So, you get charged by the minute for your Workflows when they run up in the GitHub cloud. But they’re usually super-duper fast and unless your environment is enormous, you will probably never even see a charge for this because you get 2000 minutes per month for free.

[00:16:27] Ned: I’ve never paid for it.

[00:16:28] Chris: Nobody ever has, except when you do. So, the jobs that Workflows and Actions can do for you are literally limited only by your imagination. Say, for example, you wanted files automatically uploaded to another repository, whether inside your own project or elsewhere. You can do that. You need to give Workflow access to do it, but you can do it.

[00:16:54] Since the Workflows are stored in the repository alongside the code they support, they are usually readable by everybody, which means that storing passwords in your Workflow files is a massively bad idea.

[00:17:09] Ned: Yeah, yeah.

[00:17:10] Chris: Just like it is in any codebase. Like I said, this is meta programming, but it’s still programming. A lot of what we’re going to talk about, these are all bad, whether you do it in Actions in GitHub, or you do it in C++ on your own. In order to allow communication and automation and modular— [ma-hu-ha] —modularization. That is not even a word, and I can’t say it right.

[00:17:33] Ned: I love it though.

[00:17:35] Chris: GitHub has a secret feature. It will inject sensitive data, such as these tokens or passwords, at the right time, based on environmental variables, again, just like regular programming. Now, in order to do this, or the reason that this is enabled, is GitHub has a whole bunch of, basically, variables that you can utilize—tokens, passwords, what have you—and again, this helps with modularization, and with security. So, now that we have all that ‘figured out’—and I’m putting ‘figured out’ in giant air quotes—why is this a problem? How is it that these Workflows can be used by the bad guys for bad things?

[00:18:20] Well, let’s talk about it. First off, it stands to reason that if an attacker were to gain any measure of control over a codebase, bad things could happen. New vulnerabilities baked into the product. That would be bad. If you have Workflows that allow code to be merged directly into production, congratulations, you have an attack vector.

[00:18:45] Remember those Runners I was talking about that automatically run code for you in the background? Well, what happens if they are spun up with more permissions than they should have? You guessed it: attack vector.

[00:18:58] Ned: Yep.

[00:18:59] Chris: And usually, once again, just like the rest of programming, these all happened because of everybody’s best friend, unintended consequences. Now, as you can also imagine, this is a bigger problem the bigger your projects get. Let’s say you have one simple Workflow that, I don’t know, automatically publishes website copy to an Azure static website the minute a new article gets pushed into a repo. I think that’s probably safe.

[00:19:28]

[00:19:28] Ned: Yeah, mostly.

[00:19:29] Chris: I mean, not definitely, but probably. But a bigger project with many, many Workflows, Runners, Actions, developers, very different story. Let’s consider a specific example that made the news this week.

[00:19:45] Ned: Yay.

[00:19:47] Chris: There is a project that is run by the good folks at Facebook called PyTorch, that… well, it’s not really important. It does a lot with neural networks, it’s got a cool name—

[00:19:59] Ned: it does.

[00:20:00] Chris: —it doesn’t matter what it does. Let’s not get distracted.

[00:20:02] Ned: Fair enough.

[00:20:02] Chris: The point is, it’s a big, popular project with many thousands of followers and sub-repositories—or product-lets? I don’t even know what to call them—all underneath the same umbrella.

[00:20:15] Ned: Right.

[00:20:16] Chris: The project has been running for ages, with some, like I said, some huge number of very, very good programmers, which means that they are very, very good at understanding Workflows and Actions, and how to make them operate to their best ability in order to make programming faster. So, that’s good because they’re getting the most out of the product. That’s bad because it also means they know how to be naughty and do things with the product that they shouldn’t, or shouldn’t be allowed to do. So, for example, we talked before that jobs are run on Runners. This particular project, PyTorch, had been using self-hosted Runners, which means that the Runner does not get spun up by GitHub, executed, and then terminated; it runs elsewhere. GitHub explicitly tells you not to do this.

[00:21:14] Ned: Yeah, well, what they try to tell you is, there are potential security issues with doing this, so unless you have a really good reason, just use the free—not free—but the publicly available GitHub Runners because literally what they do is they have a whole swath of virtual machines sitting there doing nothing, and then when someone wants to run a GitHub Action, it gets attached to one of those Runner VMs, whatever containers are involved with the Action gets spun up on that VM, it does its work, and then that VM is thrown away. It is never used again. It is deleted, and a new one is spun up, and it’s placed to take whatever the next Action is. That’s good. You want that to happen. If you’re using self-hosted Runners, then there’s a really good chance that you’re not doing that last part where you’re just throwing it away after every run, and bad things can happen.

[00:22:10] Chris: And that is exactly what happened. Or at least part of it. So, what you’re talking about is Runners that are ephemeral. They exist for a moment of time, and then they are destroyed completely, and they are started from scratch, which means that even in the unlikely event that a Runner that is existing for only seconds somehow gets compromised, it gets deleted immediately.

[00:22:34] Ned: Right.

[00:22:35] Chris: So, we always talk in security about blast radius; this is an amazing way to minimize blast radius. Well, that virtual machine was compromised, but it only existed for 16 seconds. Compromised, not great, but 16 seconds of danger, better than an infinite amount of danger, which is what the PyTorch people found themselves in: an infinite amount of danger.

[00:22:59] Ned: [laugh] Yay.

[00:23:00] Chris: So, security researchers have been aware of the issues with GitHub Actions and Workflows for a while, and in this specific case, they were able to string together a few different tricks, which is often how these kinds of hacks happen. To quote the article, the researchers, quote, “Submitted a trivial fix for a typo in a markdown file, and then made a pull request that triggered an action on self-hosted Runners including malicious scripts, granting root access.”

[00:23:31] Ned: eek.

[00:23:34] Chris: Got all that?

[00:23:36] Ned: I do—we can unpack it a bit, but I feel like you’re going to do that as we go on.

[00:23:40] Chris: No, I was just going to end the show right there.

[00:23:42] Ned: Okay, excellent [laugh] .

[00:23:43] [laugh]

[00:23:45] Chris: This was escalations of permissions. It allowed them to run a malicious script on a self-hosted Runner that was not ephemeral. That script sat and waited for other Workflows to run, and it grabbed the all-important—and this is all in capital letters—GITHUB_TOKEN for the project that gave them root access to the repository.

[00:24:10] Ned: Yeah, that’s not good. You don’t want to do that.

[00:24:12] Chris: Remember, unintended consequences. So, a lot of things had to be wrong for this to work. First of all, this was self-hosted, non-ephemeral Runner. Regular Runners would be isolated in addition to being deleted when they were done. The attackers could conceivably have gotten their malicious script to run, but no other Actions would ever be present, so it wouldn’t be able to pull the GitHub token.

[00:24:37] Second, that pull requests accessed a pretty big secret, which was the GitHub token itself, which was not secure. Finally, Actions triggered by pull requests needed to be treated with more care. Pull requests are just a download. They are ordinarily not restricted, meaning that it is almost always going to run, meaning that if you have a job on a pull request that has more permissions than it should, it will run. And if it ends up running a malicious script, it will run with enhanced permissions.

[00:25:12] Ned: Right. And this is specific to public repositories—

[00:25:16] Chris: oh, yeah. Yeah, yeah.

[00:25:17] Ned: —because the thing about public repositories is, ideally, anybody can contribute to them. They’re open-source, and you know, everyone is available to inspect the code, create issues, and if they want to contribute back00, like fixing a typo, that’s a legitimate thing that someone might want to do—all they need to do is make a fork of the code, fix that issue, and then submit a pull request back to the main branch of the public repository, and that kicks off a Workflow. And usually, that’s fine.

[00:25:49] Chris: Usually, that’s the end of the story.

[00:25:51] Ned: Right. The important part is, these attackers weren’t just fixing a typo, they were also going into that GitHub Actions or GitHub Workflow directory, and adding their own Actions to it. And the thing about the pull request is, it will execute the Actions as defined, in the code that you’re trying to merge in, not the code as it is today. So, that Runner will run that updated Action and any scripts that they tacked on to it, which again, like you said, typically not a problem because it’s happening on an ephemeral Runner that’s just going to go away. So, that script wouldn’t be able to do anything.

[00:26:28] But if you’re using a self-hosted Runner, that doesn’t go away, and that script persists running after that initial Action, and then a new Workflow kicks off, it’s going to load a bunch of environment variables as part of its Action. And now you, as the script, can harvest those environment variables and exfiltrate them to whatever attacker is trying to gain access to the repository. And now you’re owned.

[00:26:55] Chris: Right. So, this specific incident was a high profile project, which got a lot of news, relatively speaking, but there are many ways to attack Workflows and Actions that have been identified over the years. This is not a new idea. There is a link in the [show notes] to GitHub user Nikita Stupin’s pwnhub repo that identifies in a lot more detail how these things can go, and has a fun little script that you can run to test your own repositories, if you so choose.

[00:27:31] Ned: [laugh] Fair.

[00:27:32] Chris: Don’t use it to hack other people. That’s how you end up in prison.

[00:27:35] Ned: Yes.

[00:27:37] Chris: Now, like I said, this has been around for a while, and GitHub themselves published an article about Action and Workflow security that is worth checking out, and also includes a command injection attack example. So, if you are more comfortable and more well-versed with Actions and Workflows, this goes into a lot more detail than I did. It includes four tips—best practices—for Actions or Workflows that are worth going over briefly, though.

[00:28:04] So, tip number one: don’t use insecure substitution syntax in the run section of Actions or Workflows. Now, this is a good one, which again, hearkens back to just programming best practices. Basically, if you have a syntax that allows expansion in an executable realm, different code can be substituted. Has been for years. This is Bobby Tables type of stuff. This is a problem with web programming and not sanitizing your inputs, abusing forms online to get reverse shells, fill in the blanks. There are ways to do it that are safer, that take a little bit more work. Do that.

[00:28:47] Ned: Do the extra work.

[00:28:48] Chris: Yeah. Number two, enable code-scanning for Workflows. Now, this is probably something you do, static code scanning for what you write—your regular code—why not do it for your meta code? Integrating code-scanning and automated testing for your Workflows and Actions helps identify vulnerabilities before they ever have a chance to impact production. GitHub themselves offers tools like CodeQL for scanning code for vulnerabilities. Automating tests as part of your Workflows ensures that every push or pull is checked for potential security issues. So this is, check the code when it’s static, check it while it’s running, make sure you know what it’s doing and why. Observation is security.

[00:29:32] Number three, use the principle of least privilege. My God. I think a lot of people, because this is not your actual code, just get a little bit… lazy.

[00:29:46] Ned: [laugh] It’s true.

[00:29:46] Chris: In the interest of speed and getting programmatic work done, corners are cut—

[00:29:52] Ned: they can be.

[00:29:53] Chris: And one of the easiest ways to cut corners to make things work is to just say ‘Allow All.’ Now, this is the same problem with programming. This is very similar to how S3 buckets used to be default read-all access, which was fixed years and years and years and years ago at this point, but in S3 world, still a problem.

[00:30:16] Ned: Right.

[00:30:17] Chris: When you have any deployment at all, when you have permissions granted, make sure that the absolute minimum is where you start. And also where you end. Now, that token that I was talking about before, the GITHUB_TOKEN, back in the day, it was, by default, much more permissive than it is now. If, however, you have a long-existing project, even though the defaults changed, your token’s default did not. So, you have to go in there and double-check it yourself.

[00:30:51] Ned: This has come back to bite me on more than one occasion where I’m setting up a new repository, and I forgot that they changed the token default permissions, and I now need to be explicit about it, and like, my Workflow won’t run. And I’ll be like, “Why?” And then I’ll realize, oh because you just didn’t give it enough permissions. And then I have to do a very quick, simple search to go, all right, what are the permissions required for this Action? And I add them. And then I’m done. So, it’s not terrible. It’s, you know, going to take an extra five minutes, but save you lots of heartache later.

[00:31:27] Chris: Exactly. And this is one of those things where the larger your project, the more important it is. Because if you have one developer, it’s an annoyance. If you have 100 developers, it is potentially catastrophic because you cannot keep this all on track in your head.

[00:31:45] Number four, enable private vulnerability reporting. Now, according to GitHub, this PVR is not just something you should talk to your doctor about. It is a built-in feature of GitHub that allows private communications about potential issues in your repository’s code. It is a good way to keep up to date with what’s going on security-wise, and how it might affect you directly. I had never heard of this.

[00:32:12] Ned: Yeah, me neither.

[00:32:13] Chris: But it sounds good.

[00:32:15] Ned: Eh.

[00:32:16] Chris: So, like I said, a lot of this feels like basic security best practices just applied to an area of your environment that maybe doesn’t often get as much attention as it should. This meta code is not your actual code; it doesn’t get nearly the play in the wider world, but it’s in control of your repositories, and it is incumbent upon you to be really careful and responsible with what permissions you allow, how do you monitor them, and what kinds of safeguards you have in place that things like the PyTorch issue would not become a problem for your environment. As usual, security has to be a constant question because it doesn’t matter how fast you wrote your code if it’s compromised. Because once that becomes public, no one is going to trust it to run.

[00:33:13] Ned: Yeah. So, the short, short version is, if you are using self-hosted Runners on GitHub today, you might want to go check on that.

[00:33:21] Chris: Yeah, don’t do that. That’s number five. Don’t do that.

[00:33:24] Ned: [laugh] Well, the most common reason for doing it is because your Runner needs to run from a place that is private, whether it’s a private virtual network in one of the public clouds, or it’s your on-premises data center, it’s somewhere that the typical public Runner cannot reach, and so you’re going to put a self-hosted Runner there that has access to those things, which makes it all the more sensitive. So yeah, you might want to take a little extra time and, hey, you know, like virtual machines are a thing, man. Like, you can just kill one and spin up another. It’s not that hard.

[00:34:02] Chris: It won’t upset the virtual machine.

[00:34:04] No.

[00:34:04] They like it.

[00:34:05] Ned: [laugh] I guess. Oh, they welcome oblivion. As do we all. Well, hey, thanks for listening or something. I guess you found it worthwhile enough if you made it all the way to the end, so congratulations to you, friend, you accomplished something today. Now, you can go sit on the couch, eat a chili dog and check on your self-hosted Runner. I’m sure it’s doing great.

[00:34:26] You can find more about this show by visiting our LinkedIn page, just search Chaos Lever, or go to our website, chaoslever.com where you’ll find show notes, blog posts, and general tomfoolery. We’ll be back next week to see what fresh hell is upon us. Ta-ta

[00:34:40] for now.

[00:34:41] Chris: Unless one of your Workflows gets compromised and the whole website comes down.

[00:34:45] Ned: It does run on GitHub Actions. [laugh] Like, no lie.