A practical pattern for keeping third-party code out of production until it's earned its way in.

Hi everyone.

Today I want to share my thoughts on how to set up the clean development perimeter principle inside a company's environment. Let's start by defining what it actually means in more formal terms.

The clean development perimeter is an approach where all external code, dependencies, images, and artifacts go through quarantine, automated checks, and moderation before use, and are only allowed into development after receiving a "trusted" status.

And right away it's worth clarifying what I mean by the internal company perimeter. It's the CI runners where prod and dev builds happen, and the internal registries. That's exactly where a dependency lands after passing the process, and that's exactly what we're protecting. One more important caveat — we're talking specifically about third-party dependencies. Internal packages that the company writes itself don't go through this process; they get all these checks during their own development, as part of the regular DevSecOps process, and that's a separate story.

So every team, every developer, can be sure that all the dependencies they want to use and are allowed to use have already passed every check the security team deemed necessary. And if a needed dependency isn't there, the developer has the option to kick off and oversee the process of bringing it into the perimeter themselves. The process itself has to be convenient — in the best case it's just a list of dependencies from a package manager, like pip freeze.

Why is this beneficial for the company itself?

First, it significantly reduces security risks. When your company has one well-established process for letting dependencies into the perimeter, you have full control over what runs in your prod clouds. And if a dangerous zero-day shows up, you can quickly, with one switch, add the package to a blacklist — and that works both as a first line of defense, preventing accidental additions, and as a way to monitor whether the package is already inside. The follow-up questions — which exact products use it, what to do about it — that's a job for subsequent security processes, handled separately from the act of blocking itself. Not all developers can keep close track of the news, especially in adjacent disciplines. So it's very easy to imagine a situation where a developer from a neighboring team, currently building a feature for an LLM product, accidentally installs and tries to push a vulnerable version of litellm into the company's perimeter — like what happened just recently.

But the clean perimeter principle can also help with compliance control. Licensing questions are critically important, and missing something there can come back as serious money — and business, as we know, has a habit of holding on to its money. So with a properly organized approach, this kind of pipeline at the dependency intake stage also enables license auditing: company lawyers can find out from developers right away how the code will be used inside the company, evaluate how that fits with the license, and kill potential problems early.

We can also make everyone's life much easier by collecting SBOMs for dependencies right here, in one place. After all, this process is essentially the only entry point for third-party dependencies into the company's perimeter. And if we collect all the necessary indirect artifacts tied to a package's existence in the perimeter — SBOMs, for example — then when the need arises we won't have to generate an SBOM for that dependency every single time. These dependencies can be used across different products, and the SBOM will already be sitting ready in the artifactory.

And of course, in the current global situation, it can also be useful to check whether a dependency contains any controversial statements — political ones, for example. In 2022 we witnessed maintainers adding to their repos various statements about their views and sympathies or anti-sympathies regarding the military conflict. We also know about cases of deliberate mischief and hacktivism. Maintaining some list of phrases, slogans, or words to scan artifacts against is reasonable behavior, because the accidental presence of such material in a product can lead to reputational costs or problems in certain jurisdictions.

Here it's important to separate threats correctly. All sorts of wipers, backdoors, and other malicious code inside a dependency — those are classic security threats, and they get caught by SAST and other scanners, just like any other software backdoor or vulnerability. But the banners, slogans, and other text material I mentioned above — that's more of a legal and reputational threat, not a security one, and it gets checked separately, against its own list of phrases. These two things live in different scanning streams, even if they fire in the same pipeline.

So how do we actually implement all of this?

Loading into the local artifactory, security checks, license checks, scans for political banners — how do we tie it all together?

First and foremost, you should start from developer convenience. This is my honest belief, formed over many years of work. As a security engineer, I'm naturally more focused on application security questions, and so are my colleagues — but because of that we often forget that our job isn't to be overseers watching over devs, but to be the people who take all the headache onto themselves and help developers deal with security.

If the process is inconvenient, it will be sabotaged. So the main thing is to make sure the process is logical, consistent, and natural from the perspective of a default programmer in the company.

Take a separate repository in the corporate GitLab, for example. The developer just runs pip freeze in their project, pastes those dependencies into a file inside that repo on a separate branch, and opens a Merge Request. That's where their work ends. All further communication happens in the MR comments.

The rest of the actions and checks run in the pipeline: pulling the artifacts and scanning them, deciding whether the package needs to stay in quarantine for a few more days (say, a week since release hasn't passed yet), pulling its license, scanning the sources and built artifacts for political content.

If one of the checks finds something, the people responsible for that area are immediately assigned as reviewers. Lawyers stamp an approve or reject based on policy; appsec looks at whether action is needed on scanner findings, and so on. Once all the approvals are in, the package gets uploaded to the corporate artifactory, and any dev can start using it.

What does a system like this give us?

Full control. We see who added these packages and why, we see that the compliance and security work was done, we know specific people and their messages, and we can provide all the evidence to counterparties or regulators if questions come up about the use of one dependency or another. We don't have to dig up information about who uploaded what to the artifactory, who reviewed it, or whether anyone reviewed it at all — we don't have to fish through DMs and email. Everything is centralized and polished in one place. A process where everyone has their role, all assignments are automatic, and each person acts within their competencies, without creating bureaucratic chaos.

We'll have a well-oiled system that also covers checks for new versions of already-loaded libraries, as well as attacks like Dependency Confusion and typosquatting. If someone needs a new version, it goes through the same process. It also lets us track license changes, update SBOMs, and run all the other checks. And for popular dependencies in the company, thanks to automation we can automatically pull and trigger the process — and if no problems were found and the licenses haven't changed, automatically push the new version through.

It also simplifies, in my opinion, emergency fixes and Emergency override scenarios. Counterintuitive as it sounds, having such a transparent process actually makes it possible to flag the case clearly, immediately identify whose responsibility it is to upload the lib, implicitly get approval from all the relevant managers and admins, and not forget about the checks in the rush — running them afterward instead. That kind of transparency is your guarantee that the necessary processes will get executed after the fix.

What about the difficulties?

But of course, a scheme like this comes with its own difficulties. You have to account for the specifics of different ecosystems. If your company uses many technologies, you'll have to set up a separate moderating repository for each one, and if a project uses several technologies, you'll have to file MRs into each of them. The difficulties, of course, aren't with the concept but with the execution: each such repository will require its own set of scripts implementing the logic. And this especially applies to generic dependencies, like binaries or executables — finding license information and the rest of it will be a separate headache.

And even though the whole process lives in an ecosystem developers are already used to — GitLab — it still puts additional burdens on them, even if it's just creating two or three MRs. The key is to remember that exceptions are possible in this scheme — for automation scripts and the like, for example. The main consumer of this system is the development organization, and you have to take their opinion into account when looking for the compromise between their convenience and the regulatory and security benefits the system provides to the company.

And of course, I expect a thought has occurred to some readers by now: okay, we've added the lib we needed, but package managers like pip give a flat list, transitive dependencies included — what do we do with those?

It's a fair question. My take — this isn't a bug, it's a feature. If those transitive dependencies can show up in the build, then they should also be reviewed by this system, at least at the license level. Our task is to build a system that lets us assemble our products inside a closed perimeter, as the upper bound — and for that you need to bring in transitive dependencies too. However, this implies the need to build a full list of transitive dependencies for those package managers that don't expose it themselves — .NET, for example. This is probably one of the most substantial problems in implementing the concept.

Since this article is mostly about the approach and the concept, I've intentionally avoided naming specific tools or showing snippets. If you'd be interested in seeing a more technically dense article — drop a like and write in the comments.