The chronic pain that is security

And much like recycling, it's a complex and continuous process; the real issue is not so much the initial investment, it's the constant follow-up and re-evaluation. Coding defensively with security in mind requires a degree of effort and a specific kind of criminal thinking (because you have to think like one to catch one). One of the common mistakes novices and laymen make is confusing the appearance of working (user clicks here - gets the expected result) with a solid, defensive, secure design. That's a real problem, one that can be mitigated by audits and penetration tests conducted by professionals who specialise in that domain. The bigger problem, however, is that new threats are introduced, discovered and addressed daily, even in the very best components, written by excellent, top-tier, security-oriented programmers. And your code depends on many of these components, which in turn means that over time your code becomes vulnerable without your applying direct changes on your end.

To better illustrate, look at the OpenSSL's vulnerabilities list; this is an excellent, high-quality library, used by countless respectable projects to perform very sensitive operations. Now, let's also consider the Linux kernel (take a look at https://linuxcvetracker.com). Two amazing projects, with extremely competent programmers and high security awareness. Now think of all your other dependencies (OS, DBs, programming languages, libraries, NPM packages, etc, etc) and ask yourself: how many known (let alone unknown) vulnerabilities do the versions I use have that I'm unaware of?

At this point, you may be concerned, and you're probably thinking: "okay, well, I suppose we should just update all our dependencies to the latest stable versions".

The short answer is: yes, you should. But before you can do that, you need to:

Ascertain what these dependencies actually are (this may sound trivial, but I assure you, most developers will struggle to name all deps their codebase requires, even, and sometimes, especially, if they've been working on it for years). There are tools, including FOSS ones, that can parse over your codebase and help you generate an inventory of dependencies, that's the good news; however, depending on your codebase (the languages you use, your approach to packaging and to utilising third-party components), this may not be so straightforward.

Evaluate the risk and, in turn, the cost of the update.

Here's the problem: when you do not continuously monitor for CVEs and update incrementally, you accumulate debt that's harder to settle. It goes like this:

• You need certain functionality; after some research and evaluation, you decide to go with libNameHere
• You start with the stable version of libNameHere available at that point, let's say it's v1.73
• Your code works and all is well, you write more code that depends on libNameHere of that version, as well as a lot of completely unrelated code
• libNameHere, meanwhile, evolves; new features are added, bugs are fixed, security exploits are reported and are also fixed, but, and here's the key: some of these are breaking changes, which means using newer versions of libNameHere with your current codebase, will not work correctly. The exact way in which it will not work will vary: if you're "lucky", it just won't compile/run at all (depending on whether you're using a precompiled language or an interpreter one), if you're very unlucky, your code will run and appear fine, until it crashes brutally, unexpectedly and with severe implications

As time goes by, you incur more and more debt. Finally, an exploit so severe in libNameHere is made public that you absolutely must address it.

But, actually, there's a pre-step: you first need to ascertain whether your version is even vulnerable. Why not just update? Again, because of breaking changes that were introduced between your version and the latest one, changes that usually have nothing to do with the exploit you must defend against.

This pre-step may seem trivial; you might expect the CVE advisory to note the vulnerable versions. That would be a great public service to all but unfortunately — that's not usually the case. To better understand the problem, let's describe the typical process of discovering and reporting a CVE:

• The reporter could be a security researcher, paid specifically to assess vulnerabilities in a given codebase, but, more commonly, it's a consumer of that component, usually a programmer, but not always
• A security researcher is more likely to take the extra time to trace the exact commit that introduced the issue, and possibly assess additional versions of the component for the same vulnerability. That helps, because with some work, you can then, with a deep enough understanding, infer whether a given version (the one you have) is also vulnerable. However, that's not always the case; in my experience, most CVE advisories do not specify the full range of vulnerable versions at all. They will note the version in which the exploit was found, and sometimes state that all versions below N are vulnerable (which often proves to be false)
• The version problem aside, there's also the question of the underlying ENV (CPU arch, OS, what compile-time options were used, what configuration options are enabled, etc, etc). A good advisory does note these details, but, and here's the key point: not all advisories are good! The quality depends largely on the reporter's abilities and goodwill, as well as those of the component's maintainers and their desire to perform a deep investigation and share the findings
• The best advisories also include a PoC you can run to determine whether you're vulnerable, or else, a clear statement that this exploit is theoretical and was never proven to work, but once more: quality varies and sometimes, details are intentionally omitted to avoid exploitation in the wild; moreover, some CVEs, albeit very real (as opposed to theoretical), are notoriously difficult to reproduce, as they rely on specific conditions, both in terms of HW and load

So, with that understanding in mind, let's move on to the next problem:

Even if the advisory notes your version as vulnerable, the component maintainers may not release a patch for said version. Reputable projects work fast when a valid CVE is reported, they will likely release a patch within days, at most. However, they'll release it for actively supported versions, not for every version ever released since the inception of the project.

If you're lucky, few changes occurred in and around the vulnerable code and you can backport the patch without significant effort or expertise — that is often not the case, however.

On top of this, there's a certain challenge that stems from how the entire ecosystem functions: you are likely not building and packaging all software yourself; if your machines are running a Linux distribution (though this applies to other OSes as well), a lot (often most, but typically not all — and this is key as well) of the packages your code relies on were deployed from the official distro repositories (either in binary form or, when using distros like Gentoo, by building locally using Portage). If you want a deeper dive into that, I partially cover it in my article here.

Different distributions have different release schedules, update policies and, key to our subject matter: different support and EOL plans.

Let's say that, back in 2014, you decided to go with RHEL. The latest release at the time was RHEL 7 (released on June 10, 2014) so you installed that on your servers. Let's further assume that you've been smart and mindful enough to, at least occasionally, update your machines with the latest packages pushed to the official distro repos. The last release in the 7 series was 7.9 (on September 29, 2020) and it reached the end of its maintenance support 2 phase on June 30, 2024. That means that, starting from that day, no new packages are published to the RHEL 7 repos, even if a most severe CVE affects them. Of course, you can still purchase Extended Life Cycle Support (ELS) for Red Hat Enterprise Linux 7 provides up to 4 additional years of maintenance support for certain critical-impact security fixes, selected urgent priority bug fixes, and troubleshooting for the last minor release of Red Hat Enterprise Linux 7 (7.9) until May 31, 2029.

But, read the bold text above carefully: certain critical-impact security fixes, not all, not by a long shot:)

I use RHEL as an example, but the same thing will be true to any Linux distro (or whatever OS you choose — be it FOSS, proprietary, community-based or commercial). No vendor supports versions from the dawn of time until its final retirement — it is simply way too consuming to be feasible.

And so, to recap, unless you actively review your software inventory at all levels/layers and update regularly (and, sadly, because it's a lot of work — too few individuals and enterprises actually do), you will eventually find yourself in the scenario we described a few paragraphs ago and be faced with two options:

• Update to the latest version[s] (or at least, a higher version that addresses the vulnerabilities you simply cannot ignore) and work, often very hard, to revise your code accordingly and then, equally hard on regression testing to confirm your product continues to function as expected and finally — pray that you've not missed anything
• Backport the patches required to address the vulnerabilities to your, often far older, version. This can be even harder and, unless you are very well versed in that component's code, is often error-prone and can lead to severe and unexpected failures, to put it very mildly

If you got this far and it made you reconsider your strategy, I consider it job done. If you need help devising a strategy to reduce your current and future risks, headaches and panic attacks, get in touch.

Happy patching:)