Is Penetration Testing Still Relevant

This started as coursework for my MSc in Cyber Security at Lancaster University. The research and argument are my own. I used AI to help tighten the writing for a general audience.

TLDR:_ Penetration testing was built for systems that mostly stayed still. Modern infrastructure does not stay still. Cloud environments autoscale by the hour, IoT and cyber physical systems run on wildly different architectures, and CI/CD pipelines ship changes multiple times a day. A test run once a quarter cannot keep pace with any of that, and treating it as a complete security guarantee creates a false sense of safety. The fix is not to drop penetration testing. It is to pair it with continuous monitoring, threat modelling, secure by design practices, and tools like digital twins, so testing becomes one layer in an ongoing process instead of a single snapshot pretending to be the whole picture._

Penetration testing has earned its reputation. Simulating a real attacker against a real system is one of the most direct ways to find out whether your defences actually hold up, and for decades it has been the default answer to "are we secure." But the systems being tested have changed shape entirely. Today's infrastructure is distributed across cloud platforms, packed with IoT devices, and increasingly made up of cyber physical systems that blend software with hardware in the physical world. Conventional penetration testing was designed for a monolithic, mostly static system. It was never built for something that reconfigures itself while you're testing it.

That mismatch raises an uncomfortable question. Does a test that captures one moment in time still provide meaningful assurance for a system that changes constantly. My answer, worked through across three arguments, is that it does not, at least not on its own. First, systems have become too dynamic for a periodic test to keep up. Second, heterogeneous environments like IoT and cyber physical systems have architectural quirks that standard testing methods were never designed to handle. Third, and most importantly, the way forward is not abandoning penetration testing but wrapping it inside a layered strategy built around continuous monitoring, threat modelling, and frameworks from NIST and ISO.

A test is a photograph, your infrastructure is video

Used in isolation, penetration testing can quietly become more dangerous than useful, because it offers the appearance of security without the substance of it. Most organisations run these tests once a year or right before a release, which only captures surface level vulnerabilities at that specific point in time. Researchers Murat, Berkan and Ali make this limitation explicit: a penetration test reports the system's posture as it existed during the test window, nothing more. In a cloud environment where autoscaling can spin up or tear down servers on demand, a vulnerability introduced an hour after the test finishes will sit there undetected until the next scheduled round, whenever that happens to be. That gap is where false assurance lives. Teams believe they are covered because a report says so, while the actual system has already moved on.

The same mismatch shows up even harder in IoT and cyber physical systems. These environments are genuinely heterogeneous, running multiple architectures under real resource constraints, and standard penetration testing tools were not built with that diversity in mind. That is part of why researchers proposed PETIoT, an IoT specific testing methodology built around its own attack chain rather than a generic one borrowed from conventional IT systems. The lesson generalises well beyond IoT: testing large scale, dynamic infrastructure, particularly in safety critical fields like healthcare and industrial control, requires assessment strategies that evolve alongside the systems they are meant to protect, not fixed methods applied on a schedule.

Why scope and speed both work against you

Penetration testing is deliberately time bound and scoped tightly, largely to avoid the very real risks of system downtime, budget overrun, and legal exposure. That constraint is reasonable on its own terms, but it creates gaps, and a system with many entry points only needs one of those gaps to matter. Yaacoub and colleagues make a useful point here: cyber physical systems are complex and varied enough that no single testing methodology can realistically capture every risk across the whole environment. An attack could originate from an edge device, a piece of physical hardware, or cloud infrastructure sitting entirely outside the tested scope. Even an IoT device that only activates during a specific window could fall outside the time period a test actually covers. Running several smaller, scoped tests across different parts of the infrastructure helps, but it introduces its own blind spot, because testing components in isolation does not capture how they interact with each other in production. Real time monitoring and detection close that gap in a way that periodic testing structurally cannot.

Modern DevOps makes this worse simply through speed. Deployment cycles now move faster than any realistic testing schedule can follow, even though penetration testing remains genuinely effective at replicating real attacks when it runs. Mebrek and colleagues built a cloud testing framework specifically to address this using repeated automation, and even they acknowledge it cannot cover every real time threat. When infrastructure changes multiple times a day under continuous integration and deployment, any vulnerability introduced after a test simply sits there, undiscovered, for however long the gap is until the next one, sometimes months. In an environment that changes constantly, testing has to be continuous too. Anything periodic will always be playing catch up.

Building security in before you have to test for it

Threat modelling offers a genuinely useful complement here, because it helps teams identify likely attack paths before testing even starts, which sharpens both the defensive plan and the scope of the test itself. Ascenção and colleagues show how integrating threat modelling early in a workflow helps teams anticipate threats rather than react to whatever a scanner happens to find. Xiong and Lagerström's review reaches a similar conclusion: threat modelling reliably points testers toward the highest risk components, letting penetration testing focus where it actually matters rather than spreading thin across an entire system. Combined, this turns penetration testing into something closer to targeted validation than a blind search, and it makes better use of limited testing resources.

Secure by design takes this a step further by trying to stop vulnerabilities from ever entering the system in the first place, which is consistently the most cost effective way to handle security. Demircioğlu and colleagues describe how applying systems security engineering and threat modelling from the earliest stages of development can prevent the exact class of issues penetration testing usually catches after the fact. Raso and colleagues studied this directly in medical devices and found that building security in from the start measurably reduced the patching effort required after release. In that model, penetration testing shifts role, from the thing that discovers architectural flaws to the thing that confirms they are not there. It is worth being honest about the limits though. Secure by design does not fully account for how threats keep evolving in dynamic systems, so it should be treated as an additional layer in the overall security posture, not a replacement for testing altogether.

Monitoring fills the gap testing cannot

Systems today adapt continuously, whether that is autoscaling under load or configuration changing through routine operations, and security assessment has to adapt at the same pace or it falls behind by definition. Penetration testing is genuinely good at revealing vulnerabilities, but only as a snapshot, because it is time limited by nature rather than ongoing. NIST's SP 800 137 defines Information Security Continuous Monitoring specifically to address this, offering a strategy for maintaining situational awareness and managing risk as it changes rather than at fixed intervals. Continuous monitoring works by examining logs, user behaviour, and configuration changes on an ongoing basis, which shortens the time between a vulnerability appearing and someone noticing it, filling exactly the gap that sits between one penetration test and the next. Penetration testing cannot provide that kind of ongoing operational visibility, particularly in environments where configuration drift happens constantly. Pairing the two together closes that visibility gap far more effectively than either approach manages alone.

This is not just a good idea, it is already how the major frameworks are written. NIST SP 800 53 and ISO/IEC 27001 both treat penetration testing as one component inside a broader risk management and monitoring cycle, not as a standalone proof of security. NIST SP 800 53 builds in periodic assessment as a formal control, while ISO/IEC 27001 requires planned reassessment and places real weight on adapting in real time as part of what a mature security programme looks like. Both frameworks emphasise threat intelligence and ongoing system change, which signals a broader shift away from static, checklist style assessment toward something genuinely adaptive. Given how widely these standards are used across industry, the direction is clear: penetration testing is one part of a larger assessment mechanism, and organisations that align with that view are managing risk more effectively while also demonstrating regulatory maturity.

Bringing it together without pretending it is simple

There is no single method that fully covers a modern, evolving system, which is exactly why a hybrid approach, penetration testing combined with continuous monitoring, automation, and secure by design principles, produces the most credible security posture available. Greco and colleagues demonstrate how AI enhanced tools can scale penetration testing across large, heterogeneous environments where devices run on entirely different protocols, using automation to surface issues that manual testing would miss simply due to time constraints. Digital twin approaches add another piece: testing a critical system without touching the live version at all, by running the test against a virtual model instead. None of these tools replace manual penetration testing. They extend its reach and depth within the time available, adding coverage and speed while manual testing still supplies the realism and judgement that automation alone cannot.

This matters most for systems where testing itself carries real risk. Live testing of critical infrastructure can cause outages, physical damage, or data corruption, which is a genuine constraint on how aggressively you can test in the first place. Atalay and Angin proposed using digital twins, virtual replicas that behave like the physical system, to sidestep exactly that risk. Qureshi and colleagues extended this to complex cyber physical environments, using digital twins to run impact assessments safely. The direction this points toward is predictive rather than reactive security testing, and it shows a real path for how penetration testing can evolve into something that fits inside continuous monitoring rather than sitting outside it. For infrastructure that genuinely cannot afford disruption, this hybrid model offers a way to validate security continuously, without taking the system down to do it.

None of this comes free, and it is worth being upfront about the cost. A hybrid model demands cost effective implementation, ongoing monitoring overhead, and real operational maturity. It needs skilled people to run threat modelling properly, the right tooling for continuous monitoring, and a plan for handling the sheer volume of data all of this generates. Building secure by design into the software delivery pipeline can slow down release speed. Digital twins introduce their own new attack surface, and if configured poorly they can misrepresent the production environment they are meant to mirror, which defeats the purpose entirely. None of this works without genuine collaboration across teams and sustained investment, but the alternative, relying on a single yearly test to represent your security posture, is a worse trade.

The honest conclusion

Penetration testing still matters. What it cannot do anymore is stand alone as proof that a heterogeneous, large scale, dynamic system is secure. A security posture captured as a single snapshot does not hold up in an environment that evolves as constantly as cloud infrastructure does, and existing tools still struggle to properly address the diversity of protocols and architectures found in modern IoT and cyber physical systems.

The response is not to discard penetration testing but to surround it. Continuous monitoring, as defined in NIST SP 800 137 and reinforced in ISO/IEC 27001, closes the detection gap between one test and the next. Threat modelling identifies the highest impact risks in advance, and secure by design reduces vulnerabilities at the architectural level before testing ever needs to catch them. Digital twin technology adds the ability to validate critical infrastructure safely, without the risk that comes with testing live systems. Together, these approaches address the timing and context blind spots that conventional penetration testing leaves open, forming a layered defence rather than a single point in time check.

The bigger shift this points to is in how security testing gets thought about at all. Penetration testing should be treated as one component inside a broader, integrated security framework, not as the final word on whether a system is safe. As systems keep growing more complex, more interconnected, and more operationally critical, the way we test them has to move from static to continuous in the same way the systems themselves have. Organisations that keep treating an annual test as sufficient are out of step with where the standards, and the systems, have already gone. Security was never really a snapshot problem. It has always been a moving target, and it is long past time testing practices caught up with that.

Bibliography

Atalay, A. and Angin, P., 2020. A digital twins approach to smart grid security testing and standardization. In: 2020 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm). IEEE, pp.1–6.

Ascenção, A., Lima, R.M., Cruz, F. and Ribeiro, C., 2025. Enhancing cybersecurity in agile software development through integrated threat modeling. ICS Journal, 7(1), pp.45–61.

Bella, G., Castelluccia, C., et al., 2023. PETIoT: Penetration testing the Internet of Things. In: Proceedings of the 2023 ACM Symposium on Security and Privacy in IoT. ACM.

Demircioğlu, A., Gür, G. and Matheu-García, S.N., 2024. An overview of secure-by-design: Enhancing systems security through systems security engineering and threat modeling. Computers & Security, 136, p.103197.

Greco, C., Fortino, G., Crispo, B. and Choo, K.-K.R., 2023. AI-enabled IoT penetration testing: State-of-the-art and research challenges. Enterprise Information Systems, 17(9), p.2130014.

International Organization for Standardization (ISO), 2022. ISO/IEC 27001:2022 — Information Security, Cybersecurity and Privacy Protection — Information Security Management Systems — Requirements. Geneva: ISO.

Mebrek, R., Lalande, J.F., Géraud, R. and Dinh, T.K., 2023. Towards automated penetration testing for cloud applications. Computers & Security, 131, p.103225.

National Institute of Standards and Technology (NIST), 2020. SP 800–137: Information Security Continuous Monitoring (ISCM) for Federal Information Systems and Organizations. Gaithersburg, MD: NIST.

National Institute of Standards and Technology (NIST), 2020. SP 800–53 Rev. 5: Security and Privacy Controls for Information Systems and Organizations. Gaithersburg, MD: NIST.

Qureshi, A.R., Asensio, A., Imran, M., Garcia, J. and Masip-Bruin, X., 2025. A survey on security enhancing digital twins: Models, applications and tools. Computer Communications, 238, p.108158.

Raso, E., Nanni, F., Lestini, F., Bracciale, L., Panico, G., Bianchi, G., Orengo, G., Marrocco, G. and Loreti, P., 2025. Improving vulnerability management for security-by-design of medical devices. Sensors, 25(14), p.4418.

Xiong, W. and Lagerström, R., 2019. Threat modeling — A systematic literature review. Computers & Security, 84, pp.53–69.

Yaacoub, J.-P.A., Noura, H.N., Salman, O. and Chehab, A., 2023. Ethical hacking for IoT: Security issues, challenges, solutions and recommendations. Internet of Things and Cyber-Physical Systems, 3, pp.280–308.

Yaacoub, J.-P.A., Salman, O., Noura, H.N., Kaaniche, N., Chehab, A. and Malli, M., 2020. Cyber-physical systems security: Limitations, issues and future trends. Microprocessors and Microsystems, 77, p.103201.Conference Name:ACM Woodstock conference