Security Engineering Assessment of a Contemporary Banking System

Having completed the first section of the TryHackMe Security Engineering learning path, I developed an interest in applying the security engineering principles learned from the path to a more complex real-world environment: the contemporary core banking system.

The objective of this series is not to claim that every banking institution implements its architecture in exactly the same way. Rather, it is to use a conceptual banking architecture to examine how security engineering principles can be applied across the different layers of a highly sensitive and interconnected system.

Banking systems are particularly suitable for this type of assessment because security failures can have consequences across multiple dimensions, including financial loss, unauthorized transactions, exposure of sensitive customer information, operational disruption, regulatory violations and reputational damage.

From a security engineering perspective, the security of a banking system should therefore not depend on a single security product or control. Instead, security should be designed into each layer of the architecture through defense in depth, least privilege, secure communication, identity and access management, authorization, segmentation, secure coding, cryptography, monitoring, validation and strong transaction controls.

Throughout this series, I will evaluate these controls against the CIA triad — Confidentiality, Integrity and Availability — while also considering authentication, authorization, accountability, resilience and transaction security.

The assessment will move progressively through the architecture, beginning with the components closest to external users and moving toward the deeper internal layers of the banking ecosystem.

Part 1 — The Client-Facing and Web/DMZ Boundary

The first part of this series focuses on the client-facing and externally exposed boundary of a banking system.

This includes two major layers:

1. Client/User Layer
2. Web/DMZ Layer

These layers represent the primary interaction point between users and the banking environment and therefore form one of the most important security boundaries in the architecture.

The client layer includes different categories of banking users and applications, including:

• Internet banking customers
• Mobile banking users
• Corporate banking users
• Bank employees
• Back-office users / Administrative and privileged users

Although back-office users are internal users rather than external users, they may still interact with banking services through web-based internal portals and applications. From a security-engineering perspective, these internal portals should ideally be separated from public-facing internet services and exposed through controlled internal or privileged-access paths.

Therefore, the architectural distinction is not simply between "external users" and "internal users." A more useful security perspective is to distinguish between different trust zones, identities, privileges and network paths.

For example, a retail banking customer should not have the same access path or privileges as a bank employee, while a back-office employee should not automatically have the same privileges as an administrator.

A conceptual architecture for this part of the system can therefore be represented as:

Internet / Client Devices

↓

Edge Firewall / DDoS Protection

↓

Web / DMZ Layer

WAF → Reverse Proxy / Load Balancer → API Gateway → Web Servers

↓

Application / Internal Services

↓

Core Banking / Transaction Services

↓

Data Access Layer

↓

Database Layer

↓

Monitoring, Logging & Security Layer

↓

External Integrations Layer

The remainder of Part 1 examines the security mechanisms that should exist across the client and web/DMZ boundary.

Layer 1 — Client and User Layer

The client layer represents the users, devices and applications through which interaction with the banking system begins.

This includes internet banking users, mobile banking applications, corporate banking clients and internal/back-office users.

From a security-engineering perspective, the most important principle at this layer is that the client should never be inherently trusted.

A user's device may be compromised, a session may be stolen, credentials may be exposed, or an attacker may attempt to impersonate a legitimate user. Consequently, the banking system must continuously enforce authentication, authorization and transaction controls rather than assuming that a request is legitimate simply because it originated from a known client.

Key controls at this layer include:

• TLS
• MFA
• Secure session management
• Device intelligence/fingerprinting
• Risk-based authentication
• Rate limiting
• Secure HTTP headers
• HSTS
• CSP
• Strong authentication and authorization
• Transaction monitoring

An important architectural principle is also that no client or internal user should have a direct path to the core banking database.

Requests should pass through controlled application and service layers where authentication, authorization, validation, transaction controls, logging and monitoring can be enforced.

This is particularly important for back-office users. Even though these users are trusted employees, their access should still be governed by least privilege, role-based access control (RBAC), strong authentication, transaction authorization and auditing.

For example, an employee in compliance should not automatically have the same access as someone in operations, while an operations employee should not automatically have administrative privileges over the banking infrastructure.

Transport Layer Security

All sensitive communication should use modern TLS configurations, preferably TLS 1.3 where supported and TLS 1.2 where required for compatibility.

TLS provides:

• Confidentiality — preventing unauthorized parties from reading data in transit.
• Integrity — helping detect unauthorized modification of transmitted data.
• Server authentication — allowing the client to verify that it is communicating with the intended server.

TLS should not be confused with user authentication. TLS secures the communication channel, while mechanisms such as passwords, MFA, certificates and biometrics establish the identity of the user or client.

Importantly, TLS should not stop at the internet-facing boundary. Sensitive internal service-to-service communications should also be protected, potentially using mTLS(mutual TLS), where both communicating services authenticate each other.

This reflects a zero-trust-oriented principle:

Internal network location should not automatically equal trust.

Multi-Factor Authentication

MFA provides an additional layer of protection when credentials are compromised.

Possible factors include:

• Passwords
• One-time passwords
• Push authentication
• Biometrics
• Hardware security keys
• Digital certificates

MFA should be particularly strong for privileged and back-office users because compromise of these accounts could provide access to sensitive operational systems.

For high-risk activities, the bank can also implement step-up authentication or transaction signing, requiring additional verification before a sensitive transaction is approved.

MFA primarily strengthens authentication and reduces account-takeover risk. It should therefore operate alongside authorization, transaction controls, monitoring and auditing.

Device Fingerprinting and Risk-Based Authentication

Device fingerprinting can provide additional context about the device being used to access a banking account.

It can help identify:

• New devices
• Unusual login patterns
• Suspicious access locations
• Abnormal behaviour
• Potential account takeover
• Suspicious transaction activity

However, device fingerprinting should not be treated as an absolute identity mechanism because devices can be shared, changed, spoofed or compromised.

Instead, device intelligence should feed into a fraud and risk engine, where it can be correlated with authentication events, transaction history and behavioural patterns.

Security events should also be forwarded to the SIEM, allowing security analysts to correlate activity across the banking environment.

Layer 2 — Web/DMZ Layer

The second layer covered in this part of the series is the Web/DMZ layer.

The DMZ provides a controlled security boundary between untrusted external networks and the internal banking environment.

Typical components include:

• Firewalls
• Web Application Firewalls
• Reverse proxies
• Load balancers
• API gateways
• Web servers

A simplified traffic flow can be represented as:

Client → Firewall/Edge Security → WAF → Reverse Proxy/Load Balancer → API Gateway/Web Server → Internal Application Services

The objective is to prevent direct exposure of sensitive internal systems while providing controlled access to legitimate banking services.

Web Application Firewall

The WAF acts as an application-layer security control between external traffic and the banking application.

It can inspect HTTP/HTTPS traffic and detect or block attacks such as:

• SQL injection
• XSS
• Malicious HTTP requests
• Suspicious automated clients
• Abnormal request rates
• Application-layer attacks

The WAF should also generate security logs and alerts that can be integrated with the bank's SIEM and security monitoring infrastructure.

However, the WAF should not be treated as a substitute for secure coding.

A banking application should still implement:

• Parameterized queries
• Strong input validation
• Output encoding
• Secure authentication
• Secure authorization
• Proper session management

The WAF therefore provides defense in depth, rather than being the sole protection against application attacks.

TLS Termination and Inspection

Encrypted HTTPS traffic cannot normally be inspected for application-layer attacks while it remains encrypted.

Consequently, a WAF or reverse proxy may terminate the incoming TLS session, decrypt the traffic, inspect the HTTP request, and then establish a separate secure connection toward the internal application.

Conceptually:

Client → HTTPS → WAF → TLS Termination → Security Inspection → New TLS/mTLS Connection → Internal Service

This allows the WAF to inspect the request for malicious payloads, abnormal behaviour and other application-layer threats.

However, the WAF becomes a highly trusted security component because it temporarily processes decrypted application traffic.

Therefore, the connection from the WAF toward internal systems should also be secured rather than assuming that traffic becomes trusted after passing through the WAF.

Reverse Proxy and Load Balancer

A reverse proxy acts as an intermediary between external clients and backend services.

It can:

• Hide internal server addresses
• Prevent direct exposure of backend infrastructure
• Route requests
• Enforce traffic policies
• Support TLS termination
• Provide an additional security boundary

Load balancers distribute traffic across multiple backend instances and therefore contribute significantly to availability.

This is important to the CIA triad because even if confidentiality and integrity are preserved, a banking system that cannot provide its services during legitimate transactions has suffered a security and operational failure.

IP Allowlisting/ WhiteListing

IP allowlisting can be used as an additional network-level control, especially for sensitive internal or administrative interfaces.

For example, access to an internal back-office portal could be restricted to approved corporate networks or secure access infrastructure.

However, IP addresses should not be treated as user identities.

An IP address may be:

• Shared by many users
• Dynamically assigned
• Hidden behind NAT
• Associated with a proxy
• Compromised

Therefore, IP allowlisting should supplement rather than replace:

Authentication + Authorization + Network Segmentation + Access Control

Secure Cookies and Session Security

For browser-based banking applications, authenticated session cookies should use appropriate security attributes.

These include:

• Secure — restricts transmission of the cookie to HTTPS connections.
• HttpOnly — prevents client-side JavaScript from directly reading the cookie.
• SameSite — controls when cookies are sent in cross-site contexts.

These controls reduce the likelihood of session theft and unauthorized cross-site requests.

Session security should also include appropriate expiration, session invalidation, secure session identifiers and reauthentication for sensitive operations.

CSRF Protection

Cross-Site Request Forgery is particularly relevant to browser applications that use cookies for authentication.

An attacker may attempt to cause an authenticated user's browser to submit an unintended request to the legitimate banking application.

For example, an attacker could attempt to induce a user's browser to perform an unauthorized sensitive action.

CSRF protections can include:

• Anti-CSRF tokens
• SameSite cookies
• Origin/Referer validation where appropriate
• Reauthentication for high-risk operations

The fundamental principle is that a valid authenticated session should not automatically make every request originating from that browser trustworthy.

Security Headers

Security headers provide browsers with instructions on how the banking application should handle content and connections.

Examples include:

Content-Security-Policy (CSP) Restricts where scripts and other resources can originate from and helps reduce the impact of certain XSS attacks.

Strict-Transport-Security (HSTS) Instructs browsers to use HTTPS rather than HTTP for the protected domain.

Content-Security-Policy frame-ancestors / X-Frame-Options Helps prevent clickjacking by controlling whether the banking application can be embedded within another webpage.

X-Content-Type-Options Helps prevent MIME-type sniffing.

These controls demonstrate an important security-engineering principle: the application should not simply respond to requests; it should also communicate security expectations to the client.

API Authentication and Authorization

Modern banking systems are heavily dependent on APIs. Mobile applications, internet banking platforms, internal services, payment systems and external integrations may all communicate through APIs.

API security therefore becomes a critical component of the web/DMZ layer.

OAuth 2.0 can be used within an authorization architecture, while JWT can serve as a token format where appropriate.

However, authentication alone is insufficient.

The API must determine both:

Who are you? → Authentication

and

What are you allowed to do? → Authorization

For example, a corporate user may be authenticated successfully but still not be authorized to initiate a high-value transfer.

API security should therefore validate tokens, scopes, audiences, issuers, expiration and required permissions according to the architecture.

Sensitive internal APIs may additionally use mTLS and service identities.

Security Engineering Perspective

The significance of this first part of the assessment is that the client and Web/DMZ layers form the initial security boundary of the banking ecosystem.

A banking system should not rely on a single control such as a firewall, WAF, MFA or TLS.

Instead, multiple controls should work together:

Client Security

↓

TLS

↓

Firewall / DDoS Protection

↓

WAF

↓

Reverse Proxy / Load Balancer

↓

API Gateway

↓

Authentication

↓

Authorization

↓

Application Security

↓

Transaction Controls

↓

Risk/Fraud Engine

↓

SIEM and Security Monitoring

The purpose of this architecture is to ensure that the failure or compromise of one security mechanism does not automatically result in compromise of the banking system.

This is the fundamental idea behind defense in depth.

A lacklustre implementation at any one of these points can weaken the overall security posture. For example, strong TLS does not compensate for weak authorization; MFA does not compensate for insecure transaction logic; a WAF does not compensate for vulnerable application code; and an IP allowlist does not compensate for weak identity management.

Therefore, in the subsequent parts of this series, the assessment will move deeper into the banking architecture, examining how security engineering principles apply to the application, service/API, transaction-processing, integration, data, risk/fraud, monitoring and AI-driven harmonization layers.

The goal is ultimately to evaluate the banking system as a security ecosystem rather than a collection of isolated security controls.