Post cover image

September 7, 2026

Building an AWS Security Posture Monitor From Scratch — Part 5: Pointing It at a Real Account

Where my hand-written fixtures met AWS, and one of them turned out to be a lie.

By Islamannafi

7 min read

Part 4 ended with a scanner that could detect public S3 buckets and open admin ports, score them by capability, and report what it couldn't evaluate. It ran entirely against dictionaries I had typed by hand.

Those dictionaries were guesses. Educated ones — I'd read the boto3 docs — but nothing had ever verified that AWS actually returns what I thought it returned.

This post is about connecting the scanner to a live account, and what that surfaced.

Two identities, on purpose

Before any code, credentials. I'd been using an IAM user with AdministratorAccess, which is wrong for a scanner in an obvious way and a subtle one.

The obvious way: a tool that reads configuration has no business being able to delete buckets. If those credentials leak, an attacker owns the account rather than merely being able to look at it.

The subtle way is more interesting. My Unit 4 acceptance criteria included "revoke a permission the scanner needs and confirm it fails loudly." With admin credentials I couldn't test that at all. Nothing is ever denied, so every access_denied path in my schema, every NotReadableError, the whole PARTIAL status machinery — all of it stayed theoretical.

So the project now uses two separate identities:

IdentityPolicyJobannafi-cliSecurityAudit (read-only)Runs scansterraform-deployPowerUser + IAMFullAccessDeploys the vulnerable lab

The scanner literally cannot modify what it scans. That's the same least-privilege argument the tool exists to enforce, applied to the tool itself.

The first thing reality corrected

get_bucket_policy does not return a policy object. It returns this:

{
  "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\"...}]}"
}

A JSON string, inside a JSON response. My fixtures had stored a parsed dict, and my check did bucket["policy"]["document"]["Statement"] — which against real data would have indexed into a string.

The fix is a json.loads in the collector. But it forces a decision: does the snapshot store what AWS literally returned, or a normalized form?

I went with parsed, because my architecture already says the collector stores what it read, not raw API responses. That has a consequence — parsing can fail independently of the API call, so the collector needs a status for "retrieved it fine, couldn't parse it." That's parse_error, sitting alongside ok and access_denied.

The other correction was smaller but the same shape. get_bucket_ownership_controls on my lab bucket returned BucketOwnerEnforced — the post-2023 default, which disables ACLs entirely. My fixtures assumed BucketOwnerPreferred. Not a bug, but it meant the ACL detection path couldn't be exercised against my own lab without explicitly configuring ownership.

When absence means something specific

get_public_access_block raises NoSuchPublicAccessBlockConfiguration when a bucket has no BPA configuration. That is not a failure — you successfully determined nothing is configured.

But my check reads bucket["bucket_bpa"]["document"]["BlockPublicPolicy"], and None["BlockPublicPolicy"] crashes.

Three options: store None and make every consumer guard for it, store None plus a marker, or normalize the absent configuration to all-false.

I normalized. No BPA configuration genuinely means nothing is blocked, so all-false is a faithful translation rather than invented data. The invariant is now: if document is a dict, all four keys are booleans.

This does flatten two states — "no configuration exists" and "a configuration exists that blocks nothing" produce identical output. I've spent this whole project keeping states like that apart, so it's worth saying why it's acceptable here: those two are operationally identical. Same exposure, same remedy. That's the distinction from bucket-with-no-policy versus bucket-whose-policy-I-couldn't-read, where the remedies differ completely.

The comment explaining that lives in the collector, because someone reading it later will wonder whether I noticed.

Multi-region broke the schema, exactly as predicted

Back in Unit 4 I wrote a design note saying the snapshot represented a single collection scope, and that multi-region would require changing the section wrapper. Then I moved on.

That bill came due. describe_security_groups is a per-region call, so scanning four regions means four calls that can independently succeed or fail. A section with one status can't express "us-east-1 worked, eu-west-1 was denied" — you either mark the whole section denied and throw away three regions of good data, or mark it ok and silently lose one.

The section became nested:

python

"security_groups": {
    "status": "partial",
    "document": [
        {"region": "us-east-1", "status": "ok", "document": [...]},
        {"region": "eu-west-2", "status": "access_denied", "document": None}
    ]
}

Same wrapper shape at both levels. The outer status aggregates: ok if every region succeeded, access_denied if none did, partial if mixed.

That last value forced a change in the check, too. It used to bail out entirely when the section status wasn't ok — which under the new schema would discard three regions of valid results because one failed. Now it iterates regions, evaluates the ones that collected, and records the rest as unevaluated targets.

Which meant generalizing unevaluated entries. They used to assume a resource:

{"resource_id": "arn:aws:s3:::bucket-c", "reason": "access_denied"}

A region isn't a resource. So the entry became typed:

{"target_type": "region", "target": "eu-west-2", "value": None, "reason": "access_denied"}
{"target_type": "bucket", "target": "arn:aws:s3:::bucket-c", "value": "policy", "reason": "access_denied"}

value is the part I nearly dropped. My NotReadableError had been carrying which specific read failed — policy, ACL, ownership controls — and I was throwing it away. "Couldn't read bucket-c's policy" tells an operator to grant s3:GetBucketPolicy. "Couldn't read bucket-c" tells them nothing actionable.

The optimization that didn't work, then did

53 buckets, 4 regions, sequential collection: 11.41 seconds.

Each bucket needs five API calls — location, policy, ACL, ownership controls, BPA — and they were running one after another. Those calls are independent across buckets, so a thread pool should help.

At three buckets, it didn't:

BucketsSequential10 workersSpeedup32.10s2.36snone5311.41s4.02s2.8×

The three-bucket row is the more useful one. Roughly seven calls are sequential no matter how many buckets exist — GetCallerIdentity, ListBuckets, account-level BPA, and one DescribeSecurityGroups per region. At small scale that fixed cost dominates, and thread-pool setup costs more than it saves.

2.8× rather than 10× has the same explanation. Most of the ~272 calls parallelize; the seven that don't set a floor.

I'd rather publish both rows than just the one that makes the optimization look good.

The test that found a real bug

Then the part I'd been unable to do with admin credentials.

I created a third IAM user with an inline policy granting exactly five actions — ListAllMyBuckets, GetBucketLocation, GetBucketAcl, DescribeSecurityGroups, GetCallerIdentity — and deliberately not GetBucketPolicy. Then ran the scan as that identity.

It crashed.

TypeError: 'NoneType' object is not subscriptable
  account_bpa["document"]["BlockPublicAcls"]

The limited policy also lacked GetAccountPublicAccessBlock, so account-level BPA came back {"status": "access_denied", "document": None}. And the S3 check read account_bpa["document"] on the very first line of its evaluation logic, without ever checking the status.

Every other value in that check was guarded. The bucket policy, the ACL, the ownership controls — all of them checked their status before use. Account BPA was read blind.

My fixtures never caught it because I'd written all of them with account_bpa readable. The design was right; the implementation had a hole; a real permission boundary found it in one run.

The fix raised a design question worth more than the bug. Account BPA is one value for the whole account, so if it's unreadable, every bucket becomes unevaluable — fifty identical entries saying the same thing. Better to fail once at the top of the check than per resource. The guard lives next to the section-status check now.

With that fixed:

ScanStatus.INCOMPLETE
3.1.4 CheckStatus.CANT_EVALUATE 0
6.3   CheckStatus.VIOLATIONS 1

Zero S3 findings — and the status says so. The security group check was unaffected, because DescribeSecurityGroups was in the limited policy. Visibility degrades per check rather than all at once.

A scanner without this design returns an empty findings list there. The operator reads "no issues," files it, and a publicly readable bucket goes unnoticed. That's the failure I built the whole thing to avoid, and this is the first time I've watched it not happen against a real IAM boundary rather than a dictionary I wrote.

A third check, to see if the interface held

Two checks both reading S3-ish data isn't much of a test of the abstraction. So: CIS 2.14, IAM policies granting : administrative privileges.

It stresses the interface in three ways the others didn't. IAM is global, so the finding's region is None — the first time the str | None decision from Unit 3 actually mattered. The resource is a policy, not a bucket or a group. And it needs two API calls per policy, because list_policies returns metadata but the document comes from get_policy_version.

Two scoping decisions:

Customer-managed only (Scope="Local"). AdministratorAccess is an AWS-managed policy granting :, attached in most accounts, and flagging it every scan is noise. CIS 2.14 is about permissions an organization granted through its own policies. The cost is real — AdministratorAccess attached to a user is a risk and I now miss it — so it's in the known-gaps list rather than glossed over.

Collect unattached policies too (OnlyAttached=False). The check only reports attached ones, but the collector's job is completeness and the check's job is relevance. Filtering at the API would mean the snapshot could never support a future check for dangerous-but-dormant policies.

That paid off immediately. My account has two : policies: the one my lab deploys, attached to a role, and a leftover from an unrelated CTF exercise with attachment_count: 0. The check flags the first and correctly skips the second — and the second is still in the snapshot if I ever want it.

The detection rule has one subtlety. "s3:*" is not :. It grants every S3 action, not every AWS action, and a check that conflates them produces constant false positives. Exact membership rather than substring matching:

if "*" in actions and "*" in resources:

Both fields need string-or-list normalization, because IAM policy JSON allows either.

The check slotted in without touching BaseCheck, the registry, or the runner. That was the question worth answering.

Where it stands

ScanStatus.COMPLETED
3.1.4 CheckStatus.VIOLATIONS 1
6.3   CheckStatus.VIOLATIONS 1
2.14  CheckStatus.VIOLATIONS 1
us-east-1 ok 2
us-east-2 ok 1
us-west-1 ok 1
us-west-2 ok 1

Three checks, three AWS services, four regions, live account. 17 tests, none of which need credentials.

Still missing

  • IPv6 isn't checked. A rule opening ::/0 on port 22 is equally public and currently missed.
  • NotAction isn't handled. A statement with NotAction: ["iam:*"] and Resource: "*" grants everything except IAM — effectively admin, and 2.14 misses it.
  • Regions are hardcoded to four US regions.
  • No backoff. At this scale the scanner doesn't hit AWS rate limits, so throttling handling is untested rather than implemented.
  • No CLI. Scans still run from Python.

Next post: the test suite and CI — including the paths that still have no coverage, like the ACL unreadable case and the parse-error branch.

Code: github.com/DevAnnafi/AWS-Security-Posture-Monitor