I Built an Offline Tool That Tears Apart Your Security Policies

They look fine. They have headings, sections, references to NIST or ISO 27001. But run them through a real gap analysis and the cracks show up fast. I know because I built one and ran it against my own sample ISMS policy. It came back with 41 missing clauses and a maturity score of 19.44%. That's the "Initial" tier. The floor.

That result is what pushed me to actually finish POLARIS.

The Real Problem With Policy Reviews

Someone gets assigned to compare your policy against a framework. They open both documents, go control by control, and make judgment calls. The whole thing takes days, the results depend entirely on who's doing it, and the moment the audit closes, nobody touches those policies for another 18 months.

I wanted something that gives the same answer every time, finishes in seconds, and doesn't just say something is missing but tells you exactly what.

How It Works

The core idea is semantic gap detection, not keyword matching.

Keyword matching breaks immediately in practice. Your policy says "revoking access when someone leaves." The framework says "access deprovisioning." A keyword search flags this as missing. Semantically, it is not.

POLARIS uses all-MiniLM-L6-v2 to embed both the framework clauses and your policy sentences into vectors. It then measures cosine similarity between each required clause and every sentence in your policy. Anything below 0.45 similarity gets flagged as a gap. The threshold is adjustable because strict regulatory language sometimes needs tighter matching.

Each control gets a score from 0 to 3 based on clause coverage. Those scores roll up into a maturity percentage across five tiers: Initial, Developing, Defined, Managed, Optimized.

The output is specific:

Control   | Function | Score | Missing Clauses
GV.OC-01  | GOVERN   |   1   | organizational context, mission objectives
RS.MA-01  | RESPOND  |   0   | escalation matrix, forensics procedure

Control   | Function | Score | Missing Clauses
GV.OC-01  | GOVERN   |   1   | organizational context, mission objectives
RS.MA-01  | RESPOND  |   0   | escalation matrix, forensics procedure

Not "your RESPOND function needs work." Exactly what clause is missing, in which control, with a remediation roadmap sorted by urgency.

Why I Made It Fully Offline

Policy documents describe real gaps in real security programs. Sending them to an external API is a privacy decision most teams are not set up to make quickly.

POLARIS makes zero external API calls during analysis. After the first run downloads the embedding model (around 90MB), everything runs locally forever. There is an optional Ollama integration that uses a local LLM to rewrite missing clauses into policy language, but even that stays on your machine. If Ollama is not installed, POLARIS falls back to static improvement text and keeps going.

This also means it works in air-gapped environments, which is exactly where sensitive compliance work usually happens.

The Bug That Almost Broke the Core Promise

Early on, same policy and same framework would sometimes produce slightly different scores depending on how embedding batches were ordered. Not a crash, not obvious at all, just quiet non-determinism in a tool whose entire value is consistency.

The fix was standardizing batch processing and writing a HashingEmbeddingModel for tests. A lightweight deterministic fallback that runs without downloading anything, so CI stays fast and offline. What I built to debug the problem became the thing that made the test suite actually trustworthy.

If your tool's main promise is reproducibility, you have to test for it directly. Assuming it works is not enough.

What You Get Out

Three formats depending on who needs the output:

Terminal for analysts doing live reviews. Gap table, coverage matrix, maturity score, roadmap, all in one screen.

PDF for leadership. Cover page, executive summary, full gap analysis, remediation roadmap. Something you can hand to an auditor.

JSON for pipelines. Structured output that plugs into SIEMs, dashboards, or automation.

python main.py --policy policies/isms.docx --framework iso27001 --format pdf

python main.py --policy policies/isms.docx --framework iso27001 --format pdf

What's Live and What's Coming

Three frameworks are supported now: NIST CSF 2.0, ISO 27001:2022, SOC 2. GDPR, HIPAA, and PCI DSS are next. You can also bring your own framework by writing a simple JSON file and passing it via --framework.

The things I want to build next: a web UI so non-CLI teams can actually use this, API mode for local automation pipelines, and evidence mapping so each covered clause links back to the exact sentence in your policy that covers it. That last one turns gap analysis into a full audit trail.

Try It

MIT licensed, fully open source: github.com/SudoXploit7/POLARIS

Run it against your own policies. The number that comes back is usually more interesting than people expect.