The Problem with Research-Grade Threat Intelligence

Most threat intelligence platforms live in one of two states: a polished commercial product behind a paywall, or a research prototype that works on the developer's machine and nowhere else. Open Intelligence Lab started as the latter. v0.1.0 through v0.4.0 built a serious intelligence engine — graph-based threat modeling, STIX 2.1 export, TAXII ingestion, MISP live feeds, and provenance validation. But none of it had a CI/CD pipeline. Every deployment was manual. Every push to main was a leap of faith.

v0.5.0 Changes That

v0.5.0 adds a full 5-stage GitLab CI/CD pipeline. Not a demo pipeline. Not a placeholder. A pipeline that actually runs lint, executes tests, builds a Docker image, scans for CVEs, and sits one manual button-press away from deploying to production.

The Intelligence Case

The original goal of Open Intelligence Lab was dynamic attribution. Static datasets do not track emerging C2 infrastructure or zero-day campaigns. v0.3.0 added STIX 2.1 and TAXII transport — the standard protocol for threat intelligence exchange that mandates TLS, encrypting all data in transit end-to-end. v0.4.0 added MISP live feed ingestion and provenance validation. Most government agencies, CERT teams, and ISACs use MISP for community-oriented threat sharing. Embedding it makes automatic updates and community features feasible without adding unnecessary complexity.

Provenance validation is the trusted handshake. Where does the data come from? Each graph needs a chain of custody that organizations can trust. Fields like timestamp, reported_by, primary_source, and trust_level together give us the chain of custody needed for a gold standard threat intelligence project.

Every component exists because the ecosystem demands it: MISP needs a database and cache to function, TAXII mandates HTTPS, and a browser will not call an HTTP endpoint from an HTTPS page — so Caddy is not optional. The API runs on HTTP internally. Caddy handles HTTPS on port 8443 and forwards to port 8000, so the API never manages certificates itself.

Pipeline Architecture

Stage 1 — Validate

Two jobs run in parallel. lint checks code quality across api/ and backend/ using flake8, black, and isort. schema_validate hits every JSON dataset and STIX export file to confirm structural integrity before anything else runs. If your data model is broken, you know before you build.

Stage 2 — Test

unit_tests runs pytest with coverage reporting, producing JUnit XML artifacts that GitLab renders as a test report. api_smoke_test spins up the actual FastAPI server inside the CI container, waits five seconds, then fires HTTP requests at /health, /intelligence/graph/summary, and /intelligence/entities. If the API does not respond 200, the pipeline fails. This is not mocking. This is the real server, running against real datasets, proving the system works end to end on every push.

Stage 3 — Build

build_docker builds a Docker image tagged with the exact commit SHA and pushes it to the GitLab Container Registry. Every commit produces a traceable, immutable artifact. tag_release stamps a build.env dotenv artifact with the version string and build timestamp.

Stage 4 — Security

dependency_scan runs pip-audit and safety against requirements.txt, catching known CVEs in the dependency tree. container_scan runs Trivy against the Docker image itself, scanning for HIGH and CRITICAL vulnerabilities. Both are allow_failure: true — they warn without blocking, because a security advisory should trigger review, not an emergency rollback.

Stage 5 — Deploy

deploy_production is a manual job. It does not auto-trigger. You look at what passed, you decide it is ready, you click the play button. Continuous delivery is not continuous deployment. The rollback_production job takes a ROLLBACK_SHA variable and deploys any previous image from the registry in seconds.

Links

Source Code: github.com/AlborzNazari/open-intelligence-lab GitLab CI/CD: gitlab.com/alborznazari4/open-intelligence-lab Visual Lab: alborznazari.github.io/open-intelligence-lab CHANGELOG: github.com/AlborzNazari/open-intelligence-lab/blob/main/CHANGELOG.md

#ThreatIntelligence #OpenSource #GitLabCICD #DevSecOps #STIX #MISP #Docker #Python #SecurityEngineering #OpenIntelligenceLab