Post cover image
Image

October 24, 2026

SCA for Data Pipelines: Your First Line of Defence Against Cyber Attacks

By Dinakar Sundar

By Dinakar

8 min read

Introduction

In April 2021, a supply-chain attack on ua-parser-js exposed millions of servers to cryptominers. In 2022, PyPI was flooded with 4,000+ malicious packages mimicking popular data science libraries. In 2023, the XZ Utils backdoor nearly compromised SSH servers globally.

Now ask yourself: when did you last audit the open-source libraries powering your Spark jobs, Glue ETL scripts, Kafka consumers, or Airflow DAGs?

Data pipelines are among the most dependency-heavy, least-audited workloads in modern engineering. They quietly ingest terabytes of sensitive data, transform it, and feed downstream analytics, ML models, and executive dashboards — yet they are rarely treated like production software from a security standpoint. That is a dangerous blind spot, and Software Composition Analysis (SCA) is how you close it.

What Is SCA (Software Composition Analysis)?

Software Composition Analysis (SCA) is the practice of identifying, cataloguing, and evaluating all open-source and third-party components in your software — and continuously checking them against known vulnerability databases, licence policies, and malicious package registries.

SCA answers one question: "What exactly is running inside my code, and is any of it dangerous?"

SCA covers three core concerns:

1. Vulnerability Detection — Each library you import carries its own CVE (Common Vulnerabilities and Exposures) history. SCA tools cross-reference every dependency in your requirements.txt, pom.xml, or package.json against the National Vulnerability Database (NVD), GitHub Advisory Database, and OSV.

2. Licence Compliance — A data platform pulling in GPL-licensed code into a commercial product can expose the organisation to legal liability. SCA enforces licence policies at build time.

3. Software Bill of Materials (SBOM) — A machine-readable inventory of every component in your software. Regulators (US Executive Order 14028, EU Cyber Resilience Act) are increasingly mandating SBOMs for software in critical infrastructure.

Why Data Pipelines Are High-Value Targets

A data pipeline is not just ETL code. It is the nervous system of your organisation's data flow. Consider what a compromised pipeline can do:

  • Exfiltrate data at ingestion — malicious code in a PySpark job can forward raw CDC records to an external endpoint before any masking is applied.
  • Poison the data lake — subtle row-level manipulation of Bronze or Silver tables corrupts analytics and ML training sets without obvious alarms.
  • Enable lateral movement — pipeline code runs with IAM roles that often have broad S3, Glue, and Redshift access. A compromised Glue job is a vector into your entire data lake.
  • Ransomware via Iceberg — an attacker with write access can drop Iceberg snapshots or corrupt table metadata, ransoming your data without touching S3 objects.
None

Real-World Attack Vectors on Data Pipeline Dependencies

Typosquatting — Attackers publish panads, bototo3, or spark-ccore on PyPI — names differing by one character from pandas, boto3, pyspark. One copy-paste error pulls malicious code into your next Glue job deployment.

Dependency Confusion — If your org uses a private PyPI mirror, an attacker publishes a higher-versioned package with the same name on public PyPI. pip's default resolver picks the higher version — injecting attacker code into your internal build.

Compromised Maintainer Accounts — A legitimate, widely-used library gets a malicious release pushed by a maintainer whose credentials were stolen. The package passes all name-based checks because it is the real package — just with a backdoor.

Malicious Transitive Dependencies — You audit your direct dependencies. But library-A depends on library-B which depends on library-C which was quietly compromised six months ago. library-C never appears in your requirements file.

CVE Exploitation in Stale Environments — A critical CVE (e.g., Log4Shell in Apache Log4j 2.x) is published. Your Spark EMR cluster, built on an AMI from 18 months ago, still runs the vulnerable version. Attackers scan and exploit within hours of disclosure.

Open-Source SCA Tools: Your First Line of Defence

1. OWASP Dependency-Check

The most widely adopted open-source SCA tool, maintained by OWASP. Scans Java, Python, .NET, JavaScript, and more against the NVD. Best for batch vulnerability scanning in CI/CD pipelines — integrates natively with Maven, Gradle, Jenkins, and GitHub Actions.

# GitHub Actions for a PySpark project
- name: OWASP Dependency-Check
  uses: dependency-check/Dependency-Check_Action@main
  with:
    project: 'data-platform-pipelines'
    path: '.'
    format: 'HTML'
    args: --enableExperimental

2. Trivy (Aqua Security)

A fast, comprehensive scanner covering application dependencies, OS packages, container images, IaC files, and SBOMs. The de facto standard for container-based data workloads.

# Scan a custom Glue container image
trivy image my-ecr-repo/glue-custom:latest --severity HIGH,CRITICAL
# Scan a Python project directory
trivy fs . --scanners vuln,secret,misconfig
# Generate an SBOM
trivy image my-ecr-repo/glue-custom:latest --format cyclonedx --output sbom.json

Standout feature: Trivy detects secrets (AWS access keys, DB passwords) accidentally committed in source or baked into container images — a catastrophically common mistake in data engineering repos.

3. pip-audit

A Python-native SCA tool by the Python Packaging Authority (PyPA). Audits installed packages or a requirements file against the Python Security Advisory database and OSV. Best for Airflow DAG environments and Lambda function dependencies.

pip-audit -r requirements.txt
pip-audit -r requirements.txt -f json -o audit-results.json

4. Syft + Grype (Anchore)

Syft generates SBOMs from container images, filesystems, and packages. Grype consumes those SBOMs and matches them against NVD, GitHub Advisory, OSV, RedHat, and Amazon Linux databases. Best for auditable SBOM compliance.

# Generate SBOM
syft my-ecr-repo/spark-emr:latest -o cyclonedx-json > sbom.json
# Scan SBOM for vulnerabilities
grype sbom:sbom.json --fail-on high

5. Semgrep

Lightweight static analysis with a rules marketplace. Detects known-vulnerable API patterns, unsafe deserialization, and hardcoded secrets in Python, Java, and Scala pipeline code. Best for shift-left security — catching issues before they reach a dependency scanner.

- uses: returntocorp/semgrep-action@v1
  with:
    config: >
      p/python
      p/secrets
      p/apache-spark

Wiz: Cloud-Native Security for Data Platform Infrastructure

Open-source SCA tools cover code and containers. But a modern data platform — built on AWS Glue, S3 Tables, Lake Formation, EMR, MSK, and Redshift — has a cloud infrastructure attack surface that static SCA cannot see. This is where Wiz fills a critical gap.

What Wiz Does: Wiz is an agentless cloud security platform that connects to your AWS account via a read-only IAM role and builds a Security Graph — a contextual model of every cloud resource, its configuration, its IAM permissions, its network exposure, and its vulnerability state.

Key Capabilities for Data Pipelines:

  1. Vulnerability Context in Running Workloads — Wiz scans the actual OS and packages running inside your Glue containers, EMR nodes, and Lambda functions at runtime. A CVE in a library baked into a 6-month-old EMR bootstrap script? Wiz finds it.
  2. Toxic Combinations — Wiz's killer feature. A CVE in a Glue job is concerning. But Wiz tells you: "This CVE is in a Glue job whose execution role has s3:* on your data lake bucket, running in a public subnet, with a security group allowing inbound from 0.0.0.0/0." That combination is a critical incident, not a medium severity finding.
  3. Data Security Posture Management (DSPM) — Discovers all S3 buckets, Redshift clusters, and Glue databases, scans for PII, and flags misconfigurations: buckets without encryption, Lake Formation tables accessible without column-level grants, Redshift with public endpoints.
  4. Identity & Entitlement Analysis — Maps every IAM role and policy across your data platform. Surfaces over-privileged Glue execution roles (a common finding: Glue jobs with AdministratorAccess), cross-account trust relationships, and unused credentials.
  5. Shift-Left with Wiz Code — Integrates into GitHub PRs to scan IaC (Terraform, CloudFormation) before deployment — catching an S3 bucket defined with acl = "public-read" or an overly permissive IAM role before it hits production.
None

Building a Defence-in-Depth Strategy

SCA is not a single tool deployed once. It is a continuous practice woven into every layer of your data engineering workflow.

Layer 1 — Dependency Governance

  • Pin all dependencies to exact versions in requirements.txt, pom.xml, and Docker base images. Never use >= or latest in production.
  • Maintain a private package mirror (AWS CodeArtifact, JFrog Artifactory) and configure pip to use it exclusively. This eliminates dependency confusion attacks.
  • Allowlist permitted packages and block everything else at the mirror layer.

Layer 2 — Shift-Left in CI/CD

  • Run pip-audit and trivy fs on every pull request. Fail the build on HIGH and CRITICAL CVEs.
  • Run Semgrep for secrets and vulnerable patterns.
  • Use trivy image to scan container images before pushing to ECR.
  • Generate SBOMs with Syft on every release tag.

Layer 3 — Runtime Scanning

  • Deploy Wiz or an open-source alternative (Falco for runtime threat detection) to continuously monitor Glue jobs, EMR clusters, and Lambda functions.
  • Set up nightly Trivy scans against your ECR image catalogue.

Layer 4 — IAM Least-Privilege

  • Every Glue job, Lambda function, and EMR cluster should have a task-specific IAM role — not a shared data-platform role.
  • Use Lake Formation fine-grained permissions as the enforcement layer, not S3 bucket policies alone.
  • Audit IAM roles quarterly using Wiz or AWS IAM Access Analyzer.

Layer 5 — Incident Response Readiness

  • Maintain an up-to-date SBOM for every pipeline environment so that when a new CVE drops (Log4Shell-style), you can answer "are we affected?" in minutes, not weeks.
  • Test your response playbook: how quickly can you patch and redeploy a Glue job when a critical CVE is disclosed?

Practical Implementation: SCA in Your CI/CD Pipeline

A production-ready GitHub Actions workflow embedding SCA at every stage for an AWS data platform:

name: Data Pipeline Security Scan
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'  # Nightly scan
jobs:
  sca-python:
    name: Python Dependency Audit
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install pip-audit
      - run: pip-audit -r requirements.txt -f json -o pip-audit-results.json
      - uses: actions/upload-artifact@v4
        with:
          name: pip-audit-results
          path: pip-audit-results.json
  sca-container:
    name: Container Image Scan (Trivy)
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t glue-pipeline:${{ github.sha }} .
      - uses: aquasecurity/trivy-action@master
        with:
          image-ref: glue-pipeline:${{ github.sha }}
          format: sarif
          output: trivy-results.sarif
          severity: HIGH,CRITICAL
          exit-code: '1'
      - uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: trivy-results.sarif
  sbom-generate:
    name: Generate SBOM
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - uses: anchore/sbom-action@v0
        with:
          image: glue-pipeline:${{ github.sha }}
          format: cyclonedx-json
          output-file: sbom-${{ github.sha }}.json
      - run: |
          aws s3 cp sbom-${{ github.sha }}.json \
            s3://your-security-bucket/sboms/$(date +%Y/%m/)/sbom-${{ github.sha }}.json
  semgrep-scan:
    name: Static Analysis (Semgrep)
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: returntocorp/semgrep-action@v1
        with:
          config: >
            p/python
            p/secrets
            p/aws-lambda
        env:
          SEMGREP_APP_TOKEN: ${{ secrets.SEMGREP_APP_TOKEN }}

Conclusion

The threat landscape facing data platforms in 2026 is qualitatively different from five years ago. Attackers are not just targeting web applications and APIs — they are targeting the supply chain of the data engineering ecosystem: PySpark libraries, Kafka connectors, Airflow providers, and Spark ML packages.

SCA is not a compliance checkbox. It is a fundamental engineering practice for anyone running data infrastructure at scale.

The playbook:

  1. Audit your dependencies today — run pip-audit and trivy fs on your pipeline repos right now. You will find something.
  2. Embed SCA in CI/CD — make dependency scanning a build gate, not an afterthought.
  3. Generate SBOMs — know exactly what is running in every pipeline environment.
  4. Layer cloud-native security — use Wiz to catch the risks that live above the code layer.
  5. Practice incident response — when the next Log4Shell drops, you need to know if you are exposed within minutes.

Your data pipeline is not just code. It is the trust layer between your organisation and its data. Protect it accordingly.

Tags: #DataEngineering #SCA #CyberSecurity #DataPipelines #AWS #OpenSource #Wiz #SupplyChainSecurity