I Run LLMs Locally. Here’s How I Know My Data Isn’t Leaking (And Yours Isn’t Either).

TL;DR (The Bottom Line)

If you download an open-source model (Llama, Mistral, DeepSeek) and run it via llama.cpp or Ollama, your prompts do not go to the cloud. The math happens on your GPU. However, your paranoia about "black boxes" is valid—so let's drop the vendor promises and look at the forensic evidence.

You don't need to keep your whole PC offline. You just need to block the process from phoning home and stop worrying about the NSA, and start worrying about malicious model files and RAM scrapers.

1. The Great Misconception: Cloud vs. Local

We need to separate the products first.

• Cloud AI (ChatGPT, Claude): Your prompt travels across the internet. They do log it. They can train on it. Your fear here is 100% justified.
• Local AI (Your PC): The model file (often 10GB–150GB) sits on your SSD. When you type a prompt, your CPU/GPU crunch numbers locally. Zero text data ever leaves your PCIe bus.

If an app claims to be "local" but requires an internet connection to generate tokens — uninstall it immediately. That is a thin wrapper over an API.

2. Killing the "Black Box" Paranoia (Forensic Auditing)

"One can't say it really doesn't save or upload." As a security analyst, I don't read marketing pages; I read syscalls.

Here is how we prove safety:

• The Wireshark Test: Fire up Wireshark or your OS firewall logs. Type a massive prompt into your local LLM. Monitor outbound packets. You will see zero traffic to external IPs during inference.
• The strace Audit (Linux) / Process Monitor (Windows): We trace the system calls the process makes. When generating tokens, we see futex (threading), cuda (GPU calls), and disk reads. We see zero sendto() or connect() syscalls attempting to reach the internet.
• The Airplane Mode Silver Bullet: Unplug your Ethernet or disable Wi-Fi. Load the model and generate a response. It works flawlessly. If the compute happened in the cloud, it would error out instantly.

Caveat: Some GUI wrappers send telemetry (crash reports). But they cannot exfiltrate your 2,000-word prompt without a massive, easily detectable data spike.

3. The "Always Offline" Myth (Solved by Network Jails)

"The machine should always be offline which is not possible."

Keeping your entire computer offline is a sledgehammer approach. We use scalpels.

You can keep your browser online for research while keeping your LLM in a network jail:

• Firewall by PID: Open Windows Firewall or macOS pfctl. Create an outbound rule that specifically blocks the executable (e.g., ollama.exe, llama-server) from accessing 0.0.0.0/0.
• Localhost Only: Ensure the server binds to 127.0.0.1 (localhost). This allows your browser to talk to the UI, but the LLM process itself cannot route to the public internet.
• The Docker Method: Run your LLM with docker run --network none .... The container literally has no network stack. It cannot phone home, even if it wanted to.

Result: You get to Google things while your AI sits in a silent, isolated bubble.

4. What We Actually Worry About (The Real Threat Model)

Since data exfiltration is off the table for properly hardened local setups, let's shift our paranoia to the actual attack vectors that keep security engineers up at night:

A. The Supply Chain (RCE via Model Weights)

This is the big one. If you download a sketchy, outdated GGUF model that relies on legacy pickle serialization (looking at you, older PyTorch files), a malicious actor can embed Remote Code Execution (RCE) into the weights. The moment you load it, your machine is pwned.

• Mitigation: Only download reputable models from HuggingFace with verified SHA-256 hashes. Avoid older .bin files; stick to modern .safetensors formats, which are designed to prevent arbitrary code execution.

B. Memory Scraping (VRAM/RAM)

Your prompts sit in plaintext in your GPU's VRAM and system RAM. If your machine is already compromised with a keylogger or a memory-scraping trojan (common in enterprise breaches), that malware can read your AI's memory space.

• Mitigation: This is a host-level issue. If your OS is pwned, the AI is pwned. Enforce Full Disk Encryption (BitLocker/LUKS) and Secure Boot. Treat the AI like a database process — secure the host first.

C. Chat Logs on Disk

Enthusiasts love saving histories. Analysts hate unencrypted SQLite databases sitting on NVMe drives. If someone steals your laptop, they can read your entire chain-of-thought.

• Mitigation: Move your chat logs to a RAM-disk (/dev/shm on Linux) that purges on shutdown, or encrypt the application's data directory using OS-level EFS.

5. The Security Engineer's Prescription (Actionable Checklist)

If you want to run local AI securely, don't just download and pray. Do this:

1. Verify Hashes: Always check the SHA-256 of the downloaded model against the official repository.
2. Egress Blocking: Create a firewall rule to block internet access for the specific LLM process PID (or run it in a --network=none Docker container).
3. Bind to Localhost: Never bind the server to 0.0.0.0 (which exposes it to your local network). Keep it on 127.0.0.1.
4. RAM-Disk Logs: Configure the app to store chat histories in a temporary directory that doesn't survive a reboot.
5. Update Religiously: The inference engines (llama.cpp, Ollama) are patched frequently for memory safety vulnerabilities. Treat them like browsers — update them weekly.

The Final Verdict

Cloud AI (ChatGPT/Claude): Data sovereignty risk = Critical. They own your data. Local AI (Properly Hardened): Data sovereignty risk = Zero (with egress blocked). Supply chain risk = Moderate (mitigable). Host compromise risk = High (but that exists regardless).

You don't need to live in a bunker with your Ethernet cable ripped out. You just need to stop treating the LLM like a "web service" and start treating it like a local database daemon.

Block its egress, verify its integrity, encrypt your disk, and generate your tokens in peace. The only entity reading your prompts at that point is the DMA controller on your motherboard — and I promise, it isn't gossiping to the cloud.

Stay paranoid, stay safe, and keep generating.