My Code Used 10GB RAM. Now It Uses 300MB

Years of writing inefficient Python code taught me one thing: speed doesn't always come from better hardware. Sometimes, it's just a smarter line of code. Let me paint a picture.

A few months ago, I was building an internal document ingestion tool that read thousands of PDFs, parsed them into JSON, ran some NLP embeddings, and stored the results. It was functional. It was accurate. But man… it was slow. And memory-hungry.

Every run ballooned up to 10GB RAM usage. My fans sounded like a jet taking off. Restarting the process felt like rebooting a dying planet.

But today? The same code does the same work using 300MB RAM. No cloud GPU rentals. No Kubernetes wizardry. Just Python. And just three tricks.

Trick 1: Forget Pandas (Temporarily)

I'm a Pandas lover. But in data-heavy loops, it's a memory hog. For one pipeline, I was converting all my scraped JSON into a Pandas DataFrame first before processing. Classic beginner move.

So I swapped this:

import pandas as pd

df = pd.read_json("large_data.json")
processed = df[df["score"] > 0.8]

import pandas as pd

df = pd.read_json("large_data.json")
processed = df[df["score"] > 0.8]

With this:

import json

with open("large_data.json") as f:
    data = json.load(f)
processed = [entry for entry in data if entry["score"] > 0.8]

import json

with open("large_data.json") as f:
    data = json.load(f)
processed = [entry for entry in data if entry["score"] > 0.8]

This single change dropped my memory footprint by over 2GB. Turns out, Pandas is overkill when you're just filtering large dictionaries in a loop.

Pro tip: Pandas is great for analysis. Not transformation. For processing JSON or text, native Python is leaner and faster.

Trick 2: Use Generators Like Your Life Depends on It

You don't need to load the entire file into memory to work with it. That's the biggest lie most devs believe. Instead of reading all at once, stream.

Here's what I did wrong initially:

with open("embeddings.txt") as f:
    vectors = f.readlines()

with open("embeddings.txt") as f:
    vectors = f.readlines()

What I replaced it with:

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.strip()

vectors = read_lines("embeddings.txt")

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.strip()

vectors = read_lines("embeddings.txt")

This lazy iterator approach not only saved memory, but it helped me chain transformations like filtering, parsing, and embedding — all without extra memory spikes.

Trick 3: Del Garbage Like a Boss

I had some stubborn memory leaks. Turned out, large variables I thought were released… weren't. Python's GC (Garbage Collector) is lazy sometimes. Especially in loops.

Here's how I fixed it:

import gc

for file in all_files:
    data = load_and_process(file)
    save(data)
    del data
    gc.collect()

import gc

for file in all_files:
    data = load_and_process(file)
    save(data)
    del data
    gc.collect()

That gc.collect() call alone shaved off gigabytes during long runs. If you're processing 10,000 files in a loop and not manually deleting unused variables, you're hoarding memory.

Automation tip: Wrap this pattern into a decorator and reuse it:

def clean_memory(func):
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        del args, kwargs
        gc.collect()
        return result
    return wrapper

def clean_memory(func):
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        del args, kwargs
        gc.collect()
        return result
    return wrapper

But Wait — Why Was This Not Obvious?

That's the funny part. These tricks aren't some niche voodoo. They're basic. But when we're too focused on feature velocity or model accuracy, we forget to revisit performance.

I didn't optimize because I was too busy building. But when things started breaking — out-of-memory errors, unresponsive UIs — I had no choice.

Lessons I'll Never Forget

If it doesn't need to be in memory, don't load it.
Generators > Lists when working with streams of data.
Garbage collection needs a push sometimes.
Pandas is great, but use it with restraint.
Optimization is boring — until it saves your job.

Real Impact

Before optimization:

RAM usage: 10.2 GB
Runtime: 1 hour 34 minutes
Crashed randomly after 800 PDFs

After optimization:

RAM usage: 312 MB
Runtime: 41 minutes
Never crashed again

Closing Thoughts

This isn't about premature optimization. It's about intentional design. You don't need AI to solve AI problems. Sometimes, the solution is hiding in a Python trick you ignored for years.

If you take anything from this article, let it be this:

"Every byte of memory you don't use is a byte you never have to debug."

Medium's audience is smart. You already know these things. But the trick is actually using them. Automate the boring stuff. Clean memory like you clean your kitchen. And stop feeding your code endless gigabytes.

Here, if you are a beginner and need some solid advice personally, I will recommend my personal guide:

Python Starter Guide for Non Programmers Want to learn Python but don't know where to start?This guide was made exactly for you.Whether you're a freelancer…

Before you go

Please take a moment to like the post and follow the writer!
Did you know that over 400,000 developers share what they're building, learning, and discovering across our platforms every month? Learn how you can contribute here

Contents