Friend link for nonmember- https://medium.com/@inprogrammer/my-python-script-ran-for-3-hours-i-fixed-it-in-20-minutes-f483bc0cd279?sk=b0bb60283e64c8791bd5f1bea80158da

I had a Python script that processed customer transactions every night. The data wasn't tiny, but not huge either. Around 800,000 rows, some calculations, a few aggregations, then export to a file.

It should have taken about ten minutes.

After three hours, it was still running. No errors. No warnings. Just slow and stuck.

I did what most developers do first. Stared at the code. Added print statements to see where time was going. Made a few guesses and tried some changes. Nothing really helped.

Then I stopped guessing and started checking properly. A profiler found the slow part in two minutes. The fix took eighteen more.

Here is exactly what happened.

Setting Up the Profiler

The tool I used was cProfile, which is built into Python and requires no installation. For visualizing the output I used snakeviz, which turns the profiler data into an interactive chart you can explore in a browser.

pip install snakeviz

Running the profiler against the script is one command:

python -m cProfile -o profile_output.prof slow_script.py

Visualizing the results is another:

snakeviz profile_output.py

This opens a browser with an interactive flamegraph showing every function call, how many times it was called, and how much total time was spent inside it.

The output was immediately clear.

What the Profiler Found

One function was responsible for 94% of total execution time. Not the database connection. Not the file writing. Not the aggregations.

A function called find_duplicate_transactions that I had written myself six months ago and never looked at since.

Here is what it looked like:

def find_duplicate_transactions(transactions):
    duplicates = []
    for i, transaction in enumerate(transactions):
        for j, other in enumerate(transactions):
            if i != j:
                if transaction["id"] == other["id"]:
                    duplicates.append(transaction)
    return duplicates

Spot the problem?

This was a nested loop. For every transaction, it checked every other transaction to find duplicates.

With 800,000 rows, that means 800,000 × 800,000 comparisons. Around 640 billion operations.

In computer science, this is called Time Complexity, specifically O(n²). It works fine on small data. On large data, it becomes painfully slow.

In my tests, I used just 500 rows. The function ran in milliseconds. In production with 800,000 rows, it ran for hours.

The print statements I added earlier never showed this problem. I was looking in the wrong place. I assumed the slow part was the database or file handling because those are the usual suspects.

The real issue was a function I had written and completely forgotten about.

The Fix

The fix was replacing the nested loop with a set lookup.

def find_duplicate_transactions(transactions):
    seen_ids = set()
    duplicates = []
    for transaction in transactions:
        if transaction["id"] in seen_ids:
            duplicates.append(transaction)
        else:
            seen_ids.add(transaction["id"])
    return duplicates

Instead of comparing every transaction with every other one, this version goes through the list just once.

For each transaction, it checks if the ID is already in a set. In Python, set lookups are constant time, which means they are fast no matter how big the data gets.

The overall Time Complexity dropped from O(n²) to O(n).

For 800,000 rows, that is the difference between 640 billion checks and just 800,000.

That is not a small improvement. It completely changes how the code behaves as the data grows.

The Results

Before the fix, the script ran for over three hours on 800,000 rows and still did not finish during my testing.

After the fix, the same script on the same data finished in 4 minutes and 12 seconds.

I ran it three times to be sure. Four minutes. Four minutes. Four minutes and 8 seconds.

Everything else in the script, database calls, aggregations, file writing, all of it together took under three minutes.

One nested loop was taking more time than the entire rest of the script combined.

Why This Was Invisible Without Measurement

Print statements did not help because they only show when code runs, not how long it runs or how often it is called.

I added prints around database operations. The timing looked normal, so I assumed the issue was somewhere else. I never checked find_duplicate_transactions because it felt like a small, harmless utility.

That assumption was wrong.

A profiler does not depend on your guesses. It measures everything and shows where time actually goes. The flamegraph made it obvious. One function was taking 94% of the total time.

That is the real difference between guessing and measuring.

Guessing works only if you suspect the right place. Measuring shows you the right place, even when you do not.

The Profiling Setup I Now Use on Every Script

After this experience I added profiling to my standard workflow for any script that processes significant data. The setup takes two minutes and has caught three more performance problems since then before they reached production.

For quick profiling during development I use line_profiler, which shows execution time per line rather than per function:

pip install line-profiler

Add the decorator to the function you want to measure:

from line_profiler import profile
@profile
def find_duplicate_transactions(transactions):
    seen_ids = set()
    duplicates = []
    for transaction in transactions:
        if transaction["id"] in seen_ids:
            duplicates.append(transaction)
        else:
            seen_ids.add(transaction["id"])
    return duplicates

Run with:

kernprof -l -v slow_script.py

This outputs a line by line breakdown showing exactly how much time each line consumed and how many times it executed. For debugging a specific function it is faster than cProfile because the output is more focused.

For production scripts I keep cProfile with snakeviz because the flamegraph gives a better overview of the entire script's behavior.

The Broader Lesson

Every slow Python script I have debugged since then follows the same pattern.

The bottleneck is almost never where I expect. It is usually in code I wrote, trusted, and stopped thinking about.

Nested loops show up more often than you think. They are easy to write and look clean. On small data, they feel fine. On large data, they become a serious problem.

There are other patterns like this too. Running a database query inside a loop instead of fetching data once. Reading the same file again and again instead of loading it once. Recalculating the same value instead of caching it.

All of these look harmless in code review. At scale, they slow everything down.

A profiler finds them in minutes. Guessing can take days

CTA

If this helped you find a bottleneck, follow me on Medium. I write about real Python production problems with fixes and numbers, not theory.

Found something similar in your code? Share it in the comments. I read every one.

And if this was useful, send it to a developer whose script runs slower than it should. It might save them a long night.