Post cover image

July 26, 2026

I Wasted 4 Months Blaming the Model. The Real Problem Was Context Engineering.

What context engineering actually means, why AI agent memory quietly breaks, and the simple 4-layer setup I now use in every single…

By Tarun Singh

7 min read

What context engineering actually means, why AI agent memory quietly breaks, and the simple 4-layer setup I now use in every single project.

My first real AI agent was beautiful. In the demo.

It answered support questions, pulled order data, wrote follow-up emails. My client clapped. I felt like a genius for about nine days.

Then real users touched it.

It forgot what the user said two messages ago. It called the refund tool when someone asked about shipping. One time it confidently invented an order ID that did not exist, and the customer actually replied to it. I still think about that one.

So I did what almost every developer does. I blamed the model. I switched to a bigger, more expensive one. My bill went up roughly 3x. The bugs stayed exactly where they were, just now with better vocabulary.

It took me an embarrassing amount of time to accept the truth: the model was never the problem. My context engineering was garbage.

What context engineering actually is (in plain English)

Here's the simplest way I can explain it.

Prompt engineering is writing a good question. Context engineering is deciding what information the model is allowed to see before it answers that question.

Think about hiring a genuinely brilliant new employee. Day one, you sit them at a desk. If you hand them no files, no customer history, no idea what happened last week — they will fail. Not because they are dumb. Because you gave them nothing to work with.

Now flip it. You dump 400 pages of documentation, 60 tool logins, and eight years of Slack history on their desk and say "figure it out." They will also fail. Different reason, same result.

Context engineering is the job of finding the middle. The right stuff, at the right time, in the right order, and nothing else.

That's it. That's the whole discipline. Everything below is just me learning that the hard way.

Mistake 1: I thought a bigger context window would save me

When models started shipping with a million token context windows, I genuinely thought the problem was solved. Just throw everything in. Full chat history, all the docs, the whole product catalog. Let the model sort it out.

It got worse.

There's a thing people now call context rot, and once you see it you can't unsee it. As you stuff more into the window, the model's attention spreads thin. Stuff in the middle gets quietly ignored. The model starts confidently answering from the wrong chunk. Your accuracy drops while your token bill climbs.

A million token window does not mean a million useful tokens. It means a million tokens you are now paying for and probably confusing yourself with.

I cut my average context from about 40k tokens down to under 6k. Answers got sharper. Latency dropped. Cost dropped a lot. That was the week context engineering stopped being a buzzword to me.

Mistake 2: I gave my agent 40 tools "just in case"

This one still makes me wince.

I had a search tool, three different lookup tools, two email tools, a calendar tool, a database tool, a refund tool, an escalation tool. Plus a bunch of MCP servers I connected because they seemed useful and connecting them was one click.

Every one of those tools has a name and a description sitting in your context. All the time. Even when it's completely irrelevant to what the user asked.

So your agent is trying to pick between 40 similar sounding options, half of which it will never need for this task. Of course it picks wrong sometimes. I would pick wrong too.

I cut it down to six tools for the main flow. Tool selection accuracy went from "mostly fine, sometimes horrifying" to actually reliable. If a task genuinely needs a different toolset, I now load a different toolset for that task. Tools are context. Treat them like context.

Mistake 3: I thought chat history was memory

This was my biggest one, and I see almost everyone do it.

Chat history is a transcript. Memory is different. A transcript is everything that was said. Memory is the small set of things worth remembering.

Once I split them properly, AI agent memory finally started working. Here's the breakdown I use, in normal human words:

Working memory — what's happening in this exact task, right now. Dies when the task ends.

Episodic memory — what happened in past sessions. "Last Tuesday this user had a delivery complaint."

Semantic memory — stable facts about the user or the world. "This user is on the enterprise plan. Their timezone is IST."

Procedural memory — how we do things here. "Always confirm the order ID before issuing a refund."

Most broken agents I've looked at are stuffing all four into one giant blob of chat history and hoping the model figures out which is which. It doesn't. It just gets slower and more expensive at being confused.

The 4-layer context I now build for every agent

After all that pain, I settled into a structure I use everywhere now. Four layers, assembled fresh on every single turn:

  1. Instructions — who the agent is, the hard rules. Short. Never changes.
  2. Memory — compressed facts about this specific user. A few lines, not a novel.
  3. Retrieved knowledge — the RAG layer. Top 3 to 5 chunks. Never more.
  4. Live state — the current task, available tools, recent turns only.

Here's roughly what that looks like in code. I've kept it deliberately simple so you can drop it into anything:

# The core idea of context engineering:
# you BUILD the context fresh every turn. You don't just append forever.

def build_context(user_question, user_id, all_tools):
    """
    Assembles the 4 layers into one prompt.
    Order matters — the model pays most attention to the start and the end.
    So instructions go first, and the actual question goes LAST.
    """

    # Layer 1 — Instructions. Keep this short and boring.
    # Long system prompts are where people waste half their context budget.
    instructions = (
        "You are a support agent for an online store. "
        "Only use the facts given below. If you don't know, say you don't know. "
        "Never guess an order ID."
    )

    # Layer 2 — Memory. Facts about THIS user, already compressed.
    # Notice we pull 3 facts, not 300. More is genuinely not better here.
    memory_facts = memory.recall(user_id, user_question, limit=3)

    # Layer 3 — Retrieved knowledge (your RAG step).
    # I capped this at 4 chunks after testing. 10 chunks made answers worse,
    # not better, which surprised me at the time.
    docs = vector_db.search(user_question, top_k=4)

    # Layer 4 — Live state. Only the tools relevant to this task,
    # and only the last few turns of conversation. Not the whole transcript.
    tools = pick_relevant_tools(user_question, all_tools)   # returns ~5, not 40
    recent = conversation.last_turns(n=4)

    # Now glue it together. The question sits at the very end on purpose.
    return f"""{instructions}

WHAT WE KNOW ABOUT THIS USER:
{format_list(memory_facts)}

RELEVANT DOCS:
{format_list(docs)}

RECENT CONVERSATION:
{format_list(recent)}

AVAILABLE TOOLS:
{format_list(tools)}

USER QUESTION: {user_question}
"""

Nothing clever there. That's kind of the point. Good context engineering is mostly discipline, not cleverness.

The memory part, where most people go wrong

The mistake I made for months was saving everything. Every message went into the memory store. Which meant my memory store became a second, worse transcript.

The fix is a write filter. Most of what a user says is not worth remembering. "ok thanks" is not a memory. "I'm allergic to dairy" absolutely is.

class SimpleMemory:
    """
    Small memory layer. Two rules that fixed 90% of my problems:
    1. Be very picky about what you WRITE.
    2. Be very picky about what you READ BACK.
    """

    def __init__(self, store):
        self.store = store  # any vector DB or even a plain dict works to start

    def maybe_save(self, user_id, message):
        # This is the write filter. Don't skip it.
        # Ask the model one cheap yes/no question instead of saving blindly.
        prompt = (
            f"Is there a durable fact about the user in this message? "
            f"A preference, a constraint, an ID, a decision. "
            f"If yes reply with just the fact in one short line. If no, reply NONE.\n\n"
            f"Message: {message}"
        )
        fact = cheap_model(prompt).strip()

        if fact == "NONE":
            return  # most messages end here, and that's healthy

        self.store.add(user_id=user_id, text=fact, created_at=now())

    def recall(self, user_id, question, limit=3):
        # Pull more than we need, then rank, then cut hard.
        candidates = self.store.search(user_id, question, top_k=15)

        for c in candidates:
            # Simple blend: how relevant it is, plus a nudge for recency.
            # Old facts about a user are often stale — a plan they upgraded,
            # an address they moved from. Recency deserves a vote.
            age_days = (now() - c.created_at).days
            recency_boost = 1 / (1 + age_days * 0.05)
            c.score = (c.similarity * 0.75) + (recency_boost * 0.25)

        candidates.sort(key=lambda c: c.score, reverse=True)
        return candidates[:limit]   # hard cut. this limit is doing real work.

That limit=3 looks trivial. It is not. It's the difference between an agent that knows the user and an agent that drowns in trivia about them.

How I actually test this now

I used to test agents by chatting with them and going "yeah that feels better." That is not testing. That's vibes.

Now I keep a boring text file with 30 real questions from real users, and the answer each one should produce. Before and after any context engineering change, I run all 30 and log two numbers: did it answer correctly, and how many tokens did it burn.

# Not fancy. Fancy is the enemy here. This little loop saved me weeks.
def run_eval(test_cases):
    passed = 0
    total_tokens = 0

    for question, expected in test_cases:
            context = build_context(question, user_id="test_user", all_tools=ALL_TOOLS)
            answer = model(context)
            total_tokens += count_tokens(context)
            # Let a model judge it. Cheaper and more honest than me eyeballing 30 answers.
            if is_correct(answer, expected):
                passed += 1
        print(f"Accuracy: {passed}/{len(test_cases)}")
        print(f"Avg context size: {total_tokens // len(test_cases)} tokens")

The first time I ran this properly, I found out that a change I was really proud of had made accuracy worse while making the context bigger. I'd have shipped it. That loop pays for itself immediately.

The checklist I run before shipping any agent now

  • Is my system prompt under roughly 200 words? If not, what's actually earning its place?
  • Am I retrieving 3 to 5 chunks, or am I retrieving 20 because it felt safer?
  • Does this task really need all these tools, or did I just connect things because I could?
  • Is my memory store saving facts, or is it quietly saving a second transcript?
  • Is the user's actual question the last thing in the context?
  • Do I know my average context size in tokens? An actual number, not a feeling.
  • Do I have 20 to 30 test cases I can run before and after a change?

The thing I wish someone had told me on day one

Everyone is out here shopping for a better model. New release drops, everyone migrates, everyone posts benchmarks.

Meanwhile the agent is failing because it's being handed 40 tools it doesn't need, a transcript it can't parse, and twelve document chunks where three would have done the job.

The models got very good while I wasn't paying attention. Mine was never the bottleneck. The pipeline feeding it was. Context engineering is not a fancy new term for prompt engineering — it's the actual system design work around the model, and right now it's the highest leverage thing most of us can improve.

Fix your context first. Then go shop for a bigger model, if you still need one.

You probably won't.

If this saved you even one weekend of debugging, hit the clap button — it genuinely helps this reach other people stuck on the same thing. And if you've got your own context engineering horror story, drop it in the comments. I collect them now.