Post cover image
Photo by Igor Omilaev on Unsplash

July 31, 2026

Why Your AI Engineering Projects Won’t Land You a Job

The 5 levels of AI Engineering projects

By Marina Wyss

9 min read

Want to know what separates entry-level AI engineering projects from the systems being built by the teams behind Claude and Codex?

The gap might seem impossibly wide, but there's actually a pretty clear progression that most AI engineers follow.

Today, I'm mapping out the five levels of AI engineering projects that separate complete beginners from the people at the top of the field.

None

I've coached hundreds of people into AI roles, and I've seen many aspiring AI engineers get stuck building the wrong types of projects that never actually land them jobs.

By the end of this post, you'll understand exactly where you are on this journey and what specific skills you need to reach the next level.

Level 1

At level 1, you're building products using a language model API with some kind of simple interface. For example, a chatbot that answers customer questions or a little feature that summarizes articles for you. Most of the time you're just using the OpenAI or Anthropic API and putting something quick like a Streamlit app in front of it.

You're spending time on prompt engineering, but you don't necessarily track or test your prompts. You have some structured output and maybe some tests to ensure it parses correctly, and perhaps you've even put some logic in place to handle fallbacks when the model flakes out on you.

None

If I was an absolute beginner and wanted to get here, I'd start with some free YouTube tutorials to learn what tools are out there and how the pieces fit together.

Then I'd spend a weekend building something that helps me solve a problem in my real life, like maybe a meal prep coach.

But the thing is, whatever you build will probably only work when it works. Because you're not tracking your prompts, you don't know if your changes are actually helping. You don't have metrics to evaluate the quality of your outputs. And your bot only knows what it's been trained on and whatever it can find on the web, which means it's missing a lot of helpful context your company might have.

Level 2

When you start addressing these issues, you reach level 2. You know you need to build a more robust system, and maybe you've been asked to make a system that can use your team's internal documentation.

So you start adding retrieval-augmented generation to your project. This lets the model interact with internal knowledge bases to give it context on how to answer questions specific to your team or company.

None

But just adding RAG is the easy part. The hard part is knowing if it is giving you correct answers, and if it messes up, figuring out why and how to fix it.

That means understanding how to properly chunk up your documents, how to embed them properly so you can store them in a vector database, and all the different ways we can use to find the right chunks when a new question comes in.

It also means setting up really specialized evaluation methods for each component of your system, as well as for the system as a whole. Not only do you need to what metrics to use, you also need to know what good looks like and how you're going to measure if the responses actually serve the customer need.

If I wanted to get to this level from level 1, I'd take the chatbot I already built and hook it up to something the model has never seen, like for that nutrition coach agent maybe I'd share my nutrition logs or favorite family recipes.

None

Once you have perfected this, you'll have an AI system that's actually really useful. You'll be able to ask it questions about your own data and get custom responses back. Not only that, you'll feel completely confident that you've built something useful and accurate.

Which is all great and good, but what happens when you want to go one step further and have the AI actually do something with that information?

Level 3

Once you start building more complex AI systems, the natural next step is to want your AI to be able to take action, not just answer questions. That takes us to level three: AI agents.

An agent is a model that can use tools and make decisions on its own. You give it a goal, and it works in a loop, deciding what to do, taking actions, checking the result, and figuring out its next move, over and over until the job is done. It can do things like search the web, call other APIs, run code, or update a database.

None

What this looks like in practice might be a research agent that gathers information from dozens of sources and helps you write a brief or maybe a customer support agent that looks up orders and processes customer requests on its own.

In my opinion this is the most fun stuff in AI engineering right now, but it's also some of the most dangerous.

Because every bit of power that you give the AI agent is another opportunity for things to go very wrong. Back at level two, an evaluation meant scoring a single answer, but now you're grading a whole bunch of different decisions. If the agent takes a wrong turn at any step, everything after that is built on that mistake.

Not to mention, there are real consequences when your agent makes a mistake with your credit card, or your code, or your email. The stakes are inherently much higher.

Once you can build an agent that handles all that responsibly, you've got something really powerful. Your AI can go and complete full multi-step tasks for you, start to finish, saving you hours of work. And it's really fun to build something that goes out and actually does things.

If I was making this move from level 2, I'd start by giving the system I already had a real set of tools, but I'd keep them read-only at first. So let's say I'm still working on that nutrition coach: the agent can search my meal logs, use a nutrition calculator, and search the web, but it can't change anything yet.

None

Then I'd get the loop working where the agent decides, takes an action, checks the result, and goes again, across all of those tools. I'd make sure I had observability instrumented and was confident the agent was reliable and secure before giving it more capabilities.

None

But there's still a gap between a single agent that's useful for you or maybe as an internal tool, and a multi-agent system that can be used reliably by potentially millions of users.

Level 4

Which brings us to level four: Enterprise-scale AI systems. When people talk about AI engineering positions paying north of $300,000 a year, these kinds of systems are what those people are building.

Instead of one agent trying to do everything, you've got a bunch of specialized agents that hand work off to each other. Maybe one agent breaks down the request and routes it, a few others each handle their bit, and another one checks the work before it goes out.

What this looks like in practice might be a customer support system where one agent reviews and routes the ticket, a billing agent and a technical agent handle their parts, and a final agent reviews the whole response before a customer ever sees it. All running for millions of people at the same time.

None

But coordinating a team of agents is a whole different beast than running one. Every agent you add is another thing that can break, and one agent's mistake gets passed straight down to the next. And now it's live, in front of real customers, where a bad response is public and expensive.

Interestingly, when researchers looked at the code behind Claude Code, the actual AI was a tiny sliver of it. Almost everything else was the scaffolding that keeps the whole thing reliable.

So most of the real work at this level is everything AROUND the agents. Beyond the stuff from all the previous levels like structured prompt engineering, RAG, and evaluation, this might be things like tracing, so you can see exactly what each agent did when something goes wrong and guardrails so nobody can trick an agent into leaking data. Not to mention keeping cost and latency under control.

If I was making the jump from level 3, I'd split my nutrition coach into a small team: one agent that plans the meals, one that builds the grocery list and maybe even orders groceries for me, and a checker agent that reviews the plan before I see it. Then I'd try to break it on purpose, like feed the planner a fake ingredient and see if the checker actually catches it, or just waves it through. The millions-of-users part is honestly something you'll only get to build on the job, but we can still build something that COULD scale by building on cloud services and thinking about cost and reliability from the start.

Once you can build teams of agents that hold up at scale, you've built something a company can actually run its business on, and will earn the salary that goes along with that value.

But even at level four, you and your team are still the ones building all of this. You're the bottleneck.

Level 5

So at level five, you get yourself out of the way. This is the most advanced work happening in AI engineering right now, and it's where you stop building the system by hand and start giving agents the job of building it themselves.

This goes way beyond just using coding agents to write the code, which you'll be doing at every level. It's even beyond spec-driven development where you tell the agent what you want and let it implement it autonomously.

Top teams like the ones building Claude Code and Codex are no longer directing agents through every change. They've started building systems that improve themselves. The idea is to set up a cycle where an agent looks at how the system is performing, comes up with a change that might make it better, makes that change, tests whether it actually helped, and then keeps it or throws it out. Then it does the whole thing again. And again. For hours or even days, with no human stepping in.

None

Andrej Karpathy ran a project where a single agent did around 700 experiments over two days completely on its own, and came back with twenty different ways to make model training faster. Anthropic recently shared that most of the code getting merged into their own codebase is now written by Claude. It's starting to build itself.

That whole cycle only works if the agent can correctly judge its own work, over and over, without anybody checking. And that turns out to be pretty hard to get right.

Because if your scoring is even a little bit off, the agent won't fix the product. It'll find the flaw in your scoring and exploit it, racking up a great score while the actual product gets worse in ways you never thought to measure.

So the skills at this level are serious. You have to design an evaluation the agent can't cheat, and make sure its performance is tied to what actually makes the product good, so it can't just manipulate an easy number. You also need to instrument everything, because you can't watch it run live anymore, you can only read the trail it leaves behind. And the last part is knowing exactly where a human still needs to step in, because even the best teams in the world aren't completely hands-off. People still pick the problem, design the scoring, and approve what actually goes into production.

So this isn't a level you're going to get to from self-study in a couple of weeks. Every skill here is something you learn from years of actually running systems in production, seeing the ways they break, and getting better over time.

Luckily, most of us don't actually need to get to level 5! Most of my work on production AI systems at Twitch is realistically level 2 or 3. Sometimes all you need is a well-done prompt and some simple evaluation to build something useful. And we're hired to be useful, not fancy.

— — —

If you're feeling like you need more support with your career, here are some other ways I can help: