2 Weeks Running an AI Software Factory

0. Table of contents

This is a longer article, and goes relatively deep in a few areas, so a guide to what's inside if you'd like to jump around or already have background:

1. Background: Software factories and autonomy
2. The goal: A music making tool
3. The initial factory design
4. Execution + evolution
5. Lessons learned after two weeks
6. The core problem: Legibility and review
7. Next steps: Cost optimization, custom harnesses, and visualization
8. Closing thoughts

1. Background: Software factories and autonomy

By now, it's fairly common knowledge that agents capable of longer-running tasks and more ambitious work. In particular there's interest in "software factories": autonomous multi-agent systems that spec the work, break it down into tasks, execute the tasks, control for quality, and ultimately ship and deploy without direct supervision.

By now I have a couple years of hands-on experience directing multiple agents at once. Tmux, worktrees, /workflow, and multi-session management in Claude and Codex each unlocked more productivity with coding agents. But a software factory pushes the multithreading to an extreme, where you don't supervise the individual coding agents at all.

This requires creating a system, not just managing conversations. Steve Yegge pioneered this with Gas Town, and his recent follow up article was a major inspiration for this experiment and this piece.

I've been interested for a while in getting a software factory up and running for myself. I've approached this, but not yet taken the full plunge. I've been waiting for the right project; one where I have the motivation to create it and the expertise to evaluate the outcome, but lacked the time to build it hands-on. A couple weeks ago, I had an idea that seemed worthy of the challenge.

2. A music making tool, and a factory to build it

The creative seed

When I worked at Splice, we learned about the power of creative seeds to help people get musical ideas off the ground. Especially for beginners, it's often easier to add ideas than to start from scratch. Many producers find a sound or loop and build an entire track around it.

When working this way, it's often hard to feel creative ownership. Finding a loop can spark the joy of discovery, but more is needed to feel a connection to what started as someone's idea. From the earliest days of hip-hop and house, finding and composing with interesting samples has been a rich source of inspiration and foundation for entire genres, but debates started from the very beginning about what constitutes "valid" creativity. How can you build a satisfying creative workflow with loops that creates music that is "yours"?

I've believed in a better core loop of hunting, matching, and shaping sounds since 2018. I made many sketches and prototypes for a new kind of "DAW" (digital audio workstation, like Garageband, Ableton, or FL studio), but lacked the programming skills and time to build it for real. In particular the hardcore DSP (digital signal processing) capabilities like timestretching and low-latency audio effects were beyond my reach. Most DAWs are built by teams of deep specialists. This is not weekend project territory, at least not at any time before our present, strange moment.

But when I caught up a couple weeks ago with an old Splice friend, philosophizing on the current state of AI music making tools, the problem space reactivated in my brain.

Constraints

I realized this would be a great test case for a software factory:

• A clear idea: I knew what I wanted and could describe it in detail
• Ambitious scope: more than I could efficiently build in 1:1 AI chats.
• A hard problem: Low latency audio playback, sample timestretching, and audio effects pushed this project beyond a template React app + ShadCN.
• Little time: a full schedule with no time to directly supervise the work. It's very busy at Miro right now (big launches on MCP and Sidekicks!), my parents were visiting, and I was still catching up from a recent trip to Japan. My rule was: an evening check-in, less than one hour of work per night before bed.
• A hobbyist budget: I have my own AI accounts for personal projects and experiments. I see the usage meter on my personal Claude and ChatGPT as a challenge: use it or lose it! I wanted to see what I could do with around $300 (a Claude Max 20x and a ChatGPT 5x).

Kickoff

We started with one of my go-to skills: "spec". Spec is simple. I dictate a ramble about what I want to build. Then the agent grills me, dissects the idea, challenges premises, and resolves ambiguity. Next we build out the data model and taxonomy. These inputs are refined into detailed documentation.

In this case, with months of previous notes and deep thinking on the topic, my ramble was longer than usual. Over two hours of dictation, I dumped out all of my ideas and refined them through a structured interview, crafting a detailed description of Dial.

The concept was simple: a creative workflow built around one magic dial, that could search for sounds, shape and sculpt, and create endless variety. The dial explores creative spaces, prepared for you by an assistant agent on command. The agent is your studio assistant, but the search space, stacking, shaping, and structure is all up to you.

So far, the workflow was similar to other "kickoffs" of agentic work I've run in the last year. But this time, I also used /spec to define the rules of operation for how the idea should be built. The end result was a folder of 11 detailed specification documents, covering every aspect of the product and the factory.

3. The initial factory design

Fable and I co-designed the initial state of the factory as follows:

Core orchestration

• Primary interface was Claude desktop app, in "code" mode. This got more flexible later, but started simple.
• The factory is orchestrated by one "lead" agent. This agent is my primary point of contact with the factory team and is explicitly instructed not to do work itself, always delegating to sub-agents. I started with Fable as the lead, though I experimented with this as well.
• Beads is used to track progress. Shoutout again to Steve Yegge (his explainer here). Beads is excellent and I highly recommend it. The orchestrator's responsibility is to break the spec down into beads, delegate to agents, and guide progress towards a complete implementation of the spec over time.
• The orchestrator is instructed to choose the correct model for the task. I gave light guidance: front-end tasks should typically be delegated to Sonnet, or Opus when they touch something structural like componentization. Architectural changes, especially ones that touch the underlying data model, or the scaffolding of a net new capability area, should be delegated to Fable (unless they are straightforward.)

Decisions and documentation

• Major decisions should be escalated to me as they arrive. The orchestrator is responsible for these escalations. I check in on the factory in the evenings, and resolve escalations through review. When we make a decision together, it is ratified into a rolling decision log.
• The spec is the source of truth. Incoming work is reviewed against the spec. If decisions change the spec, or we learn during implementation, the spec is updated. Spec is also part of the codebase, and spec changes are subject to review just like code changes.
• All agent traces are recorded and made legible to the orchestrator agent so that the factory can "reflect" on itself.
• The factory protocol is included in the codebase, and can be edited via pull requests like any other code.

Merging and review

• All sub-agents work in worktrees and open pull requests against main. Work is always pull requested, which has a nice side effect of auto-documenting the evolution over time.
• The entire codebase is covered by tests, and CI runs on every push. Failing branches cannot be merged, and branches without test coverage should be marked as blocked.
• Pull requests cannot be merged until they have undergone architectural review by an adversarial architectural agent. The adversarial agent is an instance of GPT 5.6 Sol that watches for new pull requests. It is also configured to run on an hourly cron to catch anything that the PR watch may have missed. It's prompted to be a steward of code quality and make comments on pull requests, just like a human reviewer.

Finally, both Claude and ChatGPT were configured to run in cloud mode, not local, so I can check their progress at any time from my phone.

With that — I set Fable to run in "auto" mode, and started the factory.

4. Execution + evolution

Phase 1: A running start

Fable orchestrated the foundation sprint beautifully. By the end of day 1, we had a working WASM audio engine, application shell, and basic audio playback up and running. By day 2, we had a first implementation of the dial that could be used to "scan" through a list of samples. By day 3, we could match and stack loops and play back looping song ideas. In the Wednesday evening check in, I felt creative fun in the prototype for the first time.

Phase 2: Attempting to scale up orchestration

From early on, I was bottlenecked on orchestrator speed. The orchestrator would typically only delegate around 2–3 tasks at a time. Especially early on, a lot of the work was serial. But even given that, the orchestrator didn't always effectively identify parallelization opportunities.

To solve this, I briefly tried creating subteams with dedicated ownership areas and their own orchestrator. I started with a subteam to work on an audio visualizer and a subteam to work on core engine audio effects.

The core engine audio team worked flawlessly. In two days we had a working Rust port of a time-stretching library and a small portfolio of audio effects. The main orchestrator implemented it across Dial, and sample playback quality improved immediately across the board.

The visualizer team was not as effective. I quickly realized that high-taste UI work didn't work well without supervision. (More on that later.)

Despite some success, I abandoned this approach. With the time and attention I had available, managing multiple orchestrators was too much overhead. It created practical problems: overcoming Beads sandboxing, dealing with PR and CI ownership. Most importantly, orchestrator compute proved a precious resource, and running multiple orchestrators burned it faster. I found that "slow and steady" simple setups that ran 24 hours beat more complex configurations that got stuck.

Phase 3: Handoff + Dealing with plan limits

In general, Fable credits would run out after 48 hours. I got lucky the first time, as my reset happened to coincide with my first usage limit. But after the second, the factory sat idle for Thursday and Friday. I briefly tried switching the orchestration to Opus 5, but it proved prone to micromanaging. I used my review hour Saturday morning to design a handoff protocol to GPT 5.6 Sol. Opus recorded the current state of the factory, updated all beads and documentation, wrote a HANDOFF.md brief, and defined procedure for future handoffs. By Saturday afternoon Sol was running the factory.

The handoff protocol proved important. I am hitting plan limits constantly, so handoffs are a key tool to keep things moving by rotating between accounts. The first couple were rocky—work was repeated and overwritten, decisions were lost, branches abandoned. But the traces helped debug and iterate the protocol. I am now on my fifth handoff and everything works smoothly.

Sol had already done a great job of architecture review, and did a great job running orchestration as well. Sol has a different style, and proved to be equally capable in a different way. Less bookish, more action oriented, a bit less imaginative, a bit more reliable. Maybe slightly sharper and more analytical. A bit hard to explain, but you'll build the same intuition if you work with both. Plenty more to say about both Fable and Sol, but after handing off back and forth a few times, I generally prefer working with Sol for orchestration. However, it is much more prone to rabbitholing on deep technical problems, and is more of a micromanager compared to Fable.

Phase 4: De-slopping and refactoring

By the end of week 1, we started hitting walls in the foundation layer. As we started work on the timeline view, core engine assumptions about loop sequencing and scheduling started to fail. A rework of core objects was needed as they became more (ahem) "load bearing."

By Saturday night, Sol and I agreed it was time for a major refactor of the scheduling system and sample data model. I was nervous about this, as it was the ultimate test — could the codebase survive a major rework of critical internals across many agents and the inevitable handoff halfway through?

The answer was "mostly". After a Sunday planning session the work kicked off with Sol driving, thanks to a well-timed usage reset from OpenAI. As Sol orchestrated the refactor, the biggest issue wound up being repeated CI failures. Parallel agent work caused conflicts, and Sol's unbound debugging enthusiasm burned multiple expensive, lengthy roundtrips while the ground was still shifting. Monday through Wednesday was boring and frustrating, reviewing in the evenings to see commit after commit (and valuable Sol usage) spent to "get tests to pass".

But we got through. By Thursday, feature work was moving forward again, just in time for Fable usage to reset for the week. Sol ran the handoff protocol, and Claude picked up the work once more.

Phase 5: UI polish and finishing touches

As we reached the end of week 2, we had most of the core product thesis functional. But over time, the cumulative weight of stubbed out UI controls, random diagnostic displays and debug UI, sometimes-questionable improv from the agents, and sometimes questionable-direction from me led to a cluttered mess. It all worked, but the workflow was clunky.

Over the weekend, I finally found some time for a dedicated, old school 1:1 chat with Opus. Over two hours together, we cleaned up the UI mess and tightened up the core flow. This was a great session, and worked well because all the core infrastructure was in place. It was a delight to carve the cruft away and pull forward the best ideas from a functional core.

5. Lessons learned after two weeks

It works (mostly)

More than anything: you can't argue with results. After 2 weeks, I have a fully functional version of my concept, that I have used to make musical ideas that I like. I don't think the codebase is a disaster either. From point audits and explorations of the generated output, it's sensibly organized and architected, and the code I read makes sense. The app proved surprisingly extensible, and in usage it feels rock-solid. It runs well, with no pops or glitches, sounds great, and runs over long sessions without problems. Especially impressive since audio is hard to get right!

It's been surprisingly feasible to hand control over to the orchestrator. For someone used to watching the agents closely and reviewing code regularly, losing control like this has been an adjustment. But the ability to execute across multiple threads in parallel without interruption, and especially to have it run overnight or while I'm away, is so much more powerful that it's been worth it.

Even if orchestration is twice as error-prone, it runs 24 hours a day instead of one. Despite the backtracking and rework, it still isn't close.

It's not the future yet

However, there's a crucial caveat: the errors stack. If a 95% success rate reduces to 90% unsupervised, the difference after 7 iterations becomes 70% vs. 48%. The key to this is constantly correcting back up to 100% at regular checkpoints, or the whole thing can tip over. Running the factory lives and dies on the quality of your safeguards and review.

I'd give the guards on this run a "C". There's plenty of slop, but I don't think it's remarkably worse than your average startup. I think it's easy to romanticize code quality, imagining some kind of bespoke crafting process, but anyone who's built at startups in a marathon of sprints can tell you the bar to clear is not as high as we often pretend.

The proof moving forward will be development speed and product quality. Do we spin for another week on major refactors? Can we support complex features like MIDI and automation? The bar will rise as the code complexity grows.

As for the factory, there have been many bumps on the way. Calling the factory "autonomous" is a bit of a stretch right now.

Minor friction can grind things to a halt

The biggest challenge was simply keeping the factory running. In addition to the usage limits, I ran into many other mechanical problems.

• App bugs with both Claude and ChatGPT (dropped sessions, unreliable event observability, memory issues with longer sessions) caused work to stop on multiple occasions.
• Inconsistent interaction with Github over the MCP connector / plugins, both from bugs and inconsistent agent behavior, led to overwritten work.
• When my Github actions budget ran out, agents couldn't figure out why jobs were failing and ran in circles for hours.
• Beads was key to task orchestration, but was locked in the initial orchestrator agent's sandbox. This broke down when I started handing off between orchestrators and required a rework of the task protocol.
• At first, subagent traces were locked up in the main orchestrator conversation. This also created handoff difficulties.

Observability is essential to debug

I learned that it's useful to record traces + summaries of all conversations. This allows better recovery if something goes wrong, better handoff, and factory "introspection". With traces, you can run regular "retrospectives" on how the factory is running, even running adversarial review of factory protocol changes against real examples of agent behavior.

Affordable compute is my primary constraint

Despite running two pro plans: the Anthropic Max account on 20x and ChatGPT Pro on 5x, the factory could only run for ~48 hours without hitting limits. By far the biggest issue was Fable usage, but ChatGPT metered out quickly on Sol as well. Cheaper agents can orchestrate, but I have a trust gap. When Opus is driving, I feel a slight difference in decision quality.

AI inference isn't the only cost. As changes piled up, so did CI spend. This was a self-own—all of the sample audio content was initially included in the repo, creating large artifacts that were built and tested on every merge.

I was surprised at how effective it was to let the orchestrator know I was concerned about cost. It redesigned the CI suite to be modular, ran scoped testing, and got strategic about when to do full test runs. It restructured the repo and stopped uploading the full sample corpus, instead scoping a small palette to run all the tests. Overall we cut CI time way down and reduced spend to a hobby-manageable level.

Agents regress from orchestration to execution

A detailed and comprehensive factory spec helped a lot with focus over long time horizons. However, the orchestrators had a constant tendency to revert to "individual contributor" work. Like any micromanager, they drifted towards getting in the weeds of subagent work. Sometimes this was effective, unsticking issues and resolving conflicts — but sometimes it led to the orchestrator stepping on subagent toes, overwriting work, and getting stuck in loops.

I suspect this is a "skill issue" on my part and can be at least partially resolved with better factory guidelines. But I also suspect this is an inherent property of today's agents. They are naturally inclined to be helpful and fix problems themselves. If you've ever been stretched between leadership and individual contributor work, you know the perils, and agents face them as well. While I ran into this with every model, Fable and Sol both were better than Opus. I suspect their post-training is better calibrated to orchestration than other models.

1:1 work still matters

It's been an interesting experience to move most of my discussions with the agent "up a layer". I am essentially communicating and collaborating with the engineering manager instead of the IC engineers. This works for most tasks, but for some work I want to have a more active collaboration:

• Design work requires a tight feedback cycle. Especially in a creative tool, there's a feel to the workflow that can only be assessed by using it and—ideally—being able to tweak or adjust things while I'm hands-on with the product. One of the biggest friction points is waiting for agents to come back with UI changes. My dream is a "jam session" where the interface choices can update in real time based on a voice conversation.
• Taxonomy requires direct input. Agents have a tendency to fall back on cryptic naming conventions that make sense to them in an internal technical language that they'll develop as they build a project. Stamping these out and replacing them with human-legible nouns and verbs is a big part of keeping the codebase comprehensible. I see "LLM-y" words creeping in constantly: substrate, ratification, and so on. I do not try to fight these in the agent conversations, as I don't want to interfere with agent's natural chain of thought, but I try to keep them out of the codebase and UI.
• Prioritization requires constant checkins. If you don't watch them closely, agents will, for example, build a cryptographic hash verification system on DSP sample exactness across Mac and Linux, having convinced themselves that it's on the way towards reliable audio engine testing. (Ask me how I know.) In this way, they remind me a bit of any extremely talented engineer: extremely capable, extremely prone to gold plating.

6. The core problem: Legibility and review

Agents drift towards cryptic communication

Especially in the latest models, both OpenAI and Anthropic models develop their own technical lingo, especially over long sessions. This phenomenon fascinates me—real teams do this too! When you're close to the work, there's a tendency to name concepts and use those names in everyday work. It's easy to lose track of your little world of bespoke vocabulary.

This can make it very hard to understand what's happening in the factory. For example, take this response to a question about backlog priorities:

The recommended queue: car D → Phase 2 (trimmed and track-renamed) → the sweeps → the engine lane (dial-02b, dial-pz2, dial-sdoj, dial-el2, dial-6ql, dial-0o8, dial-h4p) → Phase 3's gates — plus the standing note that your micro-R1 is ready whenever you are.

Thanks, Claude. 👍

Understanding changes through text and code is a nightmare

This cryptic language becomes even worse when trying to assess code changes with huge surface area. What's actually changing? Even if you're an expert in the system, auditing changes at code level is infeasible at scale. CTOs can't review every pull request, even in the smallest organizations. Review needs to move up a level, and trust is required that the work matches the plan.

The same applies to product and design: to abuse a famous quote, writing about design is like dancing about architecture. I need to see it to give good feedback; whether it's a UI or a project plan. I think visually, and as I pore over pages and pages of code, agent responses, debug traces — I reflect constantly on the low information bandwidth of text.

It's like trying to lead an organization by reading every Slack message in every channel, hoping that the sum total of one off responses in threads adds up to something more. This type of feedback is important sometimes, but if it's your only way of engaging, it's exhausting.

Familiar challenges

I have a new appreciation for the challenge any executive leader faces. They show up every day to hundreds, thousands of small decisions being made at the team level. Most of them are good, some of them aren't. Can you tell which? Can you even understand what is being decided? Is time being wasted on problems not worth solving? I can't review every message any more than a CEO can sit in every meeting.

When I started leading a team of agents, I found myself reaching for the same tools and channels I'm used to. Design reviews, tech reviews, decision logs, prioritization discussions, strategy discussions, kickoffs. It's clear this layer is missing, and the pain gets more acute with scale.

For these conversations to be effective, I need more than a wall of text: I need artifacts. Slides, diagrams, roadmaps, backlogs. Til now, I've thought of agents making these artifacts primarily as a way for me to accelerate my own work: communicating with stakeholders, setting strategy for the team. But running my agent team over the last two weeks, I'm feeling the contour of a new frontier: artifacts for agents to explain what they're doing to me, and help me guide them at a higher level.

7. Next steps

Cost optimization

It's clear that I could be using cheaper models for much of the factory's operation. I'm surprised at my reluctance to do this. When you're working on something you care about, you want your best engineer on everything. But a deep lesson from the last two weeks (and years of experience playing Factorio!) is that you're only as good as your tightest bottleneck. Throughput is everything, even for quality: you can't fix bugs or get CI green if you're out of Fable credits.

The answer to make this workable is stronger safeguards on review and evals of agent performance against task type, to optimize routing over time. That seems like the next major frontier for the factory. I am particularly interested in experimenting with Luna and open source models for execution. But I can't do this without better quality guards and evals.

Experiment with custom orchestration

Multiple baton passes have taught me that good documentation can help the codebase survive harness and model changes. I'd like to experiment with a custom orchestrator that runs the factory, rather than relying on the features in Claude and the GPT desktop app.

My needs are fairly simple (orchestrator, subagents, adversarial review, watches, cron), so I resisted building at first. It's easy to get lost in building the orchestrator rather than building the product itself. But now that I have experience from a couple weeks of operation, I can see many places where custom orchestration would help a lot.

In particular, building custom eval harnesses to evaluate different agent routing, combined with cost optimizing for cheaper models, could recursively self-improve the factory, not just the codebase.

A visualization layer

I'm interested in whether I can create skills that prompt agents to schedule morning or evening reviews with me:

• walking through their major decisions
• visualizing the state of work, burndown, progress through epics
• conversations about high-level prioritization and roadmap

Here worlds collide with my day job: Miro has all the core primitives for workflow visualization and project coordination. Now that they're available over MCP, I want to experiment with Miro as a "frontend" for the factory. I've just started experimenting with this — more on this soon.

Closing thoughts

I'm paused again, waiting for another usage reset. I like what I made, and it was cool to see an idea from my old notebooks come to life. I'm not sure I'll pursue this idea further, but the background nature of the factory makes the stakes low to run it for another couple weeks and see how it evolves.

In the end, this might be the most enduring lesson from this project: It's useful to have a stack of projects on the back burner, and a machine that can build them with low intervention. But to really make this work—and to make it fun—you need ways for the agents to show their work and explain themselves. Getting agents to "manage up" is still an unsolved problem.

p.s. : If you'd like to try this out yourself, just send this article to ChatGPT or Claude, and say that you want to set up a factory.