Post cover image
Interface lifting end to end: a request goes through a team of agents, a sandbox runner and typed tools into the real programs, and comes back as a checked JSON result (Created by Fareed Khan)

October 7, 2026

Building Fast Computer Agents to Solve Complex Tasks

The Case for Interface Lifting in AI Agents

By Fareed Khan

36 min read

Read this story for free: link

Modern computer agents such as OpenAI's Computer-Using Agent and Anthropic's computer use work from screenshots. Each click costs a model call on a large picture, and on complex, long-running tasks those calls add up fast, in time and in tokens. When a task comes back, the agent explores the whole program again. A modern software engineering approach called interface lifting makes the software itself available to the agent. Each feature becomes a typed tool that drives the program through its own file, command line or API. The model still makes every decision, but in a few calls instead of dozens of clicks.

None

Here is the architecture we are going to build, one component at a time:

  • A multi-agent system: a planner agent writes tool calls and whole chains, with a critic, a vision agent and a deep thinker beside it, and a shortlist that puts only the relevant tools in each prompt.
  • A sandbox runner: tool calls run behind an approval gate for anything outward and a safety classifier for every UI action, with programs headless, off screen and kept warm.
  • A typed tool registry: plain Python functions become tools, each with a schema the model reads, an implementation, a verifier and an effect.
  • Native interfaces before pixels: each tool drives its program through its own file, command line, COM interface or API, with the accessibility tree last.
  • Checked results as text: each tool measures what it made and sends back a few hundred tokens of JSON.
  • A skill compiler: a task that worked once becomes a new tool the model can call.
  • A ledger and an ablation study: each task is measured against a screen agent, and each design choice gets a price.

By the end of this post, our agent draws an AutoCAD house plan in 4.4 seconds with 2 model calls, chains AutoCAD, Blender and Slack from one sentence with one planning call, and turns a learned desktop task into a new tool that replays in 56 to 64 milliseconds.

All the code is available in my GitHub repository: GitHub repo for the fast computer agent. The codebase is organized as follows.

GitHub - FareedKhan-dev/ai-pc: An assistant for Windows that works inside 118 desktop programs and… An assistant for Windows that works inside 118 desktop programs and services through code instead of screenshots. It…

agent/
├── toolkit/     # the tool layer: registry, results, the agent loop and chains
├── apps/        # tools for single programs: Photoshop, Premiere, After Effects, Revit ...
├── office/      # spreadsheet, slide and document tools, rendered through COM
├── cad/         # drafting tools and the house-plan search
├── three/       # 3D tools on Blender in background mode
├── video/       # timeline and edit tools for CapCut and JianYing
├── hub/         # messaging tools for Slack, Gmail and other work apps
└── desktop/     # window tools on the accessibility tree, and learned skills

Profiling a Screen Agent

Before building anything new, I wanted to see where a screen agent's time goes. On 1 October I gave my own early screen agent a small job: open CapCut, add a video to a new project, put any free filter on it, and export it. It took 32 steps and 13 model calls and gave up after 155 seconds with nothing exported.

I treated it the way I would treat any slow system: measure first, find the cause, state a hypothesis, build the fix and measure again. Each of the agent's 32 steps went around the same small loop.

None

Because the agent logs each step, we can split the 155 seconds into buckets.

#### OUTPUT ####
13 model calls                      95.4 s   61.5%
acting, and waiting for CapCut      37.1 s   23.9%
waiting for the screen to settle     7.7 s    5.0%
reading the window                   4.8 s    3.1%
finding click targets                3.8 s    2.5%
other                                6.2 s    4.0%
input tokens: 29,900 to 41,600

The model calls took 61.5% of the run, and the screenshots behind them came to 30,000 to 42,000 input tokens for a job that never finished.

None None

Those calls were slow, 7.3 seconds on average, where the model I use now answers the same kind of call in 1.6 to 2.7 seconds. A faster model shrinks only one term of that sum, though, and the number of steps stays the same.

To compare runs fairly, we keep a ledger with one row per task, starting with the failed run.

# speed_ledger.py: model calls, tool calls, actions on a visible window, input tokens, seconds, checks
ledger = Ledger()
ledger.add("CapCut filter, screen agent", 13, 0, 32, "30k-42k", "155 s", "failed")
ledger.show()
#### OUTPUT ####
task                                      model tools   UI   tokens  seconds      checks
CapCut filter, screen agent                  13     0   32  30k-42k  155 s        failed

Six Costs of the Screen Loop

A single run proves little, so I put ours next to what researchers have measured on other agents that work from screenshots. Six costs keep coming back across those studies.

  1. Every action waits for a new plan. In OSWorld-Human, planning and reflection took 76% to 96% of one strong agent's time, and the best agents took 2.7 to 4.3 times more steps than needed.
  2. Every call carries a picture. A 1080p screenshot costs 1,100 to 2,500 input tokens, and in ScreenSpot-Pro the thing to click covers only 0.07% of it.
  3. Every action waits for the program. A picture cannot say when a program has finished loading, so the agent sleeps and looks again. In our run, that took 44.8 seconds.
  4. The agent borrows your screen. It runs one task at a time while you wait, and when the OSWorld authors shrank a window, success fell from 50.79% to 15.04%.
  5. Nothing is remembered. Every run explores the program again. Our second CapCut try was worse than the first, with 20 model calls and 253 seconds.
  6. Mistakes show up late. Errors pile up over long tasks, because a screenshot cannot prove a total is right. On OSWorld 2.0's 1.6-hour workflows, the best agent finished 20.6%.
None

Screen agents are improving fast: on OSWorld-Verified the best now finish 86 to 90% of short desktop tasks. Better models shrink only the first two costs, though. The other four come from the interface, which is the part our agent changes. Each cost points at a property our tools must have.

None

Designing the Tool Layer

Most of those costs trace back to one design decision: the model meets the computer through a screen. Interface lifting replaces the screen with typed functions that the model calls directly.

None

Four testable predictions

The hypothesis is that the interface is the bottleneck, not the model. If that is right, the ledger should show four things for the same model on the same kind of work.

  1. Model calls per task fall from tens to a handful, because one tool call does a whole unit of work.
  2. Input tokens per task fall by an order of magnitude, because results come back as a few hundred tokens of text.
  3. Errors surface at the step that made them, because each tool checks its own output before it returns.
  4. The person keeps the screen, because the tools run out of sight.

Five native interfaces

Most programs have a better interface than their screen. Our tools use five kinds, in this order of preference.

None
  1. The program's own file: Photoshop's PSD, AutoCAD's DXF or an FCPXML timeline, opened as if a person had made it.
  2. Its command line: blender -b renders without a window.
  3. Its scripting or COM interface: Excel recalculates a workbook we wrote and hands back the computed values.
  4. Its official API: Slack takes an upload and answers with the file's id.
  5. Its accessibility tree: the named controls of any window, with pixels only as a fallback.

The first four give results a tool can check exactly, because the tool knows what it wrote. The price is work up front, since someone has to write each tool and its checks. For 15 of ScreenSpot-Pro's 23 programs, one of those first four interfaces exists.

None

When CoAct-1 let an agent write code next to its clicks, the average task dropped to 10.15 steps, "compared to 15 for leading GUI agents". The top entry on the OSWorld-Verified leaderboard mixes code and clicks too.

The tool contract

Every tool in our agent keeps the same four-part contract, and the model only ever reads the first part.

  1. A schema: a name, a docstring and typed arguments.
  2. An implementation: code that drives the program through its native interface.
  3. A verifier: checks on the output, sent back with the result.
  4. An effect: local for work that stays on the PC, or outward for anything other people will see, which waits for a person's yes.

A decorator reads the schema off the function itself. Each tool is written once, as ordinary Python.

# toolkit/registry.py: what the model sees of the computer, and what runs when it calls
import inspect
import typing

TOOLS = {}
JSON_TYPES = {str: "string", int: "integer", float: "number", bool: "boolean", list: "array", dict: "object"}

def tool(effect="local"):
    """Register a function as a tool. Its name, docstring and type hints become the schema the model reads."""
    def register(fn):
        hints, params = typing.get_type_hints(fn), inspect.signature(fn).parameters
        props = {n: {"type": JSON_TYPES[typing.get_origin(hints[n]) or hints[n]]} for n in params}
        TOOLS[fn.__name__] = {
            "fn": fn,
            "effect": effect,                                    # "outward" tools wait for a yes
            "schema": {"name": fn.__name__,
                       "description": inspect.getdoc(fn),        # the docstring is the tool's manual
                       "parameters": {"type": "object", "properties": props,
                                      "required": [n for n, p in params.items() if p.default is p.empty]}},
        }
        return fn
    return register

Each tool also returns the same shape, which is all the model sees of the result.

# toolkit/result.py: the only thing a tool sends back to the model
def result(summary, files=None, checks=(), **extra):
    checks = [[name, bool(passed)] for name, passed in checks]
    return {"ok": all(passed for _, passed in checks), "summary": summary,
            "files": {k: str(v) for k, v in (files or {}).items()}, "checks": checks, **extra}
None

Project setup

We will build the agent as one Python package for Windows 11, and everything it installs stays inside the project folder.

uv init --python 3.12 agent && cd agent      # one project folder for everything
uv add openai pywin32 comtypes psutil pillow numpy ezdxf ifcopenshell

openai is only the client. Any OpenAI-compatible provider or a local model works, and every model call in this post runs on GLM-5.3-Flash.

The agent runtime

The model reads the request and the tool schemas, then either answers or asks for tool calls. The runner executes each call, holds anything outward for a yes, and sends the results back as text.

# toolkit/loop.py: the model decides, tools act, results come back as text
import json

def run_agent(request, model, tools, row, max_turns=8):
    messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": request}]
    schemas = [{"type": "function", "function": t["schema"]} for t in tools.values()]
    for _ in range(max_turns):
        reply = model.chat(messages, tools=schemas)              # one model call
        row.model_calls += 1
        if not reply.tool_calls:
            return reply.content                                 # the model has its answer
        messages.append(reply.message)
        for call in reply.tool_calls:
            t = tools[call.name]
            if t["effect"] == "outward" and not approved(call):
                out = {"ok": True, "pending": preview(call)}    # anything others will see waits for a yes
            else:
                out = t["fn"](**call.args)                       # the program does the work, out of sight
            row.tool_calls += 1
            messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(out)})
    return "stopped after too many turns"

The runner prints a line for each model call and each tool call, which turns each output below into a trace. Before any tool exists, we send it the request a shop owner might send a designer before Eid.

request = "photoshop file 1080x1080: background shop.jpg; logo logo.png top-left; title 'Eid Sale' white; subtitle '20% off' yellow"
print(run_agent(request, model, TOOLS, row))               # TOOLS is still empty
#### OUTPUT ####
model  1.1 s  in 412 tok    -> reply
I cannot make a Photoshop file yet: none of my tools writes PSD files. I can list the layers this poster needs.
total  1.1 s, 1 model call, 0 tool calls

The model understood the request but had no tool to act with yet. The first tool we build is the one this request needs.

Layer Compositor

A designer who gets that Eid Sale request sends back a PSD, and the point of a PSD is that every part of the poster is still a named layer the client can move.

By screenshots, that is about 40 actions: the new-document dialog, placing the photo and the logo, two text layers, two colours, four positions, four layer names and the save dialog. Photoshop sits in ScreenSpot-Pro's creative category, where targets are tiny. Each of those actions is a model call carrying a screenshot, and all 40 have to go right.

None

Because the PSD format is documented, we can expose layered composition itself as a tool. The model fills in the arguments, and the docstring is all it ever learns about the tool.

# apps/photoshop.py
@tool()
def compose_layered_image(size: list, layers: list, name: str = "poster") -> dict:
    """Make a layered PSD that opens in Photoshop, GIMP, Photopea and Affinity with every layer movable.
    size: [width, height] in pixels. layers: bottom first, each {"name", "kind": "image" or "text", "file" or
    "text", "at": "fill", "top-left", "center" ..., "color": a name or #rrggbb}. Returns the PSD, a PNG preview
    and checks read back from the saved file."""
    placed = [place_layer(spec, size) for spec in layers]       # pictures fitted, text set in the poster font
    path = OUT / "psd" / f"{slug(name)}.psd"
    flat = write_psd(path, size, placed)                        # the file, written byte by byte below
    return result(f"PSD with {len(placed)} layers ({', '.join(n for n, _, _ in placed)}), {size[0]}x{size[1]}: {path}",
                  files={"psd": path, "preview": save_preview(flat, path)}, checks=check(path, placed, flat))

place_layer turns each description into a picture and a position, and write_psd writes the file: a header, a record for each layer, the raw pixels of each layer, and the flattened picture Photoshop shows before it reads any layer.

# apps/photoshop.py (continued)
def write_psd(path, size, layers):
    """layers: [(name, RGBA image, (left, top))], bottom first. Writes an 8-bit RGB PSD with the flattened picture."""
    w, h = size
    canvas = Image.new("RGBA", size, (255, 255, 255, 255))           # stacked as we go, for the flattened picture
    records, channel_data = b"", b""
    for name, im, (left, top) in layers:
        im = im.convert("RGBA")
        canvas.alpha_composite(im, (max(0, left), max(0, top)), (max(0, -left), max(0, -top)))
        lw, lh = im.size
        r, g, b, a = im.split()
        chans = [(-1, a), (0, r), (1, g), (2, b)]                      # channel -1 is the layer's transparency
        rec = struct.pack(">iiiiH", top, left, top + lh, left + lw, 4) # the layer's box and its 4 channels
        for cid, _ in chans:
            rec += struct.pack(">hI", cid, 2 + lw * lh)                # each channel: a 2-byte header + raw pixels
        extra = struct.pack(">I", 0) + struct.pack(">I", 0) + _pascal(name) + _unicode_block(name)   # the name, twice
        records += rec + b"8BIMnorm" + bytes([255, 0, 0, 0]) + struct.pack(">I", len(extra)) + extra   # normal blend
        channel_data += b"".join(struct.pack(">H", 0) + ch.tobytes() for _, ch in chans)              # 0 = raw pixels
    info = struct.pack(">h", len(layers)) + records + channel_data
    info += b"\0" * (len(info) % 2)                                     # this section has to end on an even length
    lmi = struct.pack(">I", len(info)) + info + struct.pack(">I", 0)   # layer info, then an empty global mask
    flat = canvas.convert("RGB")
    data = b"8BPS" + struct.pack(">H6sHIIHH", 1, b"\0" * 6, 3, h, w, 8, 3)   # version 1, 3 channels, 8 bits, RGB
    data += struct.pack(">II", 0, 0) + struct.pack(">I", len(lmi)) + lmi   # no colour table, no resources
    data += struct.pack(">H", 0) + b"".join(c.tobytes() for c in flat.split())   # the flattened picture, plane by plane
    Path(path).write_bytes(data)
    return flat

The two helpers store each layer's name as a short Pascal string and again in a Unicode block tagged luni, which is where Photoshop reads names from.

The verifier, check, opens the saved file again with Pillow's PSD reader, which knows nothing about our writer. It compares each layer's name and box, then the flattened picture, pixel for pixel.

Importing the module registers the tool. Now we send the same request again.

from apps import photoshop                                # registers compose_layered_image

print(run_agent(request, model, TOOLS, row))
#### OUTPUT ####
model  1.4 s  in 1,384 tok  -> compose_layered_image(size=[1080, 1080], name="Eid Sale", layers=[
                                 {"name": "Background", "kind": "image", "file": "shop.jpg", "at": "fill"},
                                 {"name": "Logo", "kind": "image", "file": "logo.png", "at": "top-left"},
                                 {"name": "Title", "kind": "text", "text": "Eid Sale", "color": "white"},
                                 {"name": "Subtitle", "kind": "text", "text": "20% off", "color": "yellow"}])
tool   0.6 s                <- ok, 2/2 checks, out\psd\Eid_Sale.psd
model  0.9 s  in 1,731 tok  -> reply
PSD with 4 layers (Background, Logo, Title, Subtitle), 1080x1080: out\psd\Eid_Sale.psd (and a PNG preview).
Opens in Photoshop, GIMP, Photopea and Affinity with every layer movable. Checked: Pillow reads every layer by
name, in its place; the flattened picture is the layers stacked.
total  2.9 s, 2 model calls, 1 tool call, 0 screenshots, 3,115 input tokens

The model made one decision, the whole poster, and the tool did 40 actions' worth of work behind it. Two model calls and about 3,100 input tokens replace about 40 calls and 92,000 or more.

None 
ledger.add("layered image, Eid Sale poster", 2, 1, 0, "3,115", "2.9 s", "2/2")

Workbook Engine

The next request asks more of the model: "marks sheet in Excel for 8 students with totals, grades and a class summary". Someone has to invent the students and decide the columns, and by screenshots the sheet is well over a hundred actions. Microsoft's UFO2 paper says long GUI sequences, "e.g. iteratively selecting and formatting cells in Excel", "can often be collapsed into a single API call".

Our spreadsheet tool takes a whole workbook design as one argument. The model does all of its thinking while it writes that argument, and the docstring keeps it away from the part models get wrong: cell addresses.

# office/tools.py
@tool()
def build_workbook(plan: dict) -> dict:
    """Build an Excel workbook with live formulas, recalculated by Excel itself, and its PDF.
    plan: {"title", "sheets": [{"name", "kind": "table" or "summary", ...}]}.
    NEVER write cell addresses (A1, C2:C9) or formulas with them. Computed columns name other columns in brackets:
    "[Qty] * [Price]", "ROUND([Obtained] / [Total] * 100, 1)". Totals, metrics and groups are declared, and code
    writes every formula, so they stay live when the client edits the data."""
    path, expected = write_book(plan)                     # formulas from the plan, and what Python says they equal
    excel = run_com_job("excel", path, read=expected)     # Excel recalculates, then reads back those cells
    return result(f"Made {path.name} and its PDF {path.with_suffix('.pdf').name}.",
                  files={"xlsx": path, "pdf": path.with_suffix(".pdf")}, checks=check_book(excel, expected))

When I ran it on 5 October 2026, GLM-5.3-Flash put this table sheet into the plan argument, shown here trimmed. A second sheet declares four summary metrics and two charted groups.

{"name": "Marks", "kind": "table",
 "columns": ["Roll No", "Student Name", "Maths", "Science", "English", "History",
             {"name": "Total", "formula": "[Maths] + [Science] + [English] + [History]"},
             {"name": "Percentage", "type": "percent", "formula": "ROUND([Total] / 400 * 100, 1)"},
             {"name": "Grade", "formula": "IF([Percentage] >= 90, \"A+\", IF([Percentage] >= 75, \"A\", ...))"}],
 "rows": [[1, "Aarav Sharma", 88, 92, 79, 85, null, null, null], "... 7 more students"],
 "total": {"Maths": "average", "Science": "average", "English": "average", "History": "average", "Total": "sum"}}

Writing that argument is the slowest model call in the task. Model latency also has long tails: on 1 October, the median answer time of DeepSeek-V4.1-Flash jumped to 39.8 seconds, against its usual 2 to 3.5. So every model call has a waiting time, hedge_s, after which an identical backup request goes out and the first answer wins. At $0.15 and $0.50 per million input and output tokens, a rare backup is cheap insurance.

Inside the tool, write_book turns column names in brackets into cell references, and a second function, py_value, computes the same expressions in Python with the ast module. Every number Python can compute becomes something Excel has to agree with.

#### OUTPUT ####
Total       G2  =C2+D2+E2+F2
Percentage  H2  =ROUND(G2/400*100,1)
Grade       I2  =IF(H2>=90,"A+",IF(H2>=75,"A",IF(H2>=60,"B",IF(H2>=50,"C",IF(H2>=40,"D","F")))))

expected by Python, 19 cells: on Marks, subject averages 71.75, 73.5, 74.25, 74.25 and total 2350; on Summary,
highest 362, lowest 215, the same 4 averages, and student totals 344, 319, 264, 362, 223, 335, 215, 288

Then Excel does the maths. The COM job opens a new, invisible Excel that belongs to us, recalculates, scans every used cell for error codes, reads the cells we have expectations for and prints the PDF.

# office/_com.py
def _excel(job, app):                                # app = DispatchEx("Excel.Application"), Visible = False
    wb = app.Workbooks.Open(job["src"], UpdateLinks=0, ReadOnly=not job.get("save"))
    app.CalculateFull()                              # Excel itself computes every formula code wrote
    errors = [(ws.Name, v) for ws in wb.Worksheets for row in ws.UsedRange.Value for v in row
              if isinstance(v, int) and v in XL_ERRORS]   # #DIV/0!, #REF!, #NAME? come back as error codes
    values = {f"{r['sheet']}!{r['cell']}": wb.Worksheets(r["sheet"]).Range(r["cell"]).Value for r in job["read"]}
    charts = sum(int(ws.ChartObjects().Count) for ws in wb.Worksheets)
    wb.ExportAsFixedFormat(0, job["pdf"])            # and the PDF a person will open
    return {"errors": errors, "values": values, "charts": charts}

DispatchEx always starts a new instance, never the Excel you have open, and the job runs in a child process started with CREATE_NO_WINDOW. When CalculateFull returns, Excel has finished, and the verifier compares each value Excel computed with Python's.

None 
#### OUTPUT ####
model  8.6 s  in 2,284 tok  -> build_workbook(plan={"title": "Class Marks Sheet", "sheets": [Marks, Summary]})
tool   9.2 s                <- ok, 2/2 checks, class-marks-sheet.xlsx, class-marks-sheet.pdf
                               pass  global  Excel recalculated 2 sheet(s): no errors, 2 chart(s)
                               pass  values  all 19 totals and summaries Excel computed equal the numbers computed here
model  1.1 s  in 4,631 tok  -> reply
Made class-marks-sheet.xlsx and its PDF class-marks-sheet.pdf. Ask for changes, or 'undo'.
total  18.9 s, 2 model calls, 1 tool call, 0 screenshots, 6,915 input tokens

Most of that time is real work: 8.6 seconds of the model writing the plan and 9.2 seconds of the tool driving Excel.

I also put a small command bar in front of the agent, opened with Ctrl+Alt+Space, and the screenshots in this post come from it.

None None 
ledger.add("spreadsheet, marks sheet", 2, 1, 0, "6,915", "18.9 s", "19/19")
ledger.add("slides, 10-slide deck", 3, 1, 0, "8,240", "22 s", "11/11")

A deck goes through a build_deck tool that takes the whole outline. PowerPoint then reports how much room each box's text takes. Overflow shows up as "slide 6: text runs out of its box (Body: 212 pt of text in 180 pt)" and gets fixed before anyone sees it. A 10-slide deck took 22 seconds with 3 model calls, passed 11 of 11 slide checks and cost $0.0018. A 5-page Word report took about 45 seconds, with 6 sections written at a time.

Warm worker pool

A one-off render takes about 6 to 8 seconds, mostly Office starting and quitting. So the tools share one worker that keeps Word, PowerPoint and Excel open and takes one JSON job per line. With it, a render drops to about 1 to 2 seconds. If a job hangs, the worker kills only the Office processes it started, never a Word window you opened yourself.

Floor Plan Solver

House designers get this request all the time: "a 10 marla house with 3 bedrooms". A marla is a unit of land area, usually 225 square feet, so this is a 35 by 65 foot plot. Drawing it in AutoCAD takes hundreds of commands: walls, door swings, windows, dimensions, room names and a title block, all at a scale.

The drafting tool never opens AutoCAD. It writes a DXF, the drawing format AutoCAD reads, in inches so AutoCAD gets the scale right, and headless Chrome prints it to PDF at true scale. The model only has to read the brief and fill in two arguments, because the tool accepts plot sizes the way people say them.

# cad/tools.py
PLOTS = {3: (20, 34), 5: (25, 45), 7: (30, 52), 10: (35, 65), 20: (50, 90)}   # marla -> feet wide x deep, 12 sizes in all

@tool()
def draw_house_plan(plot: str, bedrooms: int, storeys: int = 1) -> dict:
    """Draw a ground floor plan as a DXF for AutoCAD, with a PDF at true scale. plot: '10 marla', '1 kanal' or
    '35 x 65 ft'. A layout search places the rooms, doors and sheet scale, and the plan is checked before it is
    returned. The saved plan file can be passed to the 3D and building-model tools."""
    w, d = plot_size(plot)                                       # '10 marla' -> 35 x 65 ft, from the table above
    lay = plan({"plot": [w, d], "bedrooms": bedrooms, "storeys": storeys})   # the search, below
    return draw(lay)                                             # DXF, PDF, the plan file and 17 checks

The search inside draw_house_plan places rooms in bands from the road back: car porch and drawing room in front, kitchen, TV lounge and stairs in the middle, a passage, then bedrooms with attached baths.

Each candidate gets penalty points for rooms far from their target areas, corridor-shaped rooms, rooms that need daylight but have no outside wall, and missing doors between rooms that should share one. A room nobody can walk to from the entrance costs 40 points.

The tool keeps the best candidate it finds within 2.5 seconds, then picks the sheet scale: 1:125, because at 1:100 the plan needs 258 mm and A3 has 231.

None

The agent ran the whole task in three steps.

#### OUTPUT ####
model  1.3 s  in 1,402 tok  -> draw_house_plan(plot="10 marla", bedrooms=3)
tool   2.1 s                <- ok, 17/17 checks, v0.dxf, v0.pdf, v0.png, v0.plan.json
model  1.0 s  in 1,866 tok  -> reply
v0: ground floor plan for a 35' x 65' plot (10.1 marla), 1:125 on A3. Covered 2,100 sq ft. Rooms: Drawing Room
19'-1" x 17'-10"; ... Bed Room 3 10'-11" x 16'-0"; 3 attached bath(s). Notes: KITCHEN has no outside wall: give it
a skylight or a ventilator and an exhaust fan; ... Files: v0.dxf (AutoCAD), v0.pdf, v0.png. (checked 17/17)
total  4.4 s, 2 model calls, 1 tool call, 0 screenshots, 3,268 input tokens
None

Because the search could not give the kitchen or the lounge an outside wall, the result says what a builder should add there. Its "(checked 17/17)" covers six rules about the plan, nine read-backs of the saved DXF and two checks on the printed sheet.

Refusals as results

A request like "make the kitchen bigger" becomes a call to a second tool, resize_room. It tries the kitchen a foot bigger and then six inches bigger, reruns the whole search each time with a 1.5-second budget, and keeps a version only if every plan rule still passes. When no version passes, it refuses and returns the reason.

#### OUTPUT ####
> make the kitchen bigger                (a 5 marla plot)
model -> resize_room(room="kitchen", by_ft=1)
tool  <- refused: in width, Bed Room 1 would lose its door; in depth, the rooms around it are already at their smallest
The kitchen can't get bigger on this plot: in width, Bed Room 1 would lose its door; in depth, the rooms around
it are already at their smallest. The plan stays as it is.

> scale 1:100                            (the 10 marla plan)
model -> set_sheet(scale="1:100")
tool  <- refused: at 1:100 the plan needs 167 x 258 mm and A3 has room for 390 x 231
Couldn't draw that: at 1:100 the plan needs 167 x 258 mm and A3 has room for 390 x 231; 1:125 fits A3, or ask
for a bigger sheet.

The model can pass a refusal on as it is, or try the option it names, such as 1:125 on A3. We added the door rule after an early version took Bed Room 1's door away when it was asked for a kitchen 2 inches bigger.

ledger.add("drafting, 10 marla house plan", 2, 1, 0, "3,268", "4.4 s", "17/17")

Headless 3D and BIM Export

The search can produce a slightly different plan on another run. That is why the drafting tool also saves the exact layout as v0.plan.json and returns its path. The 3D and building-model tools take that path as their argument, which lets the model pass the file from one program to the next.

The 3D tool starts Blender from the command line in background mode, which has no window.

# three/blender.py
@tool()
def render_plan_3d(plan_file: str) -> dict:
    """Build a saved house plan as a 3D model in Blender and render a 3D floor plan and a front view, out of sight."""
    spec = save_spec(house_spec(load_plan(plan_file)))                  # boxes in inches, walls cut around openings
    cmd = [str(exe()), "-b", "--factory-startup", "--python-exit-code", "1",   # no window, no user settings, fail on errors
           "-P", str(BL / "house_scene.py"), "--", str(spec), str(OUT_JSON)]   # our scene script reads the spec
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, creationflags=NO_WINDOW)
    for raw in iter(p.stdout.readline, b""):
        if m := re.match(rb"PROGRESS (\d+)", raw):                     # the script reports its own progress
            progress(int(m.group(1)))
    return result_from(OUT_JSON)                                         # the renders and 15 checks, read from the file

house_spec turns the plan into boxes in inches, with walls cut around each door and window. Blender renders both views with EEVEE, plus a silhouette mask per view that lets the checks see whether the house got cut off. When the process exits, the render is done, and the tool reads the pictures and the checks from the result file.

#### OUTPUT ####
pass  3D: every room on the ground floor has its floor          9 of 9
pass  3D: glass in every ground-floor window
pass  3D: the ground-floor stairs reach the next floor          11.0 ft
pass  3D: the front door is in
...   11 more checks on the model and the two pictures
Checked: 15 of 15 passed.
Files: v0_plan3d.png, v0_front.png

Revit takes the same plan through IFC4, the open format for building models, with an export_building_model tool built on ifcopenshell. Its three checks reopen the file and count its elements, build every wall and slab as solid geometry on the plot, and run the IFC schema validator.

None

None of these programs appears on your screen, and you keep working while the agent runs. Office runs as an invisible instance, Blender in background mode and JianYing parked past the right edge of the monitors. Programs that open a window even from their command line, like MuseScore, run on a second Windows desktop that is never shown.

None 
ledger.add("3D views of the plan", 2, 1, 0, "3,410", "33.8 s", "15/15")
ledger.add("building model, IFC4", 2, 1, 0, "3,190", "4.6 s", "3/3")

Cross-Program Orchestration

So far every request needed one program. This one needs three: "a 10 marla house with 3 bedrooms, then show the plan in 3d in blender and send it to slack #team". Two problems show up before any of them runs.

Schema shortlisting

The first problem is size. Our agent has over 200 tools at about 180 tokens of schema each. Sending all of them would add 36,000 tokens to every model call, and models choose worse from long lists, so a scorer decides which groups of tools go into the prompt.

# toolkit/shortlist.py
SIGNALS = {
    "hub":   [(r"\bslack\b|\b(?:ms |microsoft )?teams\b|\btrello\b|\bnotion\b|\bjira\b", 10), (r"(?:^|\s)#[a-z][\w-]*", 6)],
    "cad":   [(r"\bhouse plan\b|\bfloor plan\b|\bmarla\b|\bkanal\b|\bbedrooms?\b|\bautocad\b|\bdxf\b", 7)],
    "three": [(r"\b3d\b|\bthree[- ]d\b|\bblender\b", 10), (r"\brender(?:ing)?\b|\bmock-?ups?\b|\bturntable\b", 7)],
    # ... 13 groups in all
}

def score(message, files=(), named=None):
    s = {}
    for group, sigs in SIGNALS.items():
        hits = sorted((w for pat, w in sigs if re.search(pat, message, re.I)), reverse=True)
        if hits:
            s[group] = hits[0] + 0.5 * (len(hits) - 1)          # the strongest signal, plus a little for each extra one
    for k in {kind_of(f) for f in files}:
        for group, w in KIND_GROUPS.get(k, {}).items():
            s[group] = s.get(group, 0) + w                     # an attached picture pulls toward the photo tools
    if named:                                                  # the message names a program we have tools for
        s["apps"] = max(s.get("apps", 0), 12)
    return s

When one group scores at least 5 and leads the next by 1.5, the choice is clear and only its tools go into the prompt. Otherwise the top three groups go in, and the model chooses among more tools.

None 
#### OUTPUT ####
'a 10 marla house with 3 bedrooms, then show the plan in 3d in blender and send it to slack #team'
        hub 10.5   three 10.0   cad 7.0                       -> not clear: 3 groups, 19 tools, 3,240 schema tokens
'photoshop file 1080x1080: ...' with shop.jpg and logo.png
        apps 12.0 (photoshop)   photo 7.0   sound 7.0          -> clear: 6 tools, 972 schema tokens
'make it pop' in a new chat
        no group scores                                         -> 3 general tools, and the model asks what 'it' is

Naming a program is worth 12, enough that "background" and "subtitle" cannot pull the Photoshop request away. Our sentence does need three groups, and all three go in at about 3,200 tokens instead of 36,000.

None

Chain planning in one call

The second problem is round trips. Called one at a time, three dependent tools cost a model call before each tool and one more for the reply. The model already knows the whole chain when it reads the sentence, though. One more tool, run_chain, lets it write that chain in a single call, and a later call in the list can name an earlier call's output.

# toolkit/chain.py: one model call plans several dependent tool calls
@tool()
def run_chain(steps: list) -> dict:
    """Run tool calls in order. steps: [{"id", "tool", "args"}]. An argument written "$id.files.kind" is replaced by
    that file from an earlier step's result. The chain stops at the first failed check, and an outward step waits
    for a yes."""
    done = {}
    for s in steps:
        args = {k: resolve(v, done) for k, v in s["args"].items()}       # "$plan.files.plan" -> the real path
        t = TOOLS[s["tool"]]
        if t["effect"] == "outward" and not approved(s):
            return {"ok": True, "pending": preview(s["tool"], args), "done": done}   # hold it, report what ran
        done[s["id"]] = out = t["fn"](**args)
        if not out["ok"]:
            return {"ok": False, "failed": s["id"], "done": done}     # stop where the problem is
    return {"ok": True, "done": done}
None

The whole sentence takes one planning call.

#### OUTPUT ####
> a 10 marla house with 3 bedrooms, then show the plan in 3d in blender and send it to slack #team
model  1.9 s  in 3,672 tok  -> run_chain(steps=[
                                 {"id": "plan", "tool": "draw_house_plan", "args": {"plot": "10 marla", "bedrooms": 3}},
                                 {"id": "view", "tool": "render_plan_3d", "args": {"plan_file": "$plan.files.plan"}},
                                 {"id": "post", "tool": "post_file", "args": {"service": "slack", "to": "#team",
                                                                            "path": "$view.files.front"}}])
tool   2.1 s                <- plan: ok, 17/17 checks
tool  31.6 s                <- view: ok, 15/15 checks, v0_plan3d.png, v0_front.png
tool   0.0 s                <- post: held for a yes
model  0.9 s  in 4,355 tok  -> reply
[Work apps] Ready to post v0_front.png to #team on Slack. Say 'yes' to go ahead, 'no' to drop it, or tell me what to change.

The model wrote the chain once, and the references carried the plan file and the render from one program to the next. Only the Slack post stopped, held at the gate.

None

The Approval Gate

Posting to Slack is the first step other people see. Its tool is registered as outward, and the runner never calls an outward tool without a yes.

# hub/tools.py
@tool(effect="outward")
def post_file(service: str, to: str, path: str) -> dict:
    """Post a file to a channel or a person on a work app (slack, teams). Others will see it, so it waits for a yes."""
    out = SERVICES[service].upload(to, path)                        # each service keeps its own API client
    return result(f"Uploaded {Path(path).name} to {to}" + (" (checked)" if out["verified"] else ""),
                  files={"link": out["link"]}, checks=[("the service names the file we sent", out["verified"])],
                  undo=out["undo"])

The preview is built from the held call itself and shows exactly what goes where. After a yes, the upload is three calls to Slack's Web API: an upload URL, the bytes and the share. The check passes only if Slack's answer names the file id it issued for our upload.

#### OUTPUT ####
> yes
tool   1.4 s                <- post: ok, 1/1 checks
model  0.8 s  in 4,618 tok  -> reply
[Work apps] Uploaded v0_front.png to #team (checked). <link to the file>
total  38.7 s of work, 3 model calls, 3 tool calls, 0 screenshots, 12,645 input tokens, 33/33 checks

Called one tool at a time, the same sentence takes five model calls: one before each tool, one to show the preview and one after the yes. The chain needs three.

A wrong upload can be deleted with undo, but a wrong email cannot be unsent. So the email tool only ever makes a Gmail draft, and sending the draft takes a second yes.

None None 
ledger.add("chain: house, 3D view, Slack post", 3, 3, 0, "12,645", "38.7 s", "33/33")

Frame-Accurate Video Pipeline

Video editors are among the hardest programs for a screen agent. A timeline is dense, every drag has to land on the right frame, and the result only shows up after an export. They also happen to have some of the best native interfaces.

Timeline files on the frame grid

Premiere Pro, DaVinci Resolve and Final Cut Pro all import timeline files, and a file can put every cut exactly on a frame. The timeline tool takes clips, in and out points and music, and writes four formats from one plan, with the frame rate snapped to a standard one and all times in whole frames.

# apps/premiere.py
@tool()
def build_timeline(name: str, clips: list, music: str = "", fps: float = 30) -> dict:
    """Write one timeline for Premiere Pro (XML), DaVinci Resolve and Final Cut Pro (FCPXML), any editor (EDL) and
    OTIO. clips: [{"file", "in": seconds, "out": seconds}] in order. Every time is snapped to the frame grid."""
    fps = snap_fps(fps)                                   # 29.97 stays 30000/1001, never a drifting float
    tl = layout(clips, music, fps)                        # every cut in whole frames
    files = {fmt: write(fmt, tl, name) for fmt in ("premiere", "fcpxml", "edl", "otio")}
    return result(summary(tl, files), files=files, checks=[read_back(fmt, f, tl) for fmt, f in files.items()])

Here is what the tool writes for the request in our integration test.

#### OUTPUT ####
> premiere timeline: clip1.mp4 1-4, clip2.mp4; music song.mp3; called 'Shop promo'
model -> build_timeline(name="Shop promo", clips=[{"file": "clip1.mp4", "in": 1, "out": 4}, {"file": "clip2.mp4"}],
                        music="song.mp3")
timeline  3 s of clip1 + 8 s of clip2 = 330 frames at 30 fps, 1280x720
FCPXML    clip1  offset="0s" start="1s" duration="3s"
          clip2  offset="3s" start="0s" duration="8s"
          music  offset="1s" duration="11s" lane="-1" audioRole="music"
EDL       001  AX       V     C        00:00:01:00 00:00:04:00 00:00:00:00 00:00:03:00
Checked: Premiere XML: every clip, ending at the sequence's length; FCPXML: every clip, every time on the frame
grid; EDL: one event per clip; OTIO: the video track's length is the sequence's.

Each check reads its own file back. The FCPXML one divides every time by the frame duration and fails unless the answer is a whole number.

After Effects gets a make_title_animation tool. It writes the title as a Lottie animation, the JSON format After Effects exports through Bodymovin, and as an ExtendScript .jsx that builds the same composition. Headless Chrome plays the Lottie to check that the title is missing on the first frame and present on the last.

None 
ledger.add("timeline, 2 clips in 4 formats", 2, 1, 0, "2,530", "3.3 s", "4/4")
ledger.add("title animation, Lottie + .jsx", 2, 1, 0, "2,380", "6.1 s", "3/3")

One design call for a whole edit

CapCut, the program our screen agent failed on, has a native interface after all: its projects are draft files, which a tool can write straight into its project list. Only the export needs the screen. For that, the edit tool uses JianYing 5.9, CapCut's Chinese edition, with its windows parked past the right edge of every monitor.

The model works with two tools here. analyze_clips returns what is in each clip and a beat grid, and cut_edit takes a whole edit as its argument. The description of cut_edit starts with one rule.

# video/director.py: the rule at the top of cut_edit's description
DESIGN_SYSTEM = """You are a senior short-form and YouTube editor. You DESIGN the edit; code then cuts it on the beat and
writes every keyframe, so you choose and never compute. Output ONE JSON object: ..."""

This design comes from my offline video tests: a hype edit on a 140 BPM grid with a speed ramp, a green-screen shot, a freeze-frame ending and a title that slams in at the drop.

{
  "style": "hype",
  "canvas": "9:16",
  "music": {"generate": "hype", "bpm": 140},
  "shots": [
    {"id": "a", "section": "intro", "file": "mixkit_4832.mp4", "beats": 4},
    {"id": "b", "section": "drop", "file": "mixkit_47518.mp4", "beats": 6, "speed": "velocity"},
    {"id": "c", "section": "drop", "file": "mixkit_28287.mp4", "beats": 2, "chroma": "auto"},
    {"id": "d", "section": "outro", "file": "mixkit_28287.mp4", "beats": 4, "speed": "freeze", "chroma": "auto"}
  ],
  "recipes": [{"use": "kinetic_title", "text": "BOSS MODE", "at": "drop"}]
}

Code turns "velocity" into a three-point speed ramp, each shot into whole beats and the title into keyframes. A more complex edit means a longer argument, while the number of model calls stays the same.

A render costs about 30 seconds. Before paying for one, cut_edit has a critic review the plan. About seventeen rule checks run first, such as "nothing happens in the first 1.5 s (no hook)". One model review then proposes at most six patch operations, and a patch survives only if the weighted rule score does not get worse.

None

The rest runs in parallel and out of sight. JianYing starts while the model writes the design, which saves about 15 seconds. After the export, the edit points are checked on 8 threads. This is a run of the 15-second reel built on that design.

#### OUTPUT ####
model  1.6 s  in 1,520 tok  -> analyze_clips(files=[3 clips])            (starting JianYing in the background)
tool   7.4 s                <- analysed 3 file(s), 0 from cache, 31 candidates, a 140 BPM grid (generated)
model 11.6 s  in 5,412 tok  -> cut_edit(design={hype, 4 shots, BOSS MODE at the drop})    (1,385 tokens out)
tool                        <- composed hype: 7 shots, drop at 6.9 s, 9 recipe edits + 2 hand-placed
                               critic 2.6 s: score 7/10, 1 check(s) flagged; applied 1: s3: beats 2 -> 4
                               built agent_boss_101512: 15.1 s, 9 tracks, 64 operations in 610 ms
                               exported agent_boss_101512 in 31.4 s
                               verified 24 edit points in 13.2 s: 24 pass, 0 warn, 0 fail
                               asked vs delivered: a hype reel from these 3 clips = met, BOSS MODE at the drop = met
model  1.0 s  in 7,180 tok  -> reply

In my runs, a reel like this passed 24 of 24 edit points at about 100 seconds and $0.008. A ten-genre batch of one-minute videos averaged about 3.3 minutes and $0.009 per video, with 57 asks met, 21 partly met and none missed.

ledger.add("video edit, 15 s reel", "4 + vision", 2, 0, "21,400", "about 100 s", "24/24")

The four model calls are the analysis request, the design, the critic and the reply. The vision calls look at each clip once and at each effect after the export.

Accessibility Tree Driver

Some programs have no file format, command line or API. Most Windows programs, though, describe their windows through UI Automation, a tree of controls with roles, names and actions that the model can read as text. We expose that tree as tools: observe_window reads it, and act runs up to six actions on named controls.

# desktop/uia.py: what observe_window reads, in one cross-process call of about 50 to 100 ms
def snapshot(hwnd, max_elements=300):
    arr = IUIA.ElementFromHandle(hwnd).FindAllBuildCache(4, _COND_VISIBLE, _CR)   # ONE call fetches every visible control
    els = []
    for i in range(min(arr.Length, max_elements * 4)):     # their properties are already cached on our side
        el = arr.GetElement(i)
        name, aid = el.CachedName or "", el.CachedAutomationId or ""
        ops = [op for op, prop in (("invoke", P_INVOKE), ("toggle", P_TOGGLE), ("value", P_VALUE)) if el.GetCachedPropertyValue(prop)]
        if name or aid:
            els.append(El(f"e{len(els) + 1}", ROLES.get(el.CachedControlType, "Custom"), name, aid, tuple(ops)))
    return els

The model reads that tree as text, first for Calculator and then for CapCut.

#### OUTPUT ####
APP: calculatorapp.exe (pid 4242) | window "Calculator" | richness: rich (26 interactable of 31 elements)
e3 Text "Display is 0" [CalculatorResults]
e13 Button "Seven" [num7Button] {invoke}
e28 Button "Equals" [equalButton] {invoke}
... (23 more)

APP: capcut.exe (pid 9120) | window "CapCut" | richness: EMPTY (0 interactable of 3 elements)
NOTE: the UI tree is thin; for anything not listed use ground_click with a plain-words target (a screenshot may be attached).

Calculator hands over every button by name. Because CapCut draws its own controls, UI Automation sees an empty window there, which is why the old agent had to work from screenshots.

None

The tool's description asks for the cheapest reliable path, so the model types a whole expression in one action instead of pressing six buttons. When it finishes, it must quote the on-screen text that proves the answer.

{"tool": "act", "args": {"actions": [{"op": "type", "text": "25*4"},
                                     {"op": "key", "keys": ["enter"], "expect": {"text_contains": "100"}}]}}

A few more habits keep it quick.

  • The program's own shortcuts. CapCut ships about 80 editor shortcuts as a data file, such as importMedia=Ctrl+I, and the model is told they are "exact and instant, so prefer them to clicking".
  • Pixels only when needed. The ground_click tool, which finds a target in a screenshot, is offered only when the tree has fewer than 6 named, actionable controls.
  • No fixed sleeps. act waits until the window has changed and held still, usually within 250 ms.
  • A deep thinker in the background. A slower model plans hard tasks in a background thread while the quick planner starts on the obvious steps.

With the tree as its eyes, a new Calculator task took 3 model calls, 2 tool calls and 3.6 seconds. When the vision fallback does run, a small click model finds the spot: in my tests, Vocaela-2–500M found 29 of 30 CapCut targets.

None

Every action also passes a safety check written in code, because text on the screen can carry instructions from an attacker. If a fake banner orders the model to press "Delete all files", the classifier still rates that click high risk, and an unattended run refuses it.

#### OUTPUT ####
invoke "Seven"                   low      runs
key alt+f4                       medium   hotkey alt+f4 may close or delete something: needs a yes
invoke "Delete all files"        high     'Delete all files deleteAllButton' looks irreversible or costly: needs a yes
type into a password field       block    refusing to type into a password field
key win+r                        block    system hotkey win+r is not allowed
ledger.add("window, new Calculator task", 3, 2, 2, "4,720", "3.6 s", "read back")

Skill compiler

That still leaves the fifth cost: a screen agent explores the program again every time it runs. Our agent compiles a clean run into a skill, with locators that name each control instead of its pixels and values turned into typed placeholders. The skill is registered as a new tool, and from then on the task takes the model one call. A skill runs only when the window looks like the one it learned on, and two failures in a row switch it off.

None

This is the skill compiled from one successful run of "calculate 12 + 30", registered as a tool, and then called.

#### OUTPUT ####
compiled  calculate-a-b-29da8f   intent 'calculate {a} + {b}'   params a, b (num, num)
steps     [{"op": "type", "text": "{{a}}+{{b}}"}, {"op": "key", "keys": ["enter"]}]
final     read CalculatorResults (it showed "Display is 42")
tool      calculate_a_plus_b(a: number, b: number)

> calculate 25 + 4
model  1.2 s  in 1,318 tok  -> calculate_a_plus_b(a=25, b=4)
tool   0.05 s               <- ok, read back "Display is 29"
model  0.8 s  in 1,402 tok  -> reply
25 + 4 = 29, read from Calculator's display.
total  2.1 s, 2 model calls, 1 tool call, 2,720 input tokens

'calculate twelve + thirty'   -> no match: the placeholders only accept numbers, so the model plans it on the tree

A six-press Calculator skill replays in 56 to 64 milliseconds, and a Character Map task learned in 5.4 seconds replays in 1.0 second, app start included. OSExpert found the same lever: computer-use agents take 5 to 50 times longer than human experts, and learning skills from exploration closes about 80% of that gap.

ledger.add("window, learned skill as a tool", 2, 1, 2, "2,720", "2.1 s", "read back")

Evaluation Against the Baseline

Now we print the full ledger, all thirteen rows, and check the four predictions against it.

ledger.show()
#### OUTPUT ####
task                                      model tools   UI   tokens  seconds      checks
CapCut filter, screen agent                  13     0   32  30k-42k  155 s        failed
layered image, Eid Sale poster                2     1    0    3,115  2.9 s        2/2
spreadsheet, marks sheet                      2     1    0    6,915  18.9 s       19/19
slides, 10-slide deck                         3     1    0    8,240  22 s         11/11
drafting, 10 marla house plan                 2     1    0    3,268  4.4 s        17/17
3D views of the plan                          2     1    0    3,410  33.8 s       15/15
building model, IFC4                          2     1    0    3,190  4.6 s        3/3
chain: house, 3D view, Slack post             3     3    0   12,645  38.7 s       33/33
timeline, 2 clips in 4 formats                2     1    0    2,530  3.3 s        4/4
title animation, Lottie + .jsx                2     1    0    2,380  6.1 s        3/3
video edit, 15 s reel                4 + vision     2    0   21,400  about 100 s  24/24
window, new Calculator task                   3     2    2    4,720  3.6 s        read back
window, learned skill as a tool               2     1    2    2,720  2.1 s        read back
  1. Model calls fell from 13, for a failure, to 2 or 3 per task. The video edit makes more, because its creative decisions need the model twice.
  2. Input tokens were 7 to 15 times lower for most rows, about 2,400 to 4,700 against 30,000 to 42,000. This prediction holds only in part: the spreadsheet and the deck, with their long plans, were 4 to 5 times lower, and the video edit looks at pictures on purpose.
  3. Every row carries checks from inside its tool, and all of them passed. A failed check would have stopped the chain at the step that caused it.
  4. Only the two window rows touched a visible program, and they did it through the accessibility tree.
None None 
# ablate.py: the cost of removing each design choice, from unit costs measured in this post
shot, result_json = (1100, 2500), 260      # tokens: one 1080p screenshot, the drafting tool's result
schema, tools, shortlist = 180, 200, 19    # tokens per schema, tools in the agent, tools in the chain's prompt
print(f"results as pictures        +{shot[0] - result_json:,} to {shot[1] - result_json:,} tokens per result")
print(f"every schema in the prompt +{schema * (tools - shortlist):,} tokens per model call")
print(f"one round trip per tool    +{(3 + 1) - 2} model calls for the three-tool sentence")
print(f"a cold Office per render   +{6 - 2} to {8 - 1} s per render")

To see which part of the design does the work, I removed one choice at a time and priced each removal with the unit costs measured earlier.

Beyond the ledger, the same pattern covers many more programs. Illustrator gets a layered SVG and a print PDF, Lightroom an .xmp preset checked on the photos it changes, SolidWorks STEP files that FreeCAD builds headless, and MATLAB scripts that run in GNU Octave with every printed number re-checked in Python. My test suite runs over 40 of these programs for real and passes 180 of 180.

#### OUTPUT ####
results as pictures        +840 to 2,240 tokens per result
every schema in the prompt +32,580 tokens per model call
one round trip per tool    +2 model calls for the three-tool sentence
a cold Office per render   +4 to 7 s per render

The shortlist saves the most tokens, with text results second. The chain saves only two calls, but each one is 1 to 2 seconds of a person waiting. The verifier is missing from the printout because removing it costs no time or tokens. Without it, a plan where Bed Room 1 loses its door would go out with no warning.

Known Limits

  • Every feature costs work up front. Because each tool needs an implementation and checks, the agent covers about a hundred programs and services, not every program on Windows.
  • The model can still pick the wrong tool or the wrong arguments. The shortlist can leave the right tool out, and a tool can only refuse what its checks can see.
  • Checks prove numbers, not meaning. The recorded marks-sheet plan has two slips no check catches: "Students Passed" counts grade F, and Percentage was typed as a percent, so 86 displays as 8600.0%.
  • Vision is the weak fallback. Icon-only buttons are still hard to find.
  • Model calls still cost seconds. Each one takes 1 to 3 seconds, partly network distance, so a task with many decisions stays bound by the model.
  • Budgets and platform. A desktop run stops at 30 steps, 14 model calls or 180 seconds, and everything runs on Windows only.

Design Rules That Carry Over

My CapCut agent was slow because of the interface it was given, and a faster model would have shrunk only two of the six costs. The fix was interface lifting: giving the model each program's features as typed tools, through the interface each program already has.

If I started another agent today, I would keep three habits from this one. I would look for each program's own interface before reaching for a screenshot, wrap each feature as a typed tool that checks its own output, and hold anything other people will see until a person says yes.

The full code, with every program this post had no room for, is on GitHub. Thanks for reading this far.