Same Model, Different Harness

I set out to test a harness I had built. What I found is that the system you wrap around an AI can matter more than the AI inside it.

Short answer: The model gets the headlines, but the harness — the system you build around the model — quietly decides what the work costs, whether you can trust it, and whether a mistake turns into a one-off or a lesson.

I thought I was comparing two reviewers.

I was actually comparing two harnesses.

I noticed it on a dashboard. One row was Aegis, a review system I had been building. The row right beside it was Codex, running quietly on the exact same pull request, looking at the exact same evidence. Same job. Same input. Two completely different ways of doing the work.

And the moment I really looked at the numbers, the question I thought I was asking rearranged itself into a different one.

Let me back up, because "harness" is doing a lot of work in that sentence.

A quick word on harnesses

If you read The Brain, The Job, The App, you already met this idea — I just called it something friendlier.

The model is the brain. Opus, Sonnet, Codex, whichever one you pick.

The app you run it in — Claude Code, a desktop app, a terminal tool — is the body and the hands. The technical name for that body is an agentic harness. Last time I said we would just call them apps. This post is about what happens when you stop treating the harness like a detail and start treating it like the thing that decides whether the work is any good.

Here is the analogy I keep coming back to.

Imagine you hire someone brilliant. Genuinely talented. Now imagine two versions of their first day.

In one version, you point them at the building and say "go." They wander everywhere, read everything, follow every hallway, and eventually come back with something useful — but it took all day and cost you a fortune in their time.

In the other version, you hand them the playbook. Here is the job. Here are the boundaries. Here is the checklist. Here is how I want you to report back what you found.

Same person. Same talent. Totally different day.

That second version is a harness.

We were already moving fast

Some context, so this story stays honest.

This was not a team sitting around waiting for AI to make us productive. Over the last three months, between me, Edgar and Gabe, we pushed 424 merges into our development branch. That is 4.56 changes a day, or 6.33 every weekday.

(A quick translation if "merge" and "pull request" are new words: a pull request, or PR, is just a proposed batch of changes waiting for a human to look it over and approve it before it becomes part of the real codebase. A merge is what happens when it gets the thumbs up.)

That second row is there on purpose. I am not trying to sell you a hockey-stick chart where AI suddenly made everything explode. We were already fast. The quarter before was actually a touch faster.

What changed was not the pace.

What changed was that review could finally keep up with the pace and I could measure it while it happened.

The uncomfortable question behind Aegis

Around this time I was pulling Aegis out of AgentShelf to live as its own standalone package, so I could improve the harness on its own without dragging the whole product along with it.

That sounded like a tidy engineering chore. But there was a harder question sitting underneath it, and I knew it.

Was the harness actually good?

Not "good" as a neat idea I was fond of. Not good because I built it. I needed to know whether Aegis gave me something other harnesses did not, or whether I just liked it the way you like anything you made yourself.

That question felt familiar, and it took me a second to place why.

When I built Storyweaver, I did not know I was building a harness. I thought I was building a little story-writing tool. Looking back, the parts that actually mattered were not the model at all — they were the loop, the writer personas, the chapter files, the story bible, the context I kept feeding it, and the human judgment at every turn.

The model mattered. But the shape I built around the model mattered more.

Aegis was that same instinct, finally made explicit.

Give the model a job. Give it context. Give it rules. Give it a way to report what it found. Then give the human a way to decide whether the work was actually any good.

Johnny 5

Then Edgar built our first digital employee: a PR reviewer.

We named it Johnny 5. Edgar grew up on Short Circuit, so did I, and the name landed instantly. A helpful little machine with a bit of personality, sitting in the workflow, reading PRs and keeping the team moving. A golden Johnny 5. It was perfect.

The first version was written in Rust. Small, compact, useful. It watched our work, reviewed each PR as it came in, and left comments on what it found. It already did something real.

And that is exactly why I saw an opening.

If I wanted to honestly test Aegis, this was the place. The reviewer already had a real job, real stakes, and enough repetition that the harness around the model would actually show up in the results. So I rebuilt it on Aegis and called it PR Watch.

The rebuild changed what the thing was.

It went from "a bot that reviews PRs" to something I could actually measure: it watches for new work, refreshes its copy of the code, runs the same fixed checks every time, only looks at the evidence that matters, and saves everything so two runs can be lined up side by side on a dashboard.

Version one reviewed PRs.

Version two made PR review something I could measure.

The first reality check

PR 2217 was the run that made the tradeoff real for me.

It was a bug fix. Aegis reviewed it and reached the right call. Codex, running quietly in the background on the same PR, reached the same call — but along the way it surfaced a genuine defect in the changed code that Aegis had not flagged.

Here is the part worth sitting with. Before I tell you the cost, "tokens" is just the meter running on an AI. The more it reads and writes and pokes around, the higher the number and the higher the bill.

I will be honest about how that felt, because it was two feelings at once.

The first was quiet pride. A thing I had built reached the correct decision for sixty-one cents while a far more expensive setup burned through roughly twenty-four times the tokens to land in the same place.

The second was humility. Codex did catch something real that Aegis missed. That is not nothing. That is the whole reason I do not trust Aegis on its own yet.

So this was never a clean win. It was a tradeoff I could finally see.

And here is the thing the harness gave me that I did not have before: that disagreement did not vanish. PR Watch caught it and saved it as a learning item. Until Aegis can reliably catch that kind of issue on its own, the expensive second opinion becomes part of how I make Aegis better.

Codex is genuinely good at exploring. It wanders the codebase, inspects paths, chases possibilities, keeps digging. That is a real strength.

Aegis was doing something different on purpose. It was not roaming the building. It had the playbook — the job, the boundaries, the checklist, the report-back format — and it stayed inside it.

Same kind of brain. Different onboarding. Different result.

The shape kept showing up

PR 2217 was the first time I saw it clearly. It was not the last.

Across the first 25 runs where I could line the two up directly, the same shape appeared over and over.

The useful thing about that table is not that it crowns a winner.

It shows the shape of the tradeoff.

Aegis was cheaper, more contained, and far easier to audit after the fact — when I went back to check its work, the trail made sense. Codex was more expensive, more exploratory, and every so often it turned up something real that was worth folding back into Aegis. The wandering was not wasted. Some of it was genuine signal.

But the cost and the shape of the work were not even close.

Why I am careful about the claim

As more comparison runs piled up, I could put a number on that learning signal too. And this is where I have to be careful, because it would be easy to oversell it.

That 30.2% is a lower bound, and I want to be clear about why. Of the 43 comparison runs, 18 did not have learning artifacts attached. Those rows are unknown — not proof that Codex found nothing. I would rather report the honest floor than a flattering guess.

In the runs where I actually have the receipts, Codex produced a real, validated finding that Aegis missed roughly half the time. Across those, I counted about 16 unique findings.

That is not me arguing Aegis is finished. It is the opposite.

It is the argument for the loop.

That is what PR Watch really gave me. Not a benchmark trophy. A way to see the whole system — including its gaps — clearly enough to improve it on purpose.

What the harness proved

I am adding more harnesses to the comparison now — Claude Code, Antigravity, local models, whatever shows up next. But the main thing is already clear to me.

What you run the model with matters.

The model matters too, obviously. But the harness is what decides the economics of the work: what it costs, how long it takes, whether you can trust it, how it fails, and whether a disagreement becomes a one-off surprise or a permanent improvement.

So — was my harness better?

Not universally. Aegis is more focused, and focus has a price. Codex caught things Aegis did not, and until Aegis catches them on its own, I am using those misses to teach it.

But Aegis had real advantages where it counted. It was cheaper to run, easier to audit, shaped around how we actually work, and built to learn from the exact places it fell short.

That is the part I am most proud of. For something this young, Aegis could stand next to far more mature setups and hold its own — not because it was smarter in every case, but because it was built into the system around it.

Why this matters if you are not building harnesses

Here is the part for everyone who does not spend their week comparing token counts.

The entire AI conversation right now is about the model. Which one is smarter. Which one to pick. New brain, new headline, every few weeks.

But for getting real work done, the model is rarely the thing that decides whether AI is cheap or expensive, trustworthy or a liability, a one-time party trick or something that quietly gets better at your work. That is the harness. The system around the model.

And almost nobody gets to choose theirs.

Most people meet AI as a chat box and a brilliant new hire with no playbook — talented, eager, and just as likely to wander off and burn an afternoon as to nail the task. I know that hire well. I let one offer to delete seventy percent of Storyweaver, and I said go. The talent was never the problem. The missing playbook was.

That is the whole reason we are building AgentShelf. Not to hand you a better brain — you can already get those anywhere. To hand you the harness: the job, the boundaries, the checklist, the memory, the way to report back. The onboarding, so the talent actually shows up for your work, without you having to wreck a Storyweaver of your own first to learn how.

The future everyone is selling is better models.

The future I am betting on is better systems for running them.