What the talk says
Everything in this section is our paraphrase of the talk, except the lines in quote marks, which come from the captions.
Katz describes the harness as "the eyes and the hands for the model". It has a runtime layer, which decides where the agent loop runs and how state is stored, and a context and orchestration layer, which decides what the model sees.
His central idea is a flywheel. The harness makes harder coding tasks reachable, more people attempt them, that work reaches the labs, and the models pick up the skill natively. Then the harness can delete the workaround it needed. His summary: "A fancy harness in this day and age, no longer adds coding intelligence."
He points to two studies. HarnessTax, from UC Berkeley and Arena, ran the same models through different coding harnesses and found that harness choice barely moved success on SWE-bench Lite, while cost moved a lot. A paper from EPFL and Apple found that, with the same model and the same time budget, elaborate harnesses gave no advantage over a minimal coding agent on MLE-bench.
Then he turns. There are, in his words, "some parts of the harness that the model can't absorb". Trust and security is one of them. He says you probably don't want to give the model all your API keys in a file, and that companies want permissions "scoped down to specific agents and specific people".
Two kinds of guardrail
The talk uses "guardrails" for two different things, and the difference matters once you start cleaning up.
The first kind patches model behaviour. Katz describes his team removing guardrails that covered gaps in the model's coding ability. He also describes how they handle the deterministic nudges they still need: each one is tracked centrally as a shim, every firing is logged, and when a shim stops firing, a bot opens a pull request to remove it (paraphrase). For behaviour patches, that is good hygiene. When the model stops needing the nudge, the nudge should go.
The second kind is a permission check: this caller may not run this tool right now. A model getting better at a task does not change whether this caller is allowed to do it. That check should not disappear because a dashboard says it fires less often.
Here is the part that is our argument and not Katz's. He places trust and security inside the harness, and he does not say how to build it. We think that if both kinds of guardrail live in the same code path, written the same way, a cleanup pass can remove the wrong one. That pass might be a person or a bot. So we keep the permission check in a separate decision service.
Where the decision service fits
In I/O Mesh, the harness calls the decision service before a tool runs. The service answers allow or deny on every call, not once per session. The caller is the mapped subject of a verified OIDC bearer token, not a string the agent fills in. The answer comes from the department's policy for that caller and tool.
I/O Mesh does not run the tool or fetch the tool endpoint. Your harness still owns execution, retries and result handling. What changes is where the check lives. It has its own policy and its own record, so deleting harness code does not delete it. The harness does still have to call it, and every tool path has to go through that call. Decide before the tool runs walks through the harness side.
When the service cannot answer
I/O Mesh fails closed. Five cases return deny to the caller:
- The audit write fails.
- The store returns an error.
- The decision times out. The default is 5 seconds.
- Confidence in the decision is uncertain.
- The grant map is full.
One more case is separate. If an allow cannot be delivered to the caller, the grant is dropped, and the audit record for that decision stays an allow. The caller gets no usable answer, so the tool cannot run on it.
Revoke takes effect on the next resolve. On the harness side, treat a timeout, an error or an unparseable response from the decision service as a deny. Fail closed covers each case.
A record you can check later
A shim that stops firing can be deleted, and nobody needs to remember it. A permission decision should leave something behind.
I/O Mesh writes one audit record per decision, allows and denies alike. Over those records it emits heartbeats. A heartbeat is a signed SHA-256 digest of a window of records, linked to the previous heartbeat. A new one is emitted after 100 records or 60 seconds, and only when at least one record is pending. A quiet interval emits none. You verify heartbeats offline with the auditor public key. The signed audit heartbeat explains how to check one.
Check the heartbeat. Read the record.
If the talk has you clearing out old harness code, it is worth sorting your guardrails into two lists first: behaviour patches and permission checks. If you want to compare notes on the second list, write to hello@iome.sh.