Most of what separates a reliable agent system from an unreliable one has nothing to do with which model is underneath. It is a handful of decisions about where state lives, what a credential can reach, who is allowed to declare the work done, and what the system measures. Each of those has been the subject of published work this year, and the results point the same way.
The architecture everyone converged on
Compare any two agent frameworks released in the last year and the differences are mostly vocabulary. All of them separate planning from execution, keep tools in a registry with typed schemas, persist state outside the loop, and wrap the whole thing in something that can stop it. That convergence is good for portability and bad for anyone betting on framework lock-in.
What it means practically is that the choice of framework is close to reversible and therefore not worth agonising over. The part that is not interchangeable is the supervisor — how a failure is detected and what happens next — and that is where the engineering effort belongs.
Why a longer timeout is the wrong fix
The usual response to an agent that fails at minute forty is to allow it sixty. That works until the failure rate compounds, and it does compound, because a run lasting an hour touches more things that can break. Checkpointing after each externally visible action costs little and converts a failed run into a resumed one; teams that adopted it report their reliability numbers improving with no change to the model at all.
The phrase doing the work is externally visible. A checkpoint after every internal step is overhead; a checkpoint after every action the outside world can see is what makes a resumption safe, because it is the boundary at which repeating a step stops being free.
Memory is a dose, not a switch
IBM Research ran eight models across AppWorld — 585 multi-step tasks over nine simulated applications — extracting behavioural guidelines from past trajectories and injecting them at inference time, with no weight updates. The results do not converge, and that is the finding.
| Model | Before | After | Change |
|---|---|---|---|
| gpt-oss-120b | 39.9% | 56.0% | +16.1 points |
| DeepSeek-V3.2 | 79.8% | 89.3% | +9.5 |
| Claude Opus 4.6 | 90.5% | 94.6% | +4.1 |
| GPT-5.5 | 92.3% | 95.2% | +2.9 |
| GLM-5 | 87.5% | 87.5% | none |
The token accounting explains why the distinction matters commercially. On gpt-oss-120b, curated retrieval added 5% — 110K tokens to 116K — while the full guideline set added 51%, reaching 166K. On DeepSeek-V3.2 the full set pushed 148K to 263K, a 78% increase. A configuration buying 16 points on one model buys nothing on another while costing three-quarters more.
- Strong models with headroom do best on the full guideline set.
- Weaker models gain most from a tight high-confidence core plus a few task-relevant guidelines retrieved per task.
- Saturated models gain nothing, and the memory is pure overhead.
The security boundary is the credential
Prompt injection gets the attention; over-scoped credentials do the damage. Almost every published agent incident this year followed the same path: the model was persuaded to call a tool it was permitted to call, using a credential broader than the task required. Hardening the prompt is worth doing and will never be sufficient, because the attacker is writing the input.
Scoping the credential to the task and expiring it with the run converts a class of incidents into a logged failure. This is the same argument that runs through our guides on sandboxing and on securing an agent that spends: a limit the model can reason about is a limit the model can be talked past, so the limit belongs outside the model.
Why an agent should not sign off its own work
In most systems the model that does the work is also the one that declares it finished. Measured on 509 source-grounded task directions extracted from agent-visible material, only 79.6% to 86.4% were actually satisfied across seven models — while completion-claim rates exceeded the official evaluator's pass rate by 28.7 to 37.9 percentage points. The agents were not lying so much as answering a different question: whether they had done what they understood, not whether the required state existed.
Separating the two helps measurably. Compiling the visible specification into source-linked obligations and letting only admissible evidence establish state raised the macro pass rate from 61.1% to 73.1% across the same seven models. A second line of work attacks the problem from the verification side and finds something counter-intuitive: disagreement between repeated runs exposes correct alternatives, while consensus conceals shared errors. Resolving the disagreements rather than counting votes added 6.2 points with one frontier model and 6.4 with another.
Measure the mechanism, not the outcome
A moral hazard study across fourteen models and 3,015 decisions put the warning sharply: optimisation can lift aggregate reward without restoring the cooperative behaviour the design was meant to produce. A model can score well on the outcome measure while doing none of the thing that measure was chosen to detect.
For a production system that is a warning about evaluation rather than about any model. A multi-agent setup judged on task completion can be improving on the metric while the coordination it depends on quietly stops happening, and nothing in the aggregate number shows it. The response is to measure at the level of the mechanism — not whether the system succeeded, but whether the agents consulted each other when the incentive said they should.
How far from home is the agent operating?
Governance of multi-agent systems collapses at a specific boundary, and a useful framework sorts the risk by the weakest governance binding any two interacting agents: singular, where one organisation deploys and governs everything and controls can simply be applied; federated, where several organisations operate under agreed rules and changing a control requires agreement; and open, where agents meet counterparties with no central authority and shared standards are adopted voluntarily if at all.
Inside the perimeter an organisation can log, sandbox and revoke. Once its agents transact with a partner's agents, and then with unknown counterparties, no single party sees the whole interaction and controls written for the first case quietly stop applying. Knowing which tier you are in tells you which of your existing controls are still doing anything.
A short list, in the order the decisions matter
- Scope every credential to the task and expire it with the run, before touching the prompt.
- Checkpoint after each externally visible action; do not extend the timeout.
- Calibrate memory per model and measure the token cost against the gain — it is a dose.
- Take completion out of the agent's hands: let evidence establish state, not the agent's claim.
- Sample repeatedly and resolve disagreement, rather than trusting consensus.
- Evaluate the mechanism as well as the outcome, or the metric will improve while the system decays.
- Prefer one well-built loop: in our own comparison, multi-agent configurations beat single-agent ones more narrowly than expected.
Questions
- Does the choice of agent framework matter?
- Less than it appears. Frameworks released over the past year converged on the same four pieces — planner, tool registry, memory store and supervisor — so the choice is largely reversible. The part that is not interchangeable is the supervisor: how failures are detected and what happens next.
- Should an agent be given more memory?
- Only after measuring it on your model. The same memory strategy across 585 tasks moved one model 39.9% to 56.0% and left another unchanged at 87.5%, while the full guideline set added up to 78% more tokens. It is a dose to calibrate, not a feature to switch on.
- Is prompt injection the main risk to an agent?
- It is the most discussed one; over-scoped credentials cause the damage. Published incidents this year followed the same path — the model called a tool it was allowed to call, with a credential broader than the task needed. Scoping and expiring credentials turns that class of incident into a logged failure.
- Why should the agent not decide when it has finished?
- Because its claim is not evidence. Across seven models, completion claims exceeded an official evaluator's pass rate by 28.7 to 37.9 percentage points, and only 79.6% to 86.4% of extracted task directions were actually satisfied. Letting admissible evidence establish state instead raised the macro pass rate from 61.1% to 73.1%.