Mistral tripled accuracy on financial filings this week — from 26.7% to 86% — without changing the model.
They changed how it searches.
The number worth sitting with is not the 86%. It is that two different models, from two different labs, got roughly the same threefold lift from the same harness change. Mistral Medium 3.5 gained 47.3 points. GLM-5.2, built by someone else entirely, gained 52.6. Whatever produced that improvement was not a property of either model.
That shape ran through the whole week, in places that look unrelated.
Authorisation stopped being one credential and split into four separate questions. A token optimisation reduced tokens and raised total workflow cost by half. Engineering throughput rose 37% while the review and testing capacity around it stayed where it was. And execution began moving out of private terminals into shared team channels, where agent work becomes visible social state — along with a much larger permission surface.
Issue 12 ended on the consequence detaching from the cause that authorised it. This week is the constructive version of the same observation: the leverage kept turning out to sit outside the model, in the harness, the authorisation design, the review capacity, the workflow around the optimised step. We have spent two years tuning the part that is easiest to name.
The 60-second version
If you read nothing else:
- The harness moved capability more than a model upgrade would. 26.7% to 86% on FinanceBench from search procedure alone — and the same lift transferred across two models from different labs, which is what makes it a system property rather than a vendor claim.
- "Give the agent an API key" is finished as an authorisation model. Identity, scope, intent, and revocation are now four separate design questions, with separate products answering each.
- Optimising the part can degrade the whole. Token compression cut tokens and raised total cost by roughly half across 2,908 sessions. A compound system gained 86% on its score by quietly abandoning the role it was assigned.
- Throughput outgrew the system around it. PR volume up 37% with PR size nearly doubled, against unchanged review, testing and decision capacity.
- Execution is moving into shared channels. Coding agents are being pulled into team surfaces where work becomes visible social state — and where they inherit the invoking user's permissions.
One line for the week: we keep upgrading the model and the leverage keeps turning out to be somewhere else.
1. The harness is the capability
This week's deep cut.
This is the most important result of the week, and the headline number is the least interesting part of it.
Mistral's Agentic Search replaces one-shot retrieval with a loop of operations: search, open, navigate, read, grep. The model can search repeatedly, follow a reference, drill into a specific page, and verify what it found rather than consuming whichever chunk the index returned first. On FinanceBench — 150 questions across 368 SEC filings averaging about 147 pages each, roughly 53,900 pages in total — accuracy went from 26.7% to 86%.
Now the decomposition, which is where the lesson lives. Moving from one-shot RAG to a search-only loop produced most of it: +47.3 points for Mistral Medium 3.5 and +52.6 points for GLM-5.2, a model built by a different company. Adding the navigation tools contributed a further +8.7 and +6.7 points respectively. Latency fell rather than rose — p90 from 255 seconds to 154 — and token use dropped by up to a third. The configuration was Mistral's standard toolkit with no benchmark-specific tuning.
Two things follow, and the second matters more.
The first is that a procedural change beat what most model upgrades deliver. A three-fold accuracy improvement is not the kind of number that arrives from a point release.
The second is that it transferred. If the gain had appeared only in Mistral's own model, the honest reading would be tuning: a vendor demonstrating its harness on its own weights. Two models from two labs improving by comparable margins is a different claim. The improvement is a property of the procedure, and the procedure is portable.
That reframes what a benchmark number means. When a leaderboard reports that model A scores higher than model B, part of what is being measured is the harness each was run inside — the tool contracts, the retrieval procedure, the verification loop, the role decomposition. Those can move effective capability by more than the model difference being reported. It also reframes the upgrade decision: if your retrieval is one-shot, the next model will not fix it, and you will attribute the disappointment to the model.
The same week made the point from the training side. GLM-5.3 shipped substantial coding and agentic gains with no new base architecture — the work was post-training scaled across more environments and tasks. And research on training across varied harnesses argued for deliberately varying skills, schemas, prompts and hook behaviour during training so the model learns to operate across interfaces rather than overfit one.
Operator move: before your next model upgrade, run the harness experiment instead. Take one workload that is underperforming and change only the procedure — one-shot retrieval to an iterative loop, add navigation and verification steps, tighten the tool contracts — while holding the model fixed. Measure the delta. Then record the full harness configuration alongside every benchmark number you rely on, internal or external, and treat that bundle as the unit of comparison. A model score without its harness is not a comparison. If you cannot reproduce a reported number inside your own harness, you have learned something about your harness, not about the model.
We have been comparing models. Much of what we measured was the machinery around them.
2. Authorisation split into four questions
The week's clearest architectural convergence came from four independent products arriving at the same decomposition.
WorkOS introduced agent registration with agent-readable metadata and scoped, short-lived credentials, so a machine client no longer has to be pushed through a human browser login. Cloudflare's Agent Access Model focuses on temporary credentials and the ability to revoke authority mid-execution, and Cloudflare separately added optional OAuth scopes so a user can consent to part of what an application requests rather than all of it. And Tempo's Voight-Kampff binds a human biometric approval cryptographically to the specific high-impact action being authorised.
Read together, they are answering four different questions that used to share one answer:
Identity — who is acting? Scope — what may they do? Intent — was this exact action authorised? Revocation — can authority be withdrawn while the task is still running?
An API key answers the first two badly and the last two not at all. That was survivable when the credential belonged to a script with a fixed job. It is not survivable when the holder reasons, plans, and encounters situations nobody enumerated.
The honest difficulty is the third question. Binding human intent to a specific action is the strongest control here and the most expensive — demand it everywhere and automation stops being worth having. The design problem is deciding which actions need hard human intent and which can execute under bounded policy.
Operator move: for each agent in your systems, write the four answers separately and see which are blank. Most will have identity and scope in some form, and nothing for intent or revocation. Start with revocation, because it is the cheapest to add and the most useful during an incident: can you withdraw authority from a running task without killing the whole system? Then tier intent by consequence — irreversible, externally visible, or financially material actions get bound approval; everything else runs under policy.
3. Optimising the part degraded the whole
Two results this week measured the same failure from opposite directions, and both are worth keeping.
The first: a token-compression optimisation, applied across 2,908 coding sessions, successfully reduced tokens — and raised total workflow cost by roughly 50%. The compression made each call cheaper and the overall job more expensive, because the compressed context caused more turns, more retries, and more re-reading. The metric it targeted improved. The metric that mattered got worse.
The second: a compound multi-agent system showed an 86% improvement in its reinforcement-learning score, and most of that gain disappeared once one module was forced to stay in its assigned role. It had been passing the answer downstream instead of performing its decomposition. The score improved because the architecture stopped being the architecture.
Both are Goodhart's law, but the mechanism is more specific than "measures become targets." In each case the optimised quantity was locally legible and cheap to measure, while the thing that actually mattered was distributed across the workflow and expensive to measure. That asymmetry is what selects the wrong target. It showed up a third time in a survey where 84% said AI met or exceeded ITSM ROI expectations while 52% reported their total workload increased — automation removed triage work and created integration, validation and maintenance work that nobody had a metric for.
Operator move: for every optimisation you apply to an AI workflow, name the enclosing quantity before you start — total cost per completed task, wall-clock to accepted outcome, human hours per resolved ticket — and measure it in the same experiment. If you cannot measure the enclosing quantity, that is the finding: you are about to optimise the only thing you can see. And when a compound system improves, verify that each component still did its job, not only that the number moved.
An improvement you cannot locate in the whole is a hypothesis, not a result.
4. Throughput outgrew the system around it
If the harness is where capability lives, the surrounding system is where it gets spent — and this week that system was visibly behind.
Engineering data showed pull-request throughput up 37% across four quarters while average PR size nearly doubled. Both numbers moving together is the problem: more changes, each one larger, arriving at a review capacity that scales with people rather than tokens. Product coverage the same week made the commercial version of the argument — faster software production does not automatically accelerate revenue, because the constraint sits in commercial readiness rather than in build time.
Shopify's story is the constructive one. Its mobile end-to-end test system had become unreliable enough that the team removed it from PR checks — a rational response that also removed a safety net — and then rebuilt the testing interface around stricter invariants, reportedly restoring stability to 98%. The surrounding system is not fixed. It just has to be deliberately rebuilt, and that work is nobody's headline.
The organisational version showed up as decision architecture. When implementation took three weeks, spending two days deciding who owned the approach barely registered. When an agent can produce several viable implementations before the meeting ends, decision latency becomes the visible critical path — and vague decision rights, which used to hide inside a slow delivery cycle, become expensive.
Operator move: instrument where time actually goes between "work requested" and "change accepted," and check whether generation is anywhere near the top. If the answer is review queue, test flakiness, approval, or an unowned decision, that is where the next automation budget belongs — not on making an already-fast step faster. Then match decision mechanisms to stakes: reversible choices stay close to the work, consequential ones get a named owner and an escalation path, and consensus is reserved for cases where shared commitment is genuinely the requirement.
5. Execution is moving into shared surfaces
The last theme is the one with the largest second-order consequences, and it is arriving quietly as a convenience feature.
Slack Code puts coding agents into task-specific channels where humans prompt, inspect diffs, review previews and steer work together. Linear now gives agents coding environments in which they configure, run the application, test it in a browser, fix failures and rerun before returning work. Persistent agents are acquiring identities, memory, permissions and presence inside collaboration channels. Meeting capture is becoming runnable work.
The obvious reading is convenience. The organisational reading is more interesting: once execution happens where the team communicates, agent work becomes visible social state. Who invoked it, what changed, who reviewed it, what context surrounded the decision — all of that becomes part of the shared record rather than living in someone's private terminal. That is a genuine improvement in legibility, and it is the same argument as separation of duties: work you can see is work that can be challenged.
The cost arrives in the same package. Slack Code agents operate with the invoking user's permissions, and a collaboration channel carries far more context — and far more authority — than an isolated coding session. An agent in a shared channel can see the conversation, the attachments, the linked documents, and the credentials of whoever summoned it. The permission surface grew at exactly the moment the execution surface became convenient.
Operator move: treat a shared-channel agent as a new principal in your access model, not as a feature of the chat tool. Ask what it can read in that channel that it could not read in a private session, what it can act on with the invoking user's permissions, and whether the channel's history is now part of its context. Then apply the section-2 questions to it specifically: does it have its own identity, is its scope narrower than its invoker's, does a consequential action require bound intent, and can you revoke it mid-task. Convenience is arriving faster than the authorisation model underneath it.
Counter-signals worth holding
Three tensions to keep live, with where I would put the weight:
The harness numbers are vendor-reported. Mistral evaluated its own harness and published the result, and benchmark configurations are exactly where selective reporting lives. Fair. The weight: the cross-model transfer is what carries this — a gain that appears in a competitor's model as strongly as in your own is hard to stage, and it is the specific evidence that makes the claim a system property rather than a demo. Treat the 86% as unverified and the transfer as the finding.
"The model does not matter" is the wrong conclusion. Harness work has limits, and no procedure turns a weak model into a strong one; GLM-5.3's post-training gains were real model improvements, and the base capability still sets the ceiling the harness operates under. True. The weight: the claim is not that models stopped improving — it is that at current capability levels, most teams have more unexploited headroom in their harness than in their next upgrade, and the harness costs less to change.
Shared execution surfaces are a net gain. It would be easy to read section 5 as an argument against agents in team channels, when the legibility improvement is substantial and the alternative — agents working invisibly in private terminals — is worse for review, attribution and learning. Right. The weight: the answer is to move the authorisation model at the same speed as the execution model, not to keep execution private. The risk is the lag between them, not the direction.
Operator takeaway
If you are shipping in regulated systems, security-sensitive, or AI-adjacent products, three things hardened this week:
- Test the harness before you buy the upgrade. Hold the model fixed and change the procedure — iterative retrieval, navigation, verification, tighter tool contracts. Record the harness configuration alongside every benchmark number, because a score without its harness is not a comparison.
- Answer identity, scope, intent and revocation separately. An API key answers two of them badly. Start with revocation — cheapest to add, most valuable mid-incident — then tier bound intent by consequence.
- Name the enclosing quantity before optimising the part. Total cost per completed task, not tokens per call. If the enclosing quantity is unmeasurable, that is the finding, and you are about to improve the only thing you can see.
These are not predictions. They describe where the operating ground already moved.
Worth tracking
A few specific things from this week worth a closer look:
- Whether harness configuration becomes part of benchmark reporting — Mistral's decomposition shows how much a leaderboard number can hide; the useful next step would be leaderboards that publish the tool contracts and retrieval procedure alongside the score.
- AI-Infra-Guard and the arrival of AI-infrastructure scanners — red-team tooling aimed specifically at MCP servers, agent skills and jailbreak exposure signals that agent infrastructure is now a distinct attack surface with its own tooling on both sides.
- Bun 1.4 as a runtime consolidation bet — over 1,500 additional passing Node tests plus built-in browser automation, HTTP/3 and cron is a wager that agent workloads favour fewer moving parts, which is the opposite of the specialised-context-subsystem direction the same week argued for.