AI agents are getting better at bounded tasks. That is useful, but it creates a measurement trap: the part we can score becomes easier to automate while the consequential remainder stays outside the score. This week, that remainder appeared in five different places. A benchmark covered a narrow slice of accounting work, not the whole job. Search agents found correct rows while still missing qualifying entities. Coding agents had useful skills installed but often failed to discover them. Security controls approved an entry point while missing downstream effects. Provenance signals correctly described origin while saying much less about safety, authorship, or responsibility. This week's spine: agents are increasingly good inside scoped tasks, but the failures are moving to the unmeasured remainder — work outside the benchmark, entities outside the result set, capabilities outside discovery, effects outside the permission check, and conclusions outside what the evidence can actually prove.
The 60-second version
- Mercor found frontier models outperforming a small group of licensed accountants on four bounded month-end-close scenarios, while explicitly warning that the study removed much of the surrounding job. A benchmark can establish task capability without establishing job coverage.
- Exa's ATLAS makes completeness visible by scoring discovery as well as correctness. Its best tested configuration reached 0.66 row F1 only at materially higher time and cost, and even expensive systems still missed a substantial share of the golden set.
- Expo increased downstream skill loading from 9% to 55% by adding one routing entrypoint, yet Relevant Skill Recall rose only from 2% to 18%. A capability that exists but is not discovered is operationally absent.
- PromptArmor, GitLab, and AWS all point at the same control problem from different angles: authorizing a tool or wrapper is weaker than constraining the effects that tool can produce.
- StepSecurity documented a malicious package released through its project's normal pipeline with a valid npm provenance attestation, while OpenAI explicitly describes text watermarking as a limited provenance signal rather than proof of authorship, accuracy, or responsibility. Evidence needs an explicit coverage boundary. The practical move is to stop treating a successful local check as proof that the relevant space was covered. Every important agent workflow needs a coverage contract: what is in scope, how completeness is tested, what remains unknown, and who owns the remainder.
A Benchmark Can Cover a Task and Miss the Job
Benchmark discussions often collapse two questions into one: can a model do this task, and can the surrounding work be delegated? They are related, but the second question contains much more surface area. Mercor's accounting study is unusually useful because the authors state the boundary clearly. Twelve licensed CPAs completed four simplified month-end-close scenarios. Claude Opus 5 scored 100% across 20 attempts, while the participating accountants averaged about 37%. The result is strong evidence that frontier models can outperform humans on this particular class of structured, file-heavy, instruction-following work. Mercor then explains what the study did not measure. The setting removed coworkers, accumulated organizational context, client communication, question asking, and other less structured parts of accounting work. The tasks were deliberately constructed around skills models currently handle well. That caveat is not a footnote to the result; it defines what the result means. Epoch AI reaches the same boundary from the opposite direction. It gave six models real internal tasks, provided the relevant resources and tools, and graded the outputs against Epoch's own standards. Frontier models were reliable on many well-defined parts but repeatedly struggled with open-ended judgment: learning implicit conventions, designing an informative experiment, noticing a flawed setup, or deciding which result actually matters. Epoch explicitly warns that its 11-task suite is small, subjective, and specific to its own work. The useful unit for adoption is therefore not the occupation and not the benchmark score. It is a responsibility boundary. Which part of the workflow has a sufficiently explicit objective, sufficiently observable acceptance criteria, and sufficiently bounded consequences that a system can own it? Which parts still depend on tacit context, stakeholder negotiation, exceptions, or judgment that cannot be reduced to the current task definition? That distinction also changes how teams interpret progress. A benchmark improvement may justify automating more of a workflow without justifying removing the person who owns the outcome. The organization may still need someone to decide which case should run, resolve ambiguous inputs, approve exceptions, communicate with affected people, or notice that the correct answer to the scored task is irrelevant to the real situation. Operator move: for every benchmark or internal eval you use to justify automation, write down the larger job boundary beside it. List the important responsibilities the eval does not exercise, then decide whether they are deliberately human-owned, covered by another test, or simply unmeasured. A benchmark is most useful when its boundary is visible. The dangerous version is not a narrow benchmark; it is a narrow benchmark presented as a wide claim.
Exhaustive Research Needs a Coverage Contract
This week's deep cut. Most research evals reward whether the answer is correct. Many production research tasks care about something harder: whether the answer is complete enough to act on. That difference matters whenever the task asks for every qualifying vendor, every exposed asset, every regulation in scope, every account matching a condition, every relevant paper, or every company that meets an investment screen. A research agent can be perfectly accurate about the entities it returns and still fail the user by omitting half of the relevant set. Exa's ATLAS benchmark is designed around that failure mode. It contains 547 tasks that require systems to discover a set of entities and enrich each entity with multiple attributes. Exa reports that 74% of the tasks require at least ten entities, while full discovery and enrichment touches a median of 18 domains. Its headline Row F1 counts a row as correct only when every required cell is correct, so recall is part of the score rather than an invisible assumption. The initial results make the cost of completeness visible. Exa reports that no tested system under $1 per task reached 0.5 Row F1. The best configuration reached 0.66 at an average of $8.92 and 18 minutes per task. Even the expensive systems were still missing roughly a third of the golden results. These are vendor-reported benchmark results, not a universal law of research, but the mechanism transfers: completeness requires search breadth, repeated verification, and an explicit stopping rule. The same pattern appears in independent work. WideSearch was created specifically because large-scale information collection remained poorly evaluated; its 200 tasks require agents to collect many independently verifiable pieces of information. The paper reported very low end-to-end success for the tested systems, despite human cross-validation being able to approach complete answers given enough time. More recently, Perplexity's WANDR evaluates 500 wide-and-deep collection tasks and separates retrieving a plausible page from supplying evidence that supports every required part of a record. Its published analysis shows that much of the loss happens after a relevant page has been found: evidence coverage and requirement coverage remain incomplete. This suggests a different contract for production research. Before the agent starts, define the population. What qualifies? What sources are admissible? What exclusions are allowed? What level of recall is economically justified? What does "no more found" actually mean? Then make the system report coverage evidence rather than only the resulting prose or table. There are several practical ways to do this. Use independent discovery paths and compare their overlap. Track newly discovered entities per search pass and stop only after the marginal yield falls below a defined threshold. Keep explicit exclusion reasons. Separate "not found" from "confirmed absent." Sample missed entities after the run. For recurring research, retain the discovery frontier so the next run can show what changed instead of starting from an opaque blank page. None of these mechanisms creates perfect exhaustiveness. The web changes, definitions remain ambiguous, private information is inaccessible, and a gold set can itself be wrong. The point is not to promise total coverage. The point is to stop hiding coverage uncertainty behind a polished answer. Operator move: add a coverage section to any research workflow where omissions matter. Require the agent to state the target population, discovery paths used, exclusions, unresolved candidates, and the evidence behind its stopping decision. Track recall on sampled or known sets separately from factual accuracy. When omission has a cost, "everything I found is correct" is not a sufficient quality metric.
Undiscovered Capabilities Are Operationally Missing
Agent platforms increasingly accumulate tools, skills, plugins, documents, and specialized procedures. That looks like capability growth on an inventory page. The runtime question is whether the agent actually reaches the right capability when the task requires it.
Expo's skill evaluation gives a clean example. The team tested Claude Code against ten greenfield mobile-app PRDs, three runs per prompt, without naming the relevant skills in the user request. Before an overview router existed, only about 9–10% of sessions loaded an Expo skill. After Expo added expo-overview as a dependable first hop, 55% loaded at least one downstream skill.
That is a substantial improvement, but the more revealing metric is Relevant Skill Recall. Expo had mapped which skills were useful for each task. Recall rose from 2% to 18%, meaning the agent still skipped many relevant capabilities even after the routing improvement. The team also observed that discovery was strongest early in the task and weakened later, as the agent increasingly relied on its own knowledge.
Expo found another boundary in the surrounding interface. In the Codex version it inspected, skill descriptions shared a fixed catalog budget tied to the context window. With many installed skills, descriptions were shortened and could eventually disappear from the catalog. More available capabilities can therefore make discovery harder if the routing surface degrades as the catalog grows.
This is a general systems problem. A tool registry can be complete while task-level capability coverage is poor. Documentation can exist without entering context. A specialist agent can be provisioned but never delegated to. A policy can recommend a procedure that the planner never retrieves. From the user's point of view, those capabilities do not exist during that run.
The evaluation target should therefore include the route, not only the destination. Did the task trigger the right capability? Was the capability visible at the moment it became relevant? Did the agent return to discovery after the task changed? Did catalog truncation, naming, or description quality alter routing? And if the capability was skipped, could the system detect the miss rather than silently continue?
Operator move: maintain a task-to-capability coverage set for important workflows. Measure discovery recall separately from execution quality, test catalogs under realistic crowding, and add a small number of stable routing entrypoints instead of assuming that a longer list of tools creates a more capable system.
Installing a capability is supply. Routing is what turns it into usable capacity.
Authorization Must Cover Effects, Not Just Calls
Permission systems for agents often operate at the interface boundary: this tool is allowed, that command requires approval, this connector is read-only. The problem is that the visible call can be a poor description of the effects that follow.
PromptArmor's Copilot CLI report demonstrated a controlled prompt-injection case where a command beginning with a supposedly read-only wrapper produced downstream execution without the expected approval. The important point is not the particular command syntax. The approval decision classified the wrapper while the consequential behavior happened deeper in the execution chain.
GitLab Threat Research documented a different path in DeepSeek-Reasonix Studio, assigned CVE-2026-102437. Under specific repository-delivery conditions, Git configuration semantics could turn what looked like repository data into an execution path. GitLab also states an important boundary: .git/config is not transferred by a normal clone, so the exact path required a different delivery mechanism. Again, the broader issue is transitive semantics. An agent that invokes Git inherits more than the command name exposed by its own tool schema.
AWS makes the architectural distinction explicit in Strands Box, currently a developer preview. AWS argues that agent-process allowlists judge the requested tool call, while enforcement points closer to the operating system can observe filesystem operations and network requests. Strands Box combines isolation with policy checks at those boundaries and can keep real credentials outside the agent environment, attaching them only to permitted outbound requests.
These examples do not imply that every agent needs an elaborate policy engine. They do show why approval coverage has to match effect coverage. A read-only label is meaningful only if the system can explain what "read" may transitively invoke. A network rule matters only if nested execution cannot reach an unobserved path around it. A credential scope is useful only if the credential cannot leak into a context where another tool can reuse it.
Operator move: threat-model agent actions by effective side effect. For high-consequence workflows, map tool calls to filesystem, network, credential, process, and external-system effects, then place enforcement at the layer that can actually observe those effects. Treat wrappers and human-readable labels as interface hints, not security boundaries.
The control surface is complete only when it covers what the system can cause, not merely what the agent asked to call.
Evidence Must State What It Does Not Prove
Agent systems are accumulating more evidence: provenance attestations, watermarks, signed traces, benchmark scores, tool logs, and policy decisions. More evidence is useful, but each signal proves something specific. Reliability drops when one proof is silently promoted into another.
StepSecurity's Tensorlake investigation is a sharp example. The malicious tensorlake@0.5.144 release was built from the project's own main branch and published through its normal release workflow, so npm showed a valid provenance attestation. The provenance was not false. It correctly described where the package was built. The failure was treating that origin evidence as evidence that the source and workflow were trustworthy.
The same distinction appears in a completely different domain. In its EU text provenance documentation, OpenAI explicitly states that a detected watermark is a limited signal about the role its systems played in generating or processing text. It does not establish ownership or responsibility, identify the user, or verify that the text is accurate. The absence of a detected watermark also does not prove human authorship.
This is the pattern to preserve in agent infrastructure: every evidence object should carry a semantic contract. A signature can establish who signed a thing. A provenance attestation can establish a build path. A test can establish behavior over the cases it exercised. A trace can establish what was recorded. A human approval can establish that a person approved the representation they were shown. None of these automatically establishes every adjacent property people may want to infer.
Once automated systems consume evidence, the distinction becomes operational. If a release gate treats provenance as a safety verdict, or a content policy treats watermark absence as proof of human origin, the system has converted a valid signal into an invalid decision. The bug is not necessarily in the evidence. It is in the inference contract around it.
Operator move: give every machine-consumed evidence type an explicit "proves / does not prove" definition. Encode the downstream decisions that may rely on it, and require additional evidence when the decision crosses that boundary. Preserve inconclusive states rather than coercing them into pass or fail.
Evidence becomes safer when its limits are represented as clearly as its positive result.
Counter-signals worth holding
More coverage is not automatically better. Exhaustive research, broader testing, deeper policy interception, and repeated capability discovery consume time and money. The evidence this week supports measuring coverage where omissions carry real cost, not maximizing recall everywhere. A low-stakes recommendation and a compliance inventory should not pay for the same search budget. Coverage mechanisms can create their own blind spots. A router can go stale, a gold set can be incomplete, an operating-system policy can miss unsupported execution paths, and a provenance schema can be interpreted too broadly. Moving uncertainty into explicit machinery makes it inspectable; it does not make the machinery infallible. Some work remains genuinely open-ended. Epoch's results are useful precisely because they show that real work contains judgments that are difficult to enumerate up front. A coverage contract should expose what remains outside formal evaluation, not pretend that every professional standard can already be compiled into a checklist.
Operator takeaway
- Move from score to coverage envelope. For every important eval or verification result, record what population, workflow stage, effect surface, or claim class it actually covers — and what remains outside.
- Measure routing and discovery as part of capability. Inventory alone is not enough. Track whether the agent reaches the right tools, skills, evidence, and reviewers at the point they become relevant.
- Make evidence semantics executable. Treat provenance, approvals, tests, and traces as typed signals with explicit downstream permissions, rather than generic proof that the system is safe or finished.
Worth tracking
- Wikimedia Foundation documented agent-associated automated activity that consumed investigation and infrastructure attention on systems outside the agent operator's organization. This is a useful reminder that workflow coverage can include externalized operational cost.
- Airbnb Engineering described production-workload capture and replay for database migrations, a concrete way to widen test coverage beyond synthetic cases without claiming that replay is identical to production.
- Truffle Security found large differences in how long exposed credential types remained usable in a historical GitHub corpus. Detection coverage and lifecycle closure are separate controls; revocation latency is worth measuring directly.