Why bolted-on AI agents keep failing at real computer work

You shouldn't have to babysit an agent through the work you gave it. Consider a familiar failure: it edits a spreadsheet, moves to a browser, misses a changed instruction and submits the old result. Every click can look competent while the task goes wrong. Bolted-on AI agents leave the person responsible for catching the gap between visible activity and finished work.

What bolted on means for computer work

Our explanation for why computer use agents fail starts with the environment: an agent added to a human desktop must infer task state from interfaces that do not expose everything it needs. Its application then has to preserve the task and govern actions across other applications. When that surrounding system leaves gaps, a capable model can still produce the wrong outcome.

A computer use agent observes an interface and takes actions through it. By bolted on, we mean an agent running as an application or remote operator over an existing human desktop or browser. It may use screenshots, page structure or accessibility information. Those choices change how it reads the interface. They do not automatically give it a durable account of the work.

When it operates through your logged-in session, the destination can see your account's authority without knowing the narrower instruction you gave the agent. A spreadsheet application knows about its document; a browser knows about its pages. Neither necessarily knows that this export must use a revised constraint received in another app.

Good agent applications can add safeguards. The burden is keeping them consistent wherever the agent can act. Our OS versus app argument concerns who owns that responsibility across applications. The long-tasks essay covers how length amplifies errors. This post examines the computer environment in which those errors become real.

What the benchmarks show, with the eras kept separate

The original OSWorld paper from 2024 reported a best model success rate of 12.24%, against a human result above 72.36%, on its 369-task benchmark. Its analysis identified trouble connecting visual controls to actions and knowing how to operate applications. Those figures describe that early evaluation, not today's ceiling.

The project's later OSWorld-Verified release includes corrected tasks and updated evaluation results. Its published results spreadsheet, fetched for this article, includes substantially higher scores. Quoting 12.24% as current computer-use performance would mislead readers. It would also miss the harder problem that the next benchmark examines.

OSWorld 2.0, introduced in June 2026, evaluates 108 longer workflows. In the paper's strongest reported configuration, Claude Opus 4.8 with maximum thinking and batched tool calls completes 20.6% of tasks under the binary metric at 500 steps, with a 54.8% partial score. That partial score measures progress; it does not mean 54.8% of workflows finished.

Read these evaluations separately

Evaluation	Reported result	Limit on the comparison
OSWorld, 2024 paper	12.24% best model success; humans above 72.36%	An early baseline on the original task set
OSWorld-Verified, later release	Updated tasks and higher published scores	Changed evaluation settings prevent a simple historical comparison
OSWorld 2.0, 2026 paper	20.6% full completion; 54.8% partial score in the strongest reported setting	108 longer workflows, evaluated with a 500-step budget

The 2.0 authors report agents losing constraints, missing information received during work and skipping verification. Failures concentrate around state the agent has to recover or maintain. These are research results from the tested configurations. They do not establish that an operating system change alone would fix them, or that every agent application fails in the same way.

A readable interface can still hide the work

The visual problem is easy to recognize. A control moves, a window covers it or the page changes between observation and action. A repeated click can send the agent into a loop. Better visual grounding helps it choose the intended control; structured interface information can reduce the need to infer meaning from pixels.

But reading a button label leaves other questions unanswered. The interface may not reveal whether a request finished after a timeout, whether a document is the approved version or whether an instruction elsewhere has changed. Our desktop AI agent explainer separates the action interface from the places where processing and saved history live.

In the illustrative spreadsheet task, recognizing the submit button perfectly would still submit the wrong result. The agent needs to connect what it sees to the current task. Improving the click cannot repair a constraint the agent never carried forward.

The task has to survive a change of context

Imagine the instruction changes from including all draft entries to excluding anything still awaiting review. That change must affect the eventual export, even if the agent has since opened another application or shortened its conversation history. Saving a transcript somewhere does not establish which instruction is now in force.

Durable task state should identify the current objective, applicable constraints, completed effects and unresolved checks. Our account of why agents forget between sessions explains the continuity problem. The practical requirement here is narrower: an agent resuming a workflow needs to recover the governing instruction before it takes the next consequential action.

Ankur Sharma and Deep Shah's June 2026 AOS paper argues that agents running as user-space applications stress existing boundaries for scheduling, state, security, observability and governance. It proposes several integration models, including control planes above existing operating systems. That is an argument for explicit system responsibilities, not evidence that every useful mechanism requires a replacement kernel.

A prompt cannot carry the whole action contract

Laurent Bindschaedler opens his AgenticOS keynote notes with three failure stories. A retry duplicates an accepted order. Context compaction drops a standing rule before customer messages go out. A rebase changes reviewed code, but a workflow reuses the earlier approval to merge it. The common defect is a rule the runtime cannot check.

Those examples concern different boundaries. A retry needs to know whether an effect already occurred. A standing restriction needs to survive shortened context. Approval needs to remain attached to the version it covered when the workflow resumes. A completion label can conceal a violation of any of those rules.

Zheng Liu and colleagues sharpen the tool boundary in The Abstractions Are Slipping: Four OS Mismatches in the Agent Era. Their M2 diagnosis is that a contract in a framework covers the routes that framework wraps. Calling the same tool from a shell can step outside that contract. M4 concerns effects beyond local state, where the system cannot infer whether an action is reversible.

They call for tools to declare the nature of their effects and for enforcement below the agent. That is the difficult part of computer use agent reliability: the restriction must still apply when the agent chooses another route to the same effect. An approval convention in the normal workflow does not establish that coverage.

Agents keep moving when they need to inspect

The OSWorld 2.0 trajectory analysis reports that recovery and repair together account for less than 7% of the budget across the analyzed systems. The paper describes agents spending little explicit effort fixing earlier mistakes. More actions can therefore extend a failure instead of correcting it.

After an uncertain submission, safe recovery starts with the destination. The system needs to establish whether the effect happened before deciding to retry. Reopening a local checkpoint cannot undo an external submission. Some effects can be reversed; others require a separate corrective action or an owner decision.

A useful test for an agent system is to interrupt a workflow immediately after an effect but before its completion record. On resume, it should inspect the unresolved outcome. If it simply repeats the last command, its recovery path can create another failure. This is a proposed evaluation check, not a result we claim for ERIKA.

What has to sit underneath the agent

The AgenticOS workshop at SOSP 2026 brings together work on agent execution, long-lived state, isolation and recovery. Bindschaedler's research language is four runtime contracts: effect, state, composition and intent. They concern when an action becomes real, what survives compaction, what holds when steps are chained and who checks generated code. They describe guarantees to state and test, rather than a settled implementation.

We argue that memory belongs with the operating system because the task must outlive its current model call and application. Its stored state needs current instructions and evidence references, with rules for updates and expiry. Keeping everything forever would leave old instructions competing with new work.

Permissions and progressive trust need an enforceable boundary around the action. Start with limited authority and expand it when the work justifies that decision. The system should check the applicable grant against the current target before allowing an effect, including during retries.

An evidence trail should connect the instruction and authority to the observed result. Recovery uses that record to establish what finished and what remains uncertain. Keeping the record outside the failing session makes it available when that session can no longer explain itself.

For the owner, control of the agent's capability includes control of this history and authority. Replacing a model should leave both inspectable. This is the system layer discussed in our AI operating system explainer; here the failure cases give concrete tests for it.

The ERIKA direction starts with the AI user

eFreedom is building ERIKA, its one product, as an operating system where the user is purely AI. That puts responsibility for durable work and its boundaries with the system the agent inhabits. Memory, permissions, evidence and recovery become system services that individual applications use.

ERIKA is in active development. The studies above diagnose failures and propose mechanisms; they do not establish ERIKA's performance or prove its recovery behavior. The ERIKA architecture explains what we are building. Judge that work by whether a changed instruction still governs the next action and whether an interrupted task can resume without blindly repeating an uncertain effect.