Every organization already sits on a recording of how its work actually gets done. It just isn't readable. Screenshots, clicks and keystrokes pile up as a stream of low-level events with no structure attached, and the knowledge inside them stays locked up.
A paper posted to arXiv on Aug. 20 proposes a way to unlock it. Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen and Diyi Yang describe Task Model Induction, or TMI, a method that reads passively captured computer-use traces and outputs symbolic models of the tasks buried in them.
Why this is hard
The obstacle is that people do not work one task at a time. A single afternoon's trace might interleave an expense report, three Slack replies and a half-finished spreadsheet, all sharing the same windows and the same input device. Nothing in the raw event log marks where one goal stops and another starts.
Prior approaches sidestep the problem by assuming it away. They either take the task as given or assume the trace contains a single workflow, and what they produce tends to be a summary of steps rather than a structured model with any real internal logic.
TMI works in two passes. First it finds the latent tasks inside an unconstrained trace, separating activity that ran concurrently. Then, for each task it isolated, it builds two paired artifacts: an objective model that decomposes the goal recursively into sub-goals, and a procedure model describing the control flow that actually organized the execution.
The numbers
Tested on controlled trajectories from both humans and agents, TMI matched ground-truth task groupings at 0.974 agreement — close to perfect separation of interleaved work. It reconstructed 74.9% of the execution steps observed in the trace, which the authors report is well ahead of the strongest workflow-induction baseline they measured against.
The extrinsic test matters more for practitioners. Skills derived from TMI's task models raised accuracy on held-out tasks by 30.0% over the best competing method, meaning the induced structure carried enough real procedural knowledge to transfer to work the system had not seen.
Auditable by design
The framing here is deliberate. As computer-use agents move into real jobs, two constituencies need something the models themselves can't provide. Agents need to learn how a task is genuinely performed inside a specific organization, not how a generic tutorial says it should be. Compliance and operations teams need to inspect that knowledge and reuse it.
A symbolic task model is legible in a way that weights or a chain of screenshots is not. Someone can read the goal decomposition, disagree with a branch, and edit it. That is a meaningfully different artifact from a demonstration an agent imitates opaquely.
The obvious caution is the input. Passively recorded screen activity is among the most sensitive telemetry a company can collect, and any deployment inherits questions about consent, retention and what happens when a trace captures a colleague's private message. The paper's evaluation also leans on controlled trajectories, which are cleaner than the genuine chaos of a working week.
Still, the direction is telling. Much of the current agent push assumes the right move is a bigger model with better tool access. TMI argues the missing ingredient is a readable representation of the work itself — and that it can be recovered from data most workplaces are already generating without meaning to.
