Skip to content

fix(agent-task): always report why the agent runtime failed and keep its transcript - #2590

Merged
chubes4 merged 1 commit into
mainfrom
fix/agent-runtime-failure-reason
Nov 15, 2026
Merged

chubes4 merged 1 commit into
mainfrom
fix/agent-runtime-failure-reason

Conversation

@chubes4

@chubes4 chubes4 commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

Problem

Docs Agent runs 43729752419 (community bootstrap) and 43729756527 (shop maintenance) on v0.33.5 failed after ~3 minutes of agent execution with workspace_apply.status: skipped, write_tool_calls.source: none, no transcript, and on the shop run no failure object at all.

Root cause

  • readNativeResult rejects native results over 1 MiB (result-too-large) and replaces the whole result with a bare {success:false, diagnostics}. The native agent-task-run/v1 result embeds the full recipe run, including the agent loop output and its messages (every workspace_read payload). At 12 turns that fit; with fix(agent-task): honor caller max_turns over package defaults and report turn-cap stops #2589 honoring max_turns: 40 the result outgrew 1 MiB. The metadata, sandbox write ledger, tool observability and transcript refs were all thrown away with it.
  • When execution.code === 0 but runtimeResult.success !== true, the executor built no failure at all, and the uploaded reviewer projection drops runtime_result and the execution tails. So the reason never reached the uploaded result. (The community run only showed a downstream verification failure, because verification still ran after the failed runtime.)
  • Tool observability was rejected outright above 64 calls (runtime-core, executor, uploader). 40-turn runs make well over 64 calls, so write evidence disappeared even when the result was readable.

Fix

  • New agent-runtime-failure.mjs: every non-zero exit or non-successful runtime yields failure{stage:"agent", code, classification, message}, plus diagnostics, exit code, runtime/terminal status, provider error and stream tails on non-zero exits. Verification is skipped after a failed runtime so it can't mask the cause.
  • MAX_NATIVE_RESULT_BYTES goes from 1 MiB to 32 MiB, and the error now reports the actual size.
  • When the native result carries no transcript ref, the executor discovers exactly one runtime-*/files/transcript.json (best effort; it never masks the failure). The uploader accepts transcript sources above the 4 MiB per-file bound, because it only stages a bounded projection.
  • Tool-observability and write-ledger projections are bounded at 512 calls instead of 64.

Tests

  • tests/agent-runtime-failure.test.ts (new unit test)

  • tests/execute-native-agent-task-lifecycle.test.mjs gains two new modes:

    • runtime-failed: a schema-valid failed result with a provider diagnostic.
    • result-missing: the CLI exits 1 without writing a result.

    Both assert the agent-stage failure in the executor result and in the uploaded result, the discovered transcript, and more than 64 tool calls surviving in the uploaded transcript.

  • Existing agent-task, reusable-workflow, max-turns, interruption and apply-outcome tests pass locally.

AI assistance: Extra Chill Bot (Claude), via a Roadie subagent.

…its transcript

A runtime that exited non-zero or returned an unreadable native result produced
success:false with no failure object, no transcript and no tool observability.
- project every agent-runtime failure into failure{stage:agent,code,message}
  with diagnostics, exit code and stream tails; skip verification after it
- raise the native result bound from 1 MiB to 32 MiB (long runs embed the full
  agent loop output and were silently discarded) and report the actual size
- discover the runtime transcript when the native result carries no ref
- raise the tool-observability bound from 64 to 512 calls so long runs keep
  write evidence; let the transcript source exceed the 4 MiB upload bound
@chubes4
chubes4 merged commit 3a6688a into main Nov 15, 2026
4 of 5 checks passed
chubes4 added a commit to Automattic/docs-agent that referenced this pull request Nov 15, 2026
…e user packages (#189)

- WP Codebox v0.33.6 (4875fb1) reports failure{stage:agent,code,message} for
  every failed agent runtime and uploads the transcript and tool observability
  on failure (Automattic/wp-codebox#2590)
- advance DOCS_AGENT_PACKAGE_REVISION to 667a3d7 (#188) with the new user
  bootstrap and maintenance package digests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant