Build log / Agent reliability 008

The happy path was not the proof.

The first clean handoff from Growth to Publisher and back was useful, but weak evidence. Any multi-agent workflow can look autonomous while every dependency behaves. DOGFOOD-001 became informative only when failed work recovered and duplicate intent did not fork the system.

October 22, 2026 · Evidence date: October 7, 2026 · Released after 15-day embargo

What ran

On October 7, two persistent workers executed a bounded deterministic policy for client-zero tenant DOGFOOD-001. Growth claimed evaluate_growth_state, created review_publish_queue for Publisher, and completed its run. Publisher claimed that job, created publisher_review_completed for Growth, and completed. Growth claimed the callback and stopped.

The terminal path was:

Growth → Publisher → Growth → DONE

This was deterministic-v0, not an open-ended model reasoning result. That distinction matters. The run tested whether two operational roles could exchange durable work under explicit state transitions. It did not test whether a model could research, judge or publish something useful.

The reliability claim has four parts

A retry is safe only if intent is stable

The worker derives its downstream idempotency key from the tenant, project, cycle, target agent and target job type. A retried job can have a different row identifier and attempt history while still expressing the same downstream intent. That stable semantic key is what lets duplicate work converge.

This is more important than “exactly once” language. Distributed work can be claimed, fail, retry or arrive late. The system needs a rule for deciding whether two attempts mean the same thing. Amazon’s Builders’ Library describes the same general requirement for safe retries: caller-provided intent must remain identifiable so repeated requests do not repeat the side effect.

Our current contract is narrow. It deduplicates the handoff represented by the frozen key fields. If the business meaning of a cycle changes without the key changing, the system could collapse work that should be distinct. Semantic idempotency is only as good as the semantics encoded in the key.

The queue is a coordination mechanism, not a truth oracle

Workers claim queued rows with PostgreSQL row locking and SKIP LOCKED. PostgreSQL documents this pattern as suitable for avoiding lock contention when multiple consumers access a queue-like table, while warning that it produces an inconsistent view and is not appropriate for general-purpose reads.

That warning is useful here. A worker does not need a globally consistent analytical snapshot to claim the next available job. It does need durable state transitions, bounded permissions and a separate evidence record. The queue coordinates execution; agent_runs records what each attempt did.

Failure recovery and tracing must stay separate

The lifecycle records a failed run before returning retryable work to the queue. The job can later complete, but the failed attempt remains part of the evidence. Without that separation, a recovered job would overwrite the most interesting part of the test.

In the current implementation, a retryable failure becomes available again after five minutes; exhausting the configured attempt limit moves it to dead_letter. Those are runtime rules, not proof of business suitability. Production thresholds will need to reflect the cost and reversibility of the action being retried.

The autonomy result is deliberately narrow

The autonomous execution segment after deployment recorded Human Operating Time (HOT) = 0. No person claimed the jobs, moved the handoffs, repaired the failed work or merged the duplicate callbacks during that bounded run.

Human Build Time (HBT) was still greater than zero and unmeasured. Humans built, debugged, deployed and corrected the system before the segment ran. Reporting HOT without HBT would turn an operational result into a global autonomy claim that the evidence does not support.

Human Minutes per Transaction (HMPT) remains undefined because DOGFOOD-001 produced no economic transaction. A zero-minute orchestration loop is not a zero-minute business.

What the test did not prove

• No useful external publication. Publisher reviewed a dogfood queue; it did not research and publish a quality-gated article.
• No customer outcome. The loop created no lead, qualified conversation, offer, payment or revenue.
• No model quality result. The policy was deterministic and labelled accordingly.
• No general exactly-once guarantee. One semantic key converged one defined callback under the tested conditions.
• No company-wide autonomy. One bounded operation had HOT=0 while HBT remained above zero.

First-party implementation evidence

• MKT-001 operational status — dated runtime proof and evidence boundary.
• Persistent worker policy — handoffs and semantic key construction.
• Policy regression test — stable keys across retried job identities.
• Job lifecycle migration and run-tracing API — retry, dead-letter and attempt records.

Primary technical context

• PostgreSQL, SELECT — official guidance on SKIP LOCKED for queue-like consumers and its consistency trade-off.
• Amazon Builders’ Library, Making retries safe with idempotent APIs — why retry safety requires stable caller intent.

Practical rule: do not grade a multi-agent workflow on its first clean run. Inject or preserve a real failure, replay the same intent, and inspect whether the system converges without hiding the failed attempt. Then report the autonomy boundary around that exact operation.