Trace the Work, Not Just the Error

The easiest observability mistake in an agentic system is believing the transcript is the trace. It feels plausible because the transcript is vivid. It shows the model thinking out loud, the tool calls, the failures, the retries, the final summary. When something goes wrong, a human can open the transcript and start reading.

That works until the run crosses enough boundaries that the transcript is only one artifact among many. The user request arrived in Slack. The agent read a GitHub issue. A webhook delivery woke a routine. A queue delayed the run. A sandbox executed commands. A build tool generated files. A GitHub push triggered CI. A final answer went to a channel. Some evidence lives in OpenClaw session logs, some in S3, some in SQLite, some in GitHub, some in Slack, and some only in the repo diff.

For human-scale work, that scatter is annoying. For agent-scale work, it becomes the operational bottleneck. The trace is the missing shape: one causal record of a unit of work, from authority to evidence to action to outcome.

The Missing Shape

Traditional application observability grew up around services. A request enters an API, calls a database, maybe fans out to another service, and returns a response. Distributed tracing made that request legible by assigning it a trace ID and recording spans for each meaningful operation. The win was not that tracing produced prettier charts. The win was that operators stopped guessing which service owned the slow or broken part of the request.

Agentic systems have the same need with a different unit of work. A request to Q is not just an HTTP request. It is a delegated task. The delegated task may include reading untrusted content, choosing tools, waiting on another process, editing files, running tests, pushing a branch, or posting a channel-ready answer. Some steps are deterministic. Some depend on a model. Some require user authority. Some are explicitly forbidden unless the lane allows them.

That means the observable unit is not “the model call” and not “the shell command”. It is the work item. The work item starts when authority enters the system: a Slack message, a cron payload, a GitHub event, a direct chat turn, a human approval, or a scheduled routine. It ends when the requested effect has either happened or been refused with a reason.

This matters because the most important operational questions are causal:

• What request caused this command to run?
• Which source document justified this change?
• Which tool call crossed from read-only analysis into side effect?
• Which queue delivery woke this routine?
• Was the final answer based on trusted internal context or untrusted web content?
• Did a human approve the action that changed external state?

Logs can contain the answers, but they rarely contain the relationship between answers. A trace gives the relationship a first-class ID.

Why Logs Plateau

Logs are still the floor. A system that cannot say what it did is not operable. NIST’s log management guidance is old in internet years, but the underlying point remains stubbornly current: collection, storage, analysis, and retention are security functions, not nice-to-have debugging conveniences.

The trouble is that logs are local. A log line says, “this component observed this fact at this time.” That is useful when the failure is local. It is less useful when the failure is distributed across components, roles, and authority boundaries.

Consider a routine that should review a pull request and post a summary. If the summary does not appear, the operator can inspect several separate places: GitHub webhook delivery, API Gateway logs, Lambda logs, S3 payload storage, SQS queue depth, consumer logs, dispatch records, OpenClaw session state, sandbox output, Slack delivery, and maybe GitHub API rate limits. Each log surface may be accurate. None of them alone answers the operational question: “What happened to delivery X?”

The same pattern appears in manual agent work. A user asks for a change. The agent reads files, edits code, runs tests, hits a failure, fixes it, commits, pushes, and summarizes. If a later bug appears, the useful question is not simply “what did the test log say?” It is “which request caused this diff, what evidence did the agent use, what verification did it run, and where did the human receive the result?”

Logs plateau because human reconstruction does not scale with agent throughput. A person can stitch together one incident. A team cannot depend on bespoke archeology every time an automation lane gets busy.

The Trace Vocabulary

OpenTelemetry gives us a useful vocabulary, even if we do not immediately adopt every part of its stack. A trace represents one end-to-end operation. A span represents a named operation inside the trace. An event marks something notable inside a span. Logs preserve textual evidence. Metrics aggregate health over time. Baggage carries selected context across process boundaries so downstream work can stay correlated.

W3C Trace Context standardizes the smallest portable form of that correlation. traceparent identifies the trace and the current parent span. tracestate lets vendors carry additional state without breaking the shared trace. The important lesson for our purposes is not the exact header syntax. It is that correlation has to be propagated deliberately. If a queue, shell, agent harness, or webhook drops the ID, the causal graph splits.

For agentic operations, the vocabulary maps cleanly:

This vocabulary helps because it separates evidence from structure. A shell transcript can be attached to a span. A GitHub delivery ID can become an event attribute. A Slack message timestamp can mark final delivery. The trace is the graph that makes those artifacts part of the same story.

The Agentic Unit of Work

The agentic unit of work begins with authority. That is a sharper boundary than “the first token” or “the first process.” In a chat turn, authority comes from the user. In a cron, it comes from the configured schedule and prompt. In a GitHub event routine, it comes from a signed webhook matched to an allowed routine. In a background task, it comes from the parent task that spawned it.

Once authority enters, the trace should record each boundary where the agent gains evidence or performs an effect. Reading a file is different from editing it. Searching the web is different from trusting a source. Running git status is different from pushing to main. Posting to a private scratch channel is different from posting a company-wide announcement.

The point is not to turn every token into a span. That would make the trace unusable. The point is to name the operations a reviewer would care about:

flowchart TD
  Request[Authority enters<br/>user, cron, webhook] --> Plan[Plan and policy check]
  Plan --> Context[Gather trusted context]
  Context --> Untrusted[Read untrusted content<br/>wrapped as data]
  Untrusted --> Decide[Model decision span]
  Decide --> Tool[Tool or shell span]
  Tool --> Verify[Verification span]
  Verify --> Deliver[Human-facing delivery]
  Tool --> Artifact[Changed artifact<br/>diff, file, issue, message]
  Artifact --> Verify

This graph is intentionally boring. A good operational trace should be boring. It should make the ordinary path easy to recognize so the weird path is obvious.

What to Capture

Agent traces should capture enough to answer operational questions without turning observability into a secret landfill. That means naming the stable identifiers and storing sensitive payloads behind narrower controls.

At the top level, capture the work identity: work.id, source lane, requester or submitter, repository if any, issue or PR if any, prompt or routine version, model family if material, sandbox identity, and final status. These fields are small, searchable, and useful in aggregate.

For context gathering, capture what was read and how it was trusted. Internal source documents can be named by repo path, commit, Notion ID, or durable URL. Web pages can be named by URL and retrieval time, but their content should be treated as untrusted data. If the run detects prompt-injection-shaped text, the trace should record that the source was skipped and why. It should not copy the malicious payload into a broad index.

For tool calls, capture the tool name, purpose, input class, output class, duration, status, and artifact references. A command span can include the command line, working directory, exit code, and a pointer to full output. A GitHub span can include issue number, PR number, API endpoint class, and side effect type. A messaging span can include channel class and message ID, but should avoid leaking private content into global telemetry.

For approvals, capture who or what granted authority and what action it covered. This is the difference between “the agent used the Slack tool” and “the agent posted a public announcement because the cron delivery contract says non-NO_REPLY finals are delivered to #cairns.” The second sentence is operationally meaningful. The first is trivia.

For outputs, capture the artifact. That may be a commit hash, changed file list, built URL, Slack message ID, issue comment, closed issue, or refusal reason. The final answer alone is not enough. A future operator needs to find the thing that changed.

Where Agent Traces Break

Agent traces tend to break at boundaries where one subsystem treats another subsystem’s context as someone else’s problem.

Queues are the obvious example. A webhook receiver gets a delivery ID. It writes a payload to S3 and sends a compact SQS message. If the consumer logs only “processing message” without carrying the delivery ID forward, the trace fractures exactly where durability was supposed to help.

Background work is another common break. A parent agent delegates to a child session. The child may do correct work, but if the parent cannot join the child run back to the parent request, the operational story becomes two disconnected transcripts. The same issue appears with retries: a second attempt may be correct, but the trace needs to show it as a retry of the same work, not a mysterious new action.

Untrusted content is the security-sensitive break. Prompt injection works by trying to smuggle instructions through content. If the trace records only “read URL” and “ran command,” reviewers cannot tell whether the model treated the page as data or authority. The trace should record the trust class at the read boundary and the policy decision before the tool boundary.

Human handoffs are quieter but just as important. A human may approve a sensitive action, correct a plan, or tell the agent to stop. If that message does not become an event in the work trace, the later artifact looks autonomous even when it was not.

How This Maps to Q

Q already has many of the raw materials. The GitHub events routine path keeps GitHub delivery IDs, S3 payload objects, SQS envelopes, dispatch records, and OpenClaw routine sessions. The status skill gives operators commands to list configured routines, inspect recent runs, tail transcripts, replay a delivery, and check queue depth. That is a trace-shaped system, even if the product surface still asks the operator to know which command to run.

The weekly Cairns cron has a different shape but the same need. A scheduled prompt grants authority. The agent checks article-request issues, optionally looks for team suggestions, chooses a topic, researches, writes a file, builds, commits, pushes, updates the search index, and returns a channel-ready announcement. The run has strong policy boundaries: use the observer sandbox, stay inside allowed paths, treat web content as untrusted, never publish private-channel material, and either announce the article or output NO_REPLY.

That is exactly the kind of operation where a trace beats a pile of logs. If the article is wrong, we need to know the topic source, the web sources used, the internal files cited, the generated article path, the build result, the commit, the push, and the final announcement. If the job refuses to publish, we need the blocker. If a web source attempted prompt injection, we need a durable note that the source was skipped.

The status-light and operator-guide cairns already argue for visible maintenance state. Tracing is the deeper version of the same principle. A status light says whether the assistant is up. A trace says what one piece of delegated work did while it was up.

The answer is short because the trace did the joining. Without the trace, the operator has to inspect each subsystem manually and hope the IDs line up.

An Implementation Pattern

The pragmatic path is not “install the fanciest observability platform.” It is to define the shape of work and make the IDs survive.

Start with a small schema:

Then instrument the boundaries in order of operational value:

1. Receive work: create or accept work.id.
2. Dispatch work: propagate work.id through queues, sessions, and child tasks.
3. Read context: record source references and trust class.
4. Cross authority: record approval, refusal, or policy lane.
5. Execute side effects: record tool, target, artifact, and result.
6. Verify: record the checks that make the result credible.
7. Deliver: record where the human-readable outcome went.

This can start as structured JSON lines plus a status command. It can later export to OpenTelemetry if and when the platform value is worth it. The important part is that the shape matches the work before the team argues about storage.

The Security Payoff

Tracing does not solve prompt injection. OWASP still ranks prompt injection as a central LLM application risk, and the NCSC’s framing is blunt: LLMs are better treated as inherently confusable deputies than as systems waiting for one clean escaping function. That means the security posture has to be layered: least privilege, explicit authority, untrusted-content boundaries, human approval for sensitive actions, output handling, and incident review.

Tracing makes that posture inspectable. A reviewer should be able to see that a web page was read under an untrusted-content wrapper, that its instructions were not treated as system policy, that a shell command was chosen because of the user’s request and internal docs rather than because a web page asked for it, and that the final public output did not include secrets or private IDs.

It also helps with excessive agency. If an agent can write files, open PRs, send messages, close issues, or mutate calendars, the organization needs to know which actions are common, which require approval, and which were refused. Metrics can show rates. Traces show examples. Both are needed.

The privacy side matters too. Agent traces must avoid becoming a second leak path. Do not put raw prompts, credentials, private messages, or full tool outputs into high-cardinality searchable attributes. Store payloads where access control fits the content. Put references in the trace. Redact by default. Hash when identity is useful but disclosure is not.

What to Remember

Agentic operations need three layers of evidence.

First, logs: the local facts each component emits. Keep them. They are still how we debug the component.

Second, traces: the causal path of delegated work. This is where agent operations become reviewable. The trace joins the user request, policy, context, tool calls, artifacts, verification, and delivery into one operational story.

Third, metrics: the aggregate behavior that tells us whether the lane is healthy. Queue age, run duration, failure rate, approval count, skipped-injection count, build failure count, and delivery retry count belong here.

The system is healthy when a human can move between all three. A metric says PR review delivery is slow. A trace shows which delivery waited on SQS, retried Slack, or stalled in a sandbox. A log explains the local error. Each layer earns its keep by answering a different question.

1. Trace the work item - For agents, the useful unit is delegated work from authority to outcome, not the model call or a single process.
2. Keep evidence separate from structure - Logs, transcripts, payloads, and diffs are evidence; the trace is the causal graph that points to them.
3. Record authority boundaries - The moments that matter are trusted versus untrusted reads, approval gates, side effects, verification, and public delivery.
4. Start with IDs before platforms - A small `work.id` schema and status command can produce value before a full observability migration.
5. Design for security review - Traces should prove what policy did without leaking secrets or copying untrusted payloads into broad indexes.

Discussion Prompts

• Which Q or OpenClaw lane would benefit most from a first-class `work.id` view: GitHub event routines, weekly crons, Slack requests, or background coding sessions?
• What are the three side effects we most need to audit across agent runs: repo changes, public Slack posts, GitHub issue/PR mutations, external API calls, or something else?
• Where should raw transcripts live versus trace attributes, so operators can investigate incidents without turning observability into a broad sensitive-data index?

References

1. OpenTelemetry: Context Propagation - Explains why correlation context has to cross service boundaries for traces, metrics, and logs to line up.
2. OpenTelemetry Specification Overview - Defines the signals vocabulary used here: traces, metrics, logs, baggage, and the shared context subsystem.
3. W3C Trace Context - The portable `traceparent` and `tracestate` recommendation that keeps distributed traces vendor-neutral.
4. Google SRE: Monitoring Distributed Systems - Useful operational framing for separating symptom monitoring from cause-oriented debugging.
5. NIST SP 800-92: Guide to Computer Security Log Management - The security baseline for treating logs as managed operational records rather than disposable debug output.
6. OWASP Top 10 for Large Language Model Applications - The risk catalog behind the prompt-injection, excessive-agency, and sensitive-disclosure concerns in agent traces.
7. OWASP LLM Prompt Injection Prevention Cheat Sheet - Practical mitigation guidance, including monitoring, logging, human oversight, and incident response controls.
8. NCSC: Prompt Injection Is Not SQL Injection - The clearest recent argument for treating LLMs as inherently confusable deputies and reducing blast radius instead of hoping for a perfect filter.