Surviving the Upgrade

Nobody plans for the upgrade that breaks things. You plan for the upgrade that improves things — new features, better performance, security patches you’ve been waiting for. The changelog reads like a gift list. You back up the config files, run the installer, rebuild the native modules, restart the gateway. Everything looks green.

Then the first cron job fires, and you discover that the execution model changed underneath you.

This isn’t hypothetical. This is what happened to our agent infrastructure over three days this week when we upgraded from OpenClaw 3.31 through 4.1 to 4.2. What follows isn’t a postmortem in the traditional sense — nothing caught fire, no data was lost, no customers were affected. But for most of those three days, a system that had been autonomously managing email, publishing articles, monitoring GitHub issues, running security reviews, and maintaining a knowledge base was reduced to answering questions in a chat window. The automation layer was gone.

The interesting part isn’t that things broke. The interesting part is that we spent two and a half days fixing the wrong thing — and the actual fix took five minutes once we understood the real problem.

What Actually Broke

The 3.31 upgrade introduced a new security model for shell execution. Previously, the agent’s exec calls were implicitly trusted — if a command was in the agent’s toolset, it ran. After the upgrade, every execution routes through an approval system that matches commands against an allowlist of resolved binary paths.

On paper, this is a good change. You want execution gating for AI agents. The first symptom was immediate: every cron job started timing out. Approval requests stacked up with 30-minute windows, each timeout spawning a follow-up session that also hit the approval gate. The gateway drowned in stacked waits. The agent appeared “wedged” — alive but unable to do anything useful.

The Wrong Rabbit Hole

Here’s where the story gets honest.

We built an elaborate allowlist. Resolved binary paths using which. Discovered that patterns need path separators (/usr/bin/git, not just git). Learned that the exec-approvals.json file needs security, ask, and askFallback fields at both the agent level and the defaults level. Added wildcard patterns (/opt/homebrew/bin/*). Added a bare * catch-all.

Direct commands started working. Pipes worked. Logical operators worked. But shell redirects (2>/dev/null) and command substitution ($(...)) kept getting blocked.

We blamed a gateway bug — issue #58691, open since 3.13, where security: "full" and ask: "off" are reportedly ignored for shell-wrapped commands. We built a library of twelve wrapper scripts to encapsulate shell syntax behind single-binary calls that could pass the allowlist. We updated every cron prompt to use these scripts. We added <exec-rules> blocks to every cron job prohibiting redirects in inline commands. This is the kind of diagnostic trap that catches experienced engineers. The symptoms matched the known bug perfectly. The workaround worked. We had no reason to question the diagnosis — until we did.

Meanwhile, a “timeout regression” was killing cron jobs at exactly 61.5 seconds regardless of configured timeout values. We attributed this to issue #59678 (a known 4.1 bug). It turned out to be a completely different problem: 4.1 introduced a security violation on custom PATH environment variables in exec calls, and the cron jobs were silently passing env: {PATH: ...}. The session wasn’t timing out — it was being killed for a security violation at roughly the same wall-clock time.

Two bugs. Two misdiagnoses. Two elaborate workarounds for problems that weren’t what we thought they were.

The Five-Minute Fix

On day three, after upgrading to 4.2, we ran one more test — and discovered the actual root cause.

OpenClaw enforces exec security through two configuration layers: openclaw.json (the agent config) and exec-approvals.json (the approval policy). The system enforces the stricter of the two. This is documented nowhere prominently.

Our config:

• openclaw.json: tools.exec: { ask: "on-miss", security: "allowlist" }
• exec-approvals.json: security: "full", ask: "off"

The system saw allowlist vs. full and enforced the stricter setting — allowlist. Every redirect and subshell failed the allowlist match and got gated, regardless of the security: "full" setting in the other file.

The fix:

// openclaw.json
"exec": { "ask": "off", "security": "full" }

// exec-approvals.json
"defaults": { "security": "full", "ask": "off" }
"agents": { "main": { "security": "full", "ask": "off" } }

Both files agree. Both say “full access, no approval needed.” Once aligned, every shell syntax pattern passed: $(), 2>/dev/null, backticks, brace groups, compound commands. The script library workaround was unnecessary. The exec-rules blocks were unnecessary. We had spent two days building infrastructure to route around a config mismatch.

The Taxonomy of Agent Infrastructure Failures

Despite the misdiagnosis, the experience maps cleanly to patterns that anyone running agent infrastructure should internalize:

Silent Behavioral Changes

The most dangerous class. The upgrade didn’t remove any capabilities — it changed how they were invoked. A command that worked yesterday returns an approval prompt today. The agent doesn’t crash. It waits. And waits. And times out. The logs show a timeout error, not the root cause.

The Stricter-Wins Trap

When multiple configuration layers control the same behavior, the system enforces the most restrictive setting. This means you can set security: "full" in one place and security: "allowlist" in another, and the allowlist wins — even if you intended the full-access setting. The failure is silent: no error message says “your settings disagree.” The stricter policy just takes effect.

Compounding Failures

A cron job hits an approval gate. The gate has a 30-minute timeout. After timeout, the gateway spawns a follow-up session to handle the failure — which also hits an approval gate. Which also times out. Meanwhile, the approval queue fills with orphaned requests. Each failure generates more failures.

Configuration Drift Under Upgrade

Our cron job payloads contained inline shell commands with redirects and subshells — patterns that were perfectly safe before the upgrade. The configuration was correct for the previous version. Nobody changed it. It just stopped working.

What the Script Library Got Right (and Wrong)

The twelve wrapper scripts we built before finding the root cause are still in production. Not because they’re needed to bypass approval gates — they aren’t, now that the config is aligned. They survived because they’re genuinely useful for a different reason: encapsulating multi-step operations behind clean interfaces.

A cron job that needs to check git status, conditionally commit, push, trigger a Cloudflare deploy, update a search index, and post to Slack — that’s better as a script than as inline commands in an LLM prompt, regardless of approval gates. The script has error handling (set -euo pipefail), structured output, and a comment header explaining what it does and why.

The design rules that emerged are worth preserving:

1. One script per task. Composability comes from calling multiple scripts in sequence.
2. Structured output. NO_CHANGES, PUSHED, FAIL: reason. The agent parses status markers, humans read logs.
3. Arguments for variable parts. git-worktree-setup.sh 42 handles all the 2>/dev/null internally. The issue number is the only thing that changes.
4. Self-documenting headers. Six months from now, the comments explain why these scripts exist.

The lesson: the scripts were the wrong solution to the approval problem, but they were the right solution to the complexity problem. Sometimes a misdiagnosis leads to a useful treatment anyway.

The Other Migrations

The upgrade also forced two changes we’d been deferring:

Embedding Model Switch

Our memory system (mem0) used mxbai-embed-large for embeddings. This model has a hard 512-token context limit, and Ollama has a bug where truncation fails for mixed Unicode text — em-dashes, smart quotes, emoji. Common in Slack messages. We’d been seeing intermittent “input length exceeds context length” errors for weeks.

We switched to nomic-embed-text (8,192 token context, 768 dimensions vs. 1,024). The catch: the vector store is SQLite-backed, not truly in-memory. The old 1,024-dimension vectors couldn’t serve 768-dimension queries. We wrote a re-indexing script that read all 1,222 active memories from the history database, embedded each one with the new model, and wrote the results back. Twenty-one stale memories (ephemeral inbox statuses, upgrade-day noise) were groomed out in the process. Zero failures.

Secrets Migration

An adversarial security review of the observer agent expansion plan revealed that openclaw.json contained seven API keys and tokens in plaintext — Slack tokens, the Anthropic API key, gateway auth, Notion, Google search. If any security boundary failed, that file was the crown jewels.

OpenClaw 4.2 has a proper secrets management system with SecretRef objects. Five core secrets moved to the keysfile provider. Two plugin secrets that don’t support SecretRef moved to env-template substitution ("apiKey": "${ENV_VAR}") with values in a chmod-600 .env file. The weekly security review cron was updated with a new secrets audit section.

Zero plaintext secrets in configuration files. The crown jewels file no longer contains any crowns.

Gall’s Law in the Machine Room

Our agent infrastructure survived this because it evolved from simple systems that worked. The first version had three cron jobs and basic Slack integration. Each new capability was added after the previous one proved stable. When the upgrade broke things, we had twelve moving parts instead of fifty — each one understood, documented, and individually recoverable.

John Gall’s observation that complex systems evolve from simple working systems isn’t just a design philosophy. It’s an operational survival strategy:

• Each cron job is independent. When seven broke, five kept running. Email checking self-healed within hours. The cairns issue monitor survived because its exec patterns happened to avoid the broken code path.

• The diagnostic was tractable. With twelve discrete jobs, we could test each failure mode independently. Direct commands? Pass. Pipes? Pass. Redirects? Fail. The fault tree had a clear shape.

• Recovery was parallel. While the config root cause was being chased, cron prompts were being updated, scripts written, and secondary issues (stale model IDs, orphaned jobs, broken delivery configs) fixed independently.

What the Industry Literature Gets Right (and Wrong)

The research on production AI agent failures describes many of the patterns we experienced: non-deterministic behavior, compounding failures, the gap between demo and production reliability. But most of the literature focuses on model reliability — hallucinations, drift, inconsistent outputs.

Our failure had nothing to do with model quality. The model was fine. Every LLM call returned coherent, correct responses. The failure was entirely in the infrastructure layer — the execution gateway, the configuration layering, the timeout semantics. The agent was intelligent and helpless.

This suggests a blind spot in the current discourse: the reliability of the scaffolding matters as much as the reliability of the model. An agent running a perfect model on a broken executor is no better than a broken model on a perfect executor. Worse, actually — because a broken model fails visibly, while a broken executor fails by waiting. There’s a philosophical question about whether an agent should need to know about implementation details of its execution layer. In an ideal world, no. In production, the agent that doesn’t know these details is the agent that spends 30 minutes waiting for an approval that never comes.

The Uncomfortable Takeaway

Here is the thing nobody wants to hear: your agent infrastructure will break. Not might. Will. The framework will ship a regression. The configuration layers will disagree silently. The version manager will resolve the wrong binary. And you will spend two days fixing the wrong thing before finding the five-minute fix.

The question isn’t whether you’ll have a week like this. The question is whether you’ve built systems that let you recover — and whether you have the discipline to keep questioning your diagnosis even when the symptoms match a known bug.

The patterns that make the difference aren’t glamorous:

1. Independent failure domains — each automation task should be able to break without cascading into others.
2. Configuration as a single source of truth — when two files control the same behavior, they will disagree, and you will spend days debugging the disagreement. Consolidate or document the precedence rules.
3. Script libraries for complexity management — not as a workaround for broken tools, but as a clean interface between the agent and the shell. The agent picks the script; the script handles the syntax.
4. Explicit constraint documentation — tell the agent about infrastructure limitations in its prompt. Trial and error costs tokens, time, and sometimes data.
5. Evolutionary complexity — add capabilities one at a time, prove each one works, then add the next. When the ground shifts, you know exactly what's standing on it.

None of this is cutting-edge. That’s the point. The cutting-edge stuff is the model, the reasoning, the memory, the multi-agent coordination. The stuff that keeps it all running is a configuration file that agrees with another configuration file.

Sometimes the most important engineering is the boring kind.

• When was the last time you spent days on a misdiagnosis? What finally made you question your assumptions?
• How do you handle configuration layering in your infrastructure? Do you know what happens when layers disagree?
• If you run AI agents in production, what's your version of the "five-minute fix" — the simple thing that was hiding behind a complex symptom?

1. 5 Production Scaling Challenges for Agentic AI in 2026 — Machine Learning Mastery's overview of operational hurdles, including the observation that agentic drift accumulates risk before overt failure.
2. Agentic AI Systems Don't Fail Suddenly — They Drift Over Time — CIO article on behavioral drift in production AI agents, relevant to the silent behavioral change pattern we experienced.
3. Taking Agents to Production is Non-Trivial — Arsanjani's candid assessment of the gap between agent demos and production reliability.
4. Gall's Law — The Personal MBA's explanation of why complex systems that work always evolve from simple systems that worked.
5. Agentic AI Infrastructure in Practice — Google Research paper on production hurdles for AI agents, with emphasis on the gap between capabilities in demo environments and production-grade rigor.