The ECO Lifecycle

An ECO Is Born

Everything starts with a NOC operator and a geometry. Someone detects an outage — maybe an alarm fires, maybe a customer calls, maybe a monitoring system flags a loss-of-light condition — and the operator creates an Emergency Callout Order in Strike’s web UI.

The creation form captures surprisingly little data, and that’s deliberate. In an emergency, you want the fastest path from “something is broken” to “someone is on their way.” The ECO captures:

• Location — a GeoJSON Point or LineString stored as PostGIS geometry. This is the incident site, and it drives everything that follows: which contractor gets paged, which Render project receives the task, which pager list applies.
• Job type — CMR (Claims Management Resource), MFR (Maintenance Field Request), or FNOL (First Notice of Loss). These categories determine downstream billing and reporting.
• Provider reference — the initiating NOC reference, often a CMR, MFR, or FNOL number, so Strike can reconcile the ECO back to the source record that started the callout.
• KCCI job ID — a work-order handle such as E2-0000-2025 during early rollout, where the prefix identifies the customer or workstream, the middle number increments the job, and the year keeps the sequence readable in operations and billing conversations.
• Description — free text from the operator describing what’s known about the outage.
• Optional attachments — photos, PDFs, screenshots, or field notes that are useful as evidence but should not block creation when the emergency needs to move.
• Tenant context — the NOC tenant creating the ECO and the OSP tenant who will work it. Every ECO exists at the intersection of these two entities.

The practical intake contract is slightly more precise than the UI may look. A location may need to accept multiple coordinate pairs when the incident covers a span rather than a single point; the early Arkansas and Oklahoma rollout notes bounded coordinates to the operating territory rather than accepting arbitrary map input. CMR, MFR, and FNOL are provider-side initiating references, while the KCCI job ID is the contractor-facing work-order handle. Keeping those identities separate, alongside the description and optional visual evidence, connects the NOC’s source record, Strike’s ECO aggregate, Render’s task model, and the later invoice or as-built package.

The moment that event hits the outbox, the system wakes up. The transactional outbox publisher — polling every 100ms — picks up the event and fans it out to three subscribers simultaneously: the ECO projection (which creates the read model for the UI), the WebSocket publisher (which notifies connected browsers), and the Render integration processor (which starts the external integration chain).

The ECO’s initial status is OPEN. Its render integration status is PENDING. Neither will stay that way for long.

Reaching the Field: Render Integration

The Render integration processor is the first process manager to act on a new ECO. It’s a single-shot workflow: receive the ECOCreatedEvent, create a task in the OSP’s Render Networks tenant, and report success or failure. Render Networks is the field workforce management platform that OSP contractors use to manage their crews. Strike doesn’t replace Render — it creates tasks in Render and polls for status changes. Field technicians never interact with Strike directly.

The processor looks up the OSP tenant’s Render instance from the database — each OSP has its own API credentials, project name, and default task type. It creates a ClientFactory that handles OAuth2 token management with singleflight to prevent token stampedes when multiple ECOs are being processed simultaneously.

The task creation request carries the ECO’s geometry, description, and the piece of metadata that defines membership: an exact, case-sensitive General label holding the tenant-formatted ECO job ID (e.g., E2-00524-2026). A Render task belongs to this ECO only when it carries that label. The payload also sets subsector to the literal ECO — Render requires a categorization value, and a neutral constant keeps Strike’s tasks from colliding with however the OSP organizes its own work — plus task_metadata.eco_id, which is provenance for the initial task and not membership.

Render’s own cloning preserves neither labels nor task metadata, so the grouping is a field procedure as much as a data model: after cloning an investigation or creating a follow-on independently, the responder adds the canonical ECO label in Render. An unlabeled follow-on is deliberately invisible to Strike rather than silently guessed at, which is why the drill briefing, the watcher, and the close-out label check are treated as required controls.

If the Render API call succeeds, the processor emits an UpdateECORenderStatusCommand with status TASKED, carrying the project task ID back from Render. If it fails — network error, authentication failure, missing Render instance — the status becomes FAILED with an error message, and the NOC sees an error state prompting escalation.

Pager Dispatch: Getting Humans Moving

Creating a task in Render puts work into the system. But tasks don’t move themselves — someone needs to answer the phone. The pager process manager is the orchestrator that turns a TASKED ECO into an acknowledged field dispatch.

When the ECO’s render integration status changes to TASKED, the pager processor wakes up. Its first job is geographic: using the ECO’s PostGIS geometry, it resolves the pager region — a polygon that maps to a contact list of OSP supervisors responsible for that area. Each OSP tenant maintains its own regional pager lists, so the same physical location can route to different people depending on which contractor is assigned.

The processor builds a contact list, generates a message ("NOC: ECO 524 - Fiber cut on Main St"), and fires a StartPagerRunCommand to create the PagerRun aggregate. Then it calls the Twilio API to initiate the first phone call.

Here’s where the design gets interesting. The pager process manager doesn’t hold state in memory. It writes a pager_process_state row to PostgreSQL with a deadline_at timestamp, then returns. A background worker polls every 5 seconds for overdue deadlines. When a Twilio webhook arrives (the contractor answered, declined, or didn’t pick up), the webhook handler updates the process state and extends the deadline. This is a deliberate architectural choice. Traditional saga frameworks keep process state in memory or in framework-managed storage. Strike uses plain PostgreSQL rows because when paging fails at 3 AM, you want SELECT * FROM pager_process_state WHERE eco_id = '...', not framework log archaeology. The debuggability wins justified the extra implementation work.

The full pager dispatch sequence looks like this:

Each webhook event — CALLING, ACCEPTED, DECLINED, FAILED — is recorded in the PagerRun aggregate’s timeline, building a complete record of every attempt. The process state uses optimistic locking (a version column) to handle the race between webhook delivery and the background timeout worker. If both try to update simultaneously, one gets a version conflict and retries.

When the pager run completes with a DISPATCHED outcome, the dispatch processor updates the ECO’s render integration status to DISPATCHED. The NOC operator, watching the live status, sees the ECO transition from “page sent” to “contractor dispatched.” They don’t see the intermediate TASKED status — that’s internal plumbing.

Watching the Work: Render Polling

Once a contractor is dispatched, Strike’s job shifts from orchestration to observation. The Render polling service runs a background loop, applying each Render instance’s configured interval and kill switch before querying pollable ECOs.

The polling service queries eco_views for ECOs in a pollable state — dispatched but not yet hard-completed. For each ECO, it fetches the ECO’s labeled tasks from the Render API with the exact label_name filter, paging until the result is exhausted — one page covers an ECO in practice — using the same ClientFactory and per-tenant credentials as the integration processor.

Change detection uses fingerprinting. Each task’s relevant fields are hashed into an MD5 fingerprint. The service caches fingerprints in memory (sync.Map) and only publishes change events when a fingerprint differs from the cached version. This prevents unchanged observations from generating event noise, but it does not prevent the Render request: capacity remains one request per ECO per cycle.

The polling service also detects late clones — when a field technician creates a new task from a completed investigation. It tracks task counts per ECO and publishes a late-clone event when the count increases, which can revert an ECO from COMPLETED back to IN_PROGRESS.

An immediate fetch also runs when an ECO becomes DISPATCHED, so a safe five-minute interval does not leave the detail screen empty for five minutes. GraphQL reads can make their own live requests too. These paths share one credential allowance but do not yet share one coordinator; Render Is a Shared Request Budget maps all of them and the durable cache between them. Late clones are a real operational pattern. A field tech completes the investigation, reports the damage, and then creates follow-on tasks for the actual repair work. The investigation might complete before the follow-on tasks even exist. Without the grace period and late-clone detection, Strike would prematurely close ECOs.

The service includes resilience features — exponential backoff on database errors, per-ECO locking to prevent concurrent read-modify-write races on fingerprint caches, and sync.Once guards for first-poll initialization. These aren’t premature optimization — they’re responses to production patterns where a single Render API hiccup shouldn’t cascade into missed updates for every ECO in the system.

Three Layers of Status

Strike maintains three distinct status models, and understanding why is essential to understanding the system’s design philosophy.

Layer 1: Event-sourced state (the truth). The ECO aggregate’s status — OPEN, IN_PROGRESS, COMPLETED — is derived by replaying its event stream. This is the authoritative record. When a StatusUpdatedEvent is applied, the aggregate validates the transition (you can’t go from COMPLETED back to OPEN without explicit intervention) and updates its internal state.

Layer 2: Render task status (external system state). Each task in Render has its own lifecycle: blueprinted → allocated → releasable → released → completed. These statuses reflect Render’s internal workflow — contractor assignment, crew dispatch, work execution. Strike polls these but doesn’t display them directly.

Layer 3: NOC display status (human-friendly view). This is what operators actually see. The display status maps Render’s six-state task lifecycle down to five human-meaningful states:

The render integration status — PENDING, TASKED, DISPATCHED, FAILED — lives alongside these as a fourth, internal-only status. It’s a backend gate confirming the Render handoff succeeded. Operators never see it. When integration succeeds, tasks simply appear. When it fails, they see a simplified error prompting escalation.

This separation costs something: there are now multiple status fields to maintain, mapping logic at the API boundary, and documentation to keep synchronized. The payoff is that each audience gets exactly the information they need, training new NOC staff doesn’t require explaining Render’s internals, and the system is decoupled from Render’s implementation details.

The Mock Server: A Simulation Engine

You can’t develop an emergency response system by waiting for emergencies. The mock server — a Go application in packages/mock/ — simulates both the Render Networks API and Twilio’s call infrastructure, providing a complete end-to-end development environment.

The mock server exposes three surfaces: a Render API mock at /render/* that simulates task creation and status progression, a Twilio API mock at /twilio/* that receives call requests and sends webhook callbacks, and a control dashboard at /dashboard for composing scenarios and watching execution in real time.

The dashboard composes test cases from five independent controls rather than one preset picker. Render and Pager are set separately, Speed multiplies every delay, Fault injects API errors, and Workflow selects the ECO orchestration template:

The default Render path progresses to completion in about five seconds. The default Pager path has the first contact accept in about six seconds. Speed can compress those delays for tight development loops or stretch them for watching timing-sensitive behavior; Fault can simulate rate limits, flaky responses, or a full outage without changing the application under test.

Each ECO’s mock state is tracked independently, keyed by ECO ID. The orchestration engine manages state transitions per ECO — you can have one ECO exercising succeeds plus accepts-first while another uses succeeds plus exhausted, simultaneously. State is in-memory (acceptable for a dev tool), protected by mutex, and visible through the dashboard’s polling UI. The mock server replaced a Python script for Render simulation and a Kotlin mock class for Twilio. Unifying them into a single Go service enabled cross-integration scenario orchestration — something that wasn’t possible when the mocks lived in separate languages and processes.

Docker Compose profiles make this practical for daily development. The principle is “start what you’re not touching” — if you’re working on the API, run just docker for-api and Docker starts PostgreSQL and the mock server. If you’re working on the frontend, just docker for-web starts everything except the frontend. The mock server’s port (8090) matches the old Python mock, so the API’s environment variables just work.

Completion and the Audit Trail

An ECO completes when every labeled task is done in Render — completed, approved, or integrated. But “complete” isn’t as simple as it sounds — the system needs to account for late-arriving work.

When the polling service detects that at least one labeled task exists and every labeled task is done, it marks the ECO as COMPLETED and starts a grace period (default: 24 hours). A task counts as done at completed, approved, or integrated — Render advances a finished task along that ladder, so requiring the literal completed once held ECOs open forever. During this period, polling continues. If a new task appears — a field technician cloned the investigation to create follow-on work — the ECO reverts to IN_PROGRESS and the NOC is notified. If a completed task is reopened, same thing.

This is why the ECO label, and not Render’s subsector, is the container. Every investigation, repair, cloned task, and follow-on task carries the same tenant-formatted job ID as an exact General label, and that label-scoped result is the only thing Strike counts. Strike does not ask whether the first task is complete; it asks whether the whole labeled work set has settled. Removing a label removes membership on the next poll; adding one can pull a task back in and reactivate an ECO during grace. The polling service translates Render’s field statuses into NOC-facing ECO state, keeps polling through the grace period after apparent completion, and only treats the ECO as operationally final after that window expires without new or reopened work.

After the grace period expires without new activity, the ECO reaches hard completion. Polling stops permanently. The ECO is finalized. Any changes after this point require manual intervention — the system won’t automatically reactivate a hard-completed ECO.

What remains after completion is the event stream — and this is where event sourcing pays its largest dividend. Every ECO carries a complete, immutable record of everything that happened:

1. ECOCreatedEvent — who created it, when, with what geometry and description
2. RenderIntegrationStatusUpdatedEvent — when the Render task was created, which instance, the project task ID
3. PagerRun events — every call attempt, every response, the timeline of human acknowledgment
4. StatusUpdatedEvent — each transition, with timestamps and actor IDs
5. Task change events — every field update detected by polling, including late clones

This isn’t a log file that rotates away or a database column that gets overwritten. It’s the data model itself. When a regulatory question arises — “who was notified about this outage, and how quickly did the response happen?” — the answer is a query against the event store, not a forensic reconstruction from scattered logs. The audit trail matters beyond compliance. Post-incident reviews can replay the exact sequence of events to identify bottlenecks — was the delay in pager acknowledgment? In field dispatch? In the completion grace period? The event stream is the single source of truth for “what actually happened.”

Putting It All Together

The full ECO lifecycle is a sequence of handoffs between process managers, each responsible for one phase of the workflow:

Each row in this timeline is backed by an event in the store. The process managers don’t communicate directly — they react to events and emit commands. The Render integration processor doesn’t know about the pager processor. The pager processor doesn’t know about the polling service. They’re connected by the event bus, and the event store is the shared record of what happened.

This is the design that lets a single button press — “Create ECO” — cascade into a multi-hour, multi-system, multi-human workflow, while every participant (human and machine) sees exactly the information they need, exactly when they need it.

1. An ECO starts as a CreateECOCommand carrying geometry, job type, description, and dual tenant context. The aggregate validates and emits a CreatedEvent.
2. The Render integration processor creates a task in the OSP's Render tenant, labeling it with the tenant-formatted ECO job ID to group all work for the ECO. Success moves the integration status to TASKED.
3. The pager process manager resolves contacts by region (PostGIS), calls via Twilio, and tracks state in a PostgreSQL table — not in memory. A background worker handles timeouts.
4. The polling service watches Render every 60 seconds, using fingerprint-based change detection to minimize noise. It handles late clones and completion grace periods.
5. Three status layers serve different audiences: event-sourced state for correctness, Render task status for integration tracking, and display status for NOC operators.
6. The mock server simulates both Render and Twilio with independent Render, Pager, Speed, Fault, and Workflow controls plus an embedded dashboard — enabling full lifecycle testing without external dependencies.
7. The event stream is the audit trail. Every state change, every pager attempt, every field update is an immutable fact in the event store — regulatory compliance as an architectural consequence.

• The grace period defaults to 24 hours. What operational patterns might justify making this shorter or longer? What signals could the system use to set it dynamically?
• The pager process manager uses PostgreSQL for state instead of a saga framework. At what scale or complexity would this trade-off start to hurt, and what would migration to a framework look like?
• The three-layer status model adds mapping complexity. Could a simpler two-layer model (internal + display) work, or does the Render integration layer earn its existence?

1. When the Fiber Goes Dark — The domain context that explains why ECOs exist and what OSP emergency response looks like.
2. The Shape of the System — Mono-repo structure, package boundaries, and the request flow that this cairn builds on.
3. Events All the Way Down — The event sourcing architecture that makes the ECO lifecycle possible, including the Watermill migration.
4. Render Networks — Field workforce management platform. External service that OSP contractors use for task management and crew dispatch.
5. Twilio — Communications API used for pager dispatch. Strike uses Twilio Studio flows for the call sequence and webhooks for status callbacks.