Render Is a Shared Request Budget

architectureoperationsintegrations

A rate limit sounds like a property of an API client. In Strike it is a property of a whole relationship: one OSP-owned Render connection, several callers, every active ECO attached to that connection, and one allowance they all spend.

That distinction changes the architecture conversation. The question is not merely whether the poller stays under two requests per second. It is whether every request path combined can remain inside 10,000 requests per day, recover sensibly from throttling, and still show operators useful field state.

This cairn reflects the request model documented on July 27, 2026, including the changes tracked in PRs #1290, #1291, #1292, and #1296. Items identified as planned remain future work.

The Credential Is the Budget Boundary

Render enforces its limits per API credential. Each render_instances row has its own credentials, so each row gets a separate request budget.

Limit	Value
Daily quota	10,000 requests/day
Rate limit	2 requests/second
Burst limit	5 concurrent requests
Page size	2,000 default and hard maximum

The daily quota is tighter than it first appears. Ten thousand requests spread across a day is one request every 8.64 seconds. Two formulas describe the current capacity model:

minimum safe interval (seconds) = requests per cycle × 8.64
maximum ECOs per instance       = interval seconds ÷ 8.64

Polling costs one request per ECO per cycle. A five-second interval consumes 17,280 requests per day for one ECO. At 60 seconds, one ECO consumes 1,440.

Interval	Requests/day per ECO	Approximate ECO ceiling
5 seconds	17,280	0.6
60 seconds	1,440	6
120 seconds	720	13
300 seconds	288	34
600 seconds	144	69

The Yelcot connection is configured at 60 seconds. That is reasonable for about six steady-state ECOs, not twelve. Twelve ECOs at that cadence consume 17,280 requests per day; they need at least 104 seconds, making 120 or 300 seconds the practical choices.

A bounded demo is different from steady state. Twelve ECOs polled every minute for three hours cost about 2,160 requests. The hazard is forgetting to restore the longer interval after the demo.

Four Paths Spend the Same Allowance

Four paths can contact Render. Three have meaningful ongoing volume; the admin probe is small but still spends the same credential allowance.

Caller	Trigger	Cadence	Concurrency cap
Background poller	Timer	Per-instance interval	3
Dispatch trigger	ECO becomes DISPATCHED	Once per dispatch event	None
GraphQL live fetch	UI read needing live data	Per page render/read	2
Admin task-types probe	Operator clicks refresh	Two requests per click	None

The poller cap of three plus the GraphQL cap of two exactly equals Render’s burst limit of five. The dispatch trigger is outside that arithmetic. Nothing coordinates these callers through one shared limiter yet.

This is why capacity cannot be inferred from the worker-pool setting alone. A pool constrains one path. The credential sees all of them.

The Background Cycle Is Linear in ECO Count

The background loop is the steady-state request source:

sequenceDiagram
    participant L as pollLoop
    participant S as Service
    participant DB as Postgres
    participant R as Render API
    participant C as render_tasks_cache

    L->>S: wait max(interval, backoff)
    Note over L,S: Backoff may only slow the loop,<br/>never speed it up
    S->>DB: FindDispatched()
    DB-->>S: TASKED + DISPATCHED ECOs<br/>with instance controls

    S->>S: dueECOs()
    Note over S: Drop disabled instances and<br/>instances not yet due

    alt nothing due
        S-->>L: no-op, keep backoff state
    else instances due
        loop per ECO, max 3 concurrent
            S->>R: GET /tasks?subsector=<job id>
            R-->>S: this ECO's tasks
            S->>S: fingerprint + exact filter
            S->>DB: update render_task_count
            S->>C: upsert seen / mark removed
            opt DISPATCHED ECO
                S->>S: publish detected changes
                S->>S: evaluate completion
            end
        end
        S-->>L: polled, reset backoff
    end

Filtering is per instance because the credential is the quota unit. For every due instance, Strike fetches each pollable ECO by its exact subsector, with the page size capped at 2,000. Exact server-side filtering keeps one ECO’s result on one page and avoids silent truncation.

TASKED ECOs are polled to fill the cache but skip change detection and completion. Their tasks may already appear complete before paging has dispatched anyone, so treating those observations as workflow completion would move the domain too early.

Fingerprinting reduces internal event noise. It does not reduce API volume: Strike must fetch the tasks before it can know whether their fingerprint changed.

Dispatch Buys Immediacy Outside the Cycle

The dispatch trigger exists because longer safe intervals create a visible delay. At a 300-second cadence, waiting for the background loop could leave a freshly dispatched ECO without task data for five minutes.

sequenceDiagram
    participant W as Watermill subscription
    participant S as Service
    participant DB as Postgres
    participant R as Render API
    participant C as render_tasks_cache

    W->>S: ECO status = DISPATCHED
    S->>DB: FindPollableByID()
    DB-->>S: keyed ECOView + FormattedJobID

    alt instance polling disabled
        S-->>W: skip
    else polling enabled
        S->>R: GET /tasks?subsector=<job id>
        Note over S,R: Bypasses interval and currently<br/>bypasses the worker pool
        R-->>S: this ECO's tasks
        S->>C: populate cache
    end

The per-instance kill switch applies here as well as in the timer loop. The trigger deliberately does not stamp the instance’s last-poll clock: it fetched one ECO, while that clock governs every ECO on the connection. Moving the clock would defer the whole background cycle and could starve sibling ECOs whenever dispatches keep arriving.

Routing this trigger through the shared limiter is the immediate coordination fix. It preserves the useful fast refresh without letting dispatch bursts exceed the connection’s concurrency allowance.

UI Reads Can Become a Second Poller

The list and detail screens have different request behavior:

sequenceDiagram
    participant U as Browser
    participant G as GraphQL resolver
    participant DB as Postgres
    participant C as render_tasks_cache
    participant R as Render API

    rect rgb(240, 248, 240)
    Note over U,R: ECO LIST — renderTaskCount
    U->>G: ecos { renderTaskCount }
    G->>DB: read eco_views.render_task_count
    alt count present
        DB-->>G: count, no Render call
    else count missing
        G->>R: live fetch, max 2 concurrent
        Note over G,R: On failure, serve cache and skip<br/>write-through so fallback remains available
        G->>DB: write through count
    end
    end

    rect rgb(248, 244, 236)
    Note over U,R: ECO DETAIL — renderTasks
    U->>G: eco(id) { renderTasks }
    G->>R: live subsector fetch
    alt success
        R-->>G: tasks, Stale = false
    else failure
        G->>C: ListPresent(ecoID)
        C-->>G: cached tasks, Stale = true
        Note over G,C: Cache may be reached only after<br/>a timeout of up to 30 seconds
    end
    end

The planned correction is cache-first reads with an optional, narrowly controlled live refresh. That removes an independent high-volume caller, prevents throttling from repeatedly forcing the same failed write-through path, and turns a Render outage into immediate stale data rather than a long wait followed by stale data.

The product decision is how stale data may be before a demand refresh is justified. That should be visible through an as-of timestamp, not hidden behind a resolver that silently contacts Render.

Restart Replays the First Poll

render_tasks_cache is durable. The service state that schedules and compares polls is not: instanceLastPoll, initialized, initOnce, and fingerprints live in in-memory maps.

sequenceDiagram
    participant P as New process
    participant S as Service
    participant DB as Postgres
    participant R as Render API

    P->>S: Start with empty in-memory maps
    S->>DB: FindDispatched()
    DB-->>S: N pollable ECOs
    S->>S: dueECOs()
    Note over S: No instance has a last-poll time,<br/>so every instance is due
    loop N ECOs, max 3 concurrent
        S->>R: GET /tasks?subsector=...
        S->>S: seed fingerprints and suppress<br/>initial change events
    end
    Note over S,DB: Initialization also stamps<br/>last_task_change_at

This happens once per process start: deploys, pod restarts, crash loops, and scale events. ArgoCD image updates make restarts routine.

Pollable ECOs	Requests per restart	Cost at 5 restarts/day	Daily quota share
12	12	60	0.6%
100	100	500	5%
1,000	1,000	5,000	50%

At current scale the quota cost is negligible. At target scale it is severe. The second effect matters sooner: initialization stamps last_task_change_at, resetting the 48-hour inactivity TTL for the whole pollable set. Repeated restarts can therefore keep stale ECOs alive.

The database already has the durable marker needed to avoid that reset. Per-project fetching later reduces restart cost from one request per ECO to the number of pages needed for each project.

Rate Limiting Fails on the Wrong Timescale

Current failure handling is layered, but the layers were designed around short transient outages:

sequenceDiagram
    participant S as Service
    participant CB as Circuit breaker
    participant R as Render API

    S->>CB: request
    CB->>R: GET /tasks?subsector=...
    R-->>CB: 429 Too Many Requests
    Note over CB,R: Retried 3 times with roughly<br/>100 ms base backoff, Retry-After ignored
    CB-->>S: error after retries
    S->>S: backoff starts at 100 ms,<br/>doubling to 30 seconds
    Note over S: Wait is max(interval, backoff),<br/>so errors cannot accelerate polling
    Note over CB: Trips after 6 consecutive fully<br/>exhausted failures

Render uses a bare 429 for both per-second throttling and daily exhaustion, without a header that distinguishes “try again shortly” from “come back tomorrow.” Retrying can therefore spend more of a budget that is already gone.

The breaker has a similar timescale mismatch. Partial throttling may prevent six consecutive fully exhausted failures, so the breaker may never open. When it does open, a roughly 30-second half-open schedule produces repeated probes against a day-scale outage.

The narrow fixes are to stop retrying 429s on the poller path, abandon a cycle after a bounded number of rejections, and tune breaker behavior for partial throttling and quota recovery.

The Cache Has Two Different Kinds of Value

render_tasks_cache is a poller-maintained local replica of Render’s current state, plus history about when tasks appeared and disappeared. It is neither a read-through cache nor an event-sourced projection.

Data	Nature	If lost
Current tasks	Disposable cache	One poll cycle loses fallback, then it self-heals
first_seen_at / removed_at	System of record	History is permanently gone

That makes “cache” a dangerous shorthand. A projection rebuild must not clear this table. Replay cannot reproduce external observations that were never domain events.

There is also a schema hazard. The ECO projection rebuilds eco_views rows by deleting and rewriting them. An ON DELETE CASCADE relationship from the task cache could silently erase the audit history during replay, so that foreign-key action must remain deliberate.

Refresh is push-based from polling. There is no TTL to invalidate. Cache freshness is bounded by the polling interval, which is why longer quota-safe intervals need an as-of timestamp and narrow demand-refresh policy.

Operator controls apply at two scopes:

Control	Scope	Effect
Polling enabled	Per Render instance	Connection-level kill switch, including dispatch trigger
Interval seconds	Per Render instance	Cadence for that credential
Inactivity TTL	Per Render instance	Stop an ECO after no task changes for N hours
Completion grace period	Per Render instance	Continue watching after completion
RENDER_POLLING_ENABLED	Whole process	Global switch, applied at startup
RENDER_POLLING_INTERVAL	Whole process	Loop tick and default instance interval
RENDER_POLLING_WORKERS	Whole process	Poll concurrency, clamped to its burst share

The preferred emergency stop is the per-instance switch in the admin UI. For one or a few ECOs, set polling_stopped_at. The last resort is disabling polling globally and restarting the deployment.

The Honest Ceiling Is About Thirty-Four ECOs

The current design is linear: one request per ECO per cycle. At five minutes, one credential supports roughly 34 steady-state ECOs before the daily allowance is consumed. At one minute, it supports about six.

The improvements have a clear dependency order:

1. Put dispatch-triggered fetches behind the shared limiter.
2. Stop restart initialization from resetting the inactivity clock.
3. Make UI reads cache-first, with explicit staleness and controlled refresh.
4. Stop retrying 429s as if every throttle were momentary.
5. Fetch per Render project with pagination instead of once per ECO.
6. Extend inactivity coverage and make the circuit breaker respond to partial throttling.

The first four make the current linear design safer. Per-project fetching changes its shape: cost follows project pages and task counts rather than ECO count. That is the change that lifts the roughly 34-ECO ceiling.

One correlation decision remains open. Strike currently uses an exact Render subsector derived from the ECO job ID. A proposed move to an exact Render label would stop consuming a customer-owned field. Both approaches support exact server-side filtering, but the per-project design should not assume the correlation field is settled.

The operational rule until then is plain: treat every Render interval as a capacity decision, every live fetch as shared-budget spending, and every kill switch as part of the reliability model.

Discussion Prompts

1. What staleness window can the ECO list and detail screen tolerate before a demand refresh is worth spending shared quota?
2. Should the admin UI show estimated daily requests from the current interval and pollable ECO count?
3. At what connection size does per-project pagination become required rather than planned?

References

1. docs/architecture/render-integration-request-model.md in Osprey Strike — The source architecture review covering callers, quota math, cache behavior, restart effects, controls, and planned work.
2. docs/decisions/ADR-007-render-task-cache.md in Osprey Strike — Defines the cache as a poller-maintained replica with non-replayable audit history.
3. Osprey Strike Render polling package — Implements the background loop, per-instance scheduling, fingerprints, cache persistence, and completion checks.
4. Osprey Strike Render integration package — Implements task creation and the immediate post-dispatch fetch path.