Replication Reliability
Make replication production-safe: retry queue for durable push replication, Codios auth cache, circuit breaker, and the ops runbook for partition recovery.
What you'll learn
- ✓Push retry queue — per-peer in-memory queue, 50k block cap, exponential backoff (1s → 60s)
- ✓PROLONGED_FAILURE_THRESHOLD alert — error logged every ~3 hours on extended outage
- ✓POST /v1/mesh/sync/:peer_id — manual catch-up pull after partition recovery
- ✓GET /v1/cluster/status — check last_sync_ms and replication lag
- ✓Codios auth cache — permit TTL (SAPIX_AUTH_CACHE_TTL_SECS), deny TTL 2s, max 1000 entries
- ✓Stale-while-revalidate — serve cached permit on Codios miss, log auth_degraded: true
- ✓Circuit breaker — SAPIX_AUTH_CIRCUIT_BREAKER=true, opens after SAPIX_AUTH_CIRCUIT_OPEN_AFTER_SECS
Simulate a peer outage: stop the replica, write 20 records to primary, restart replica, trigger manual sync, verify all 20 records arrived. Then enable the auth cache and confirm Codios latency drops on repeated writes.
What you'll learn
Make mesh replication production-safe with a durable push retry queue, Codios auth cache, circuit breaker, and a runbook for partition recovery.
The problem with fire-and-forget replication
Lesson 51 shows how to register a peer with POST /v1/mesh/peers. Out of the box, SapixDB pushes each write to registered peers immediately. But if a peer is temporarily down, that block was previously dropped silently. The only recovery was a manual pull-sync from the replica side.
Two production-reliability features solve this: 1. Push retry queue — blocks are retried automatically until the peer accepts them 2. Codios auth cache — eliminates the Codios round-trip on every write
Push replication retry queue
When a push to a peer fails, the block is placed into that peer's RetryQueue. A background task wakes every 500 ms, checks which peers have messages ready (backoff elapsed), and drains them.
Retry queue properties
| Property | Value |
|---|---|
| Cap per peer | 50,000 blocks (~50 MB for typical payloads) |
| Initial backoff | 1 second |
| Backoff doubling cap | 60 seconds |
| On cap hit | Oldest block dropped; replica catches up via pull-sync |
| Heal detection | First successful push resets backoff; queue drains at full speed |
| Extended outage alert | tracing::error! logged every ~3 hours (PROLONGED_FAILURE_THRESHOLD) |
The retry queue is in-memory — the WAL on the primary is already durable. If the primary restarts during a partition, pull-sync from the persisted cursor handles catch-up.
Check replication lag
# Check cluster status on the replica — look at last_sync_ms
curl -s http://replica:7475/v1/cluster/status \
-H "Authorization: Bearer spx_root_REPLICA_ROOT_KEY" \
| python3 -m json.toolIf last_sync_ms is stale (too far in the past), the replica is lagging.
Manual catch-up after a partition
# Trigger pull-sync on the replica — fetches only missing blocks using sync cursor
curl -s -X POST http://replica:7475/v1/mesh/sync/primary \
-H "Authorization: Bearer spx_root_REPLICA_ROOT_KEY" \
| python3 -m json.toolThe sync cursor is persisted in sync_cursors.json — pull-sync fetches only blocks written after the cursor, not the entire strand.
Codios auth cache
By default, every POST /v1/records makes a synchronous HTTP call to SAPIX_CODIOS_URL for authorization. In multi-region setups this adds 5–30 ms per write.
The auth cache stores the result of each (operation, resource) Codios call. Subsequent writes with the same key are served from cache with no HTTP call.
Stale-while-revalidate: if Codios is unreachable on a cache miss and a stale permit entry exists for the same key, SapixDB serves the stale permit and logs auth_degraded: true rather than blocking the write.
Auth cache configuration
# docker-compose.yml
environment:
SAPIX_AUTH_CACHE_TTL_SECS: 5 # permit TTL in seconds (default: 5)
# Deny TTL is hard-coded to 2s for fast policy propagation| Setting | Default | Env var |
|---|---|---|
| Permit cache TTL | 5 seconds | SAPIX_AUTH_CACHE_TTL_SECS |
| Deny cache TTL | 2 seconds | (hard-coded) |
| Max cache entries | 1,000 | (hard-coded) |
The cache is always on — no opt-in required. It activates as soon as SAPIX_CODIOS_URL is set.
Circuit breaker (opt-in)
For deployments where Codios availability should not gate write throughput at all:
environment:
SAPIX_AUTH_CIRCUIT_BREAKER: true # enable (default: false)
SAPIX_AUTH_CIRCUIT_OPEN_AFTER_SECS: 30 # seconds before circuit opens (default: 30)When Codios has been continuously unreachable for longer than SAPIX_AUTH_CIRCUIT_OPEN_AFTER_SECS, the circuit opens. All writes proceed using cached auth decisions; every bypass is logged as auth_degraded: true. When Codios responds again, the circuit closes automatically.
Multi-region recommendation
Place each SapixDB node and its Codios instance in the same availability zone. The cache handles burst throughput; cross-region latency is eliminated during cache hits.
Ops runbook — partition recovery
`bash
# 1. Check replica lag
curl -s http://replica:7475/v1/cluster/status \
-H "Authorization: Bearer $REPLICA_KEY" \
| python3 -m json.tool
# 2. If last_sync_ms is stale — trigger manual pull-sync curl -s -X POST http://replica:7475/v1/mesh/sync/primary \ -H "Authorization: Bearer $REPLICA_KEY" \ | python3 -m json.tool
# 3. Verify chain integrity on replica after sync curl -s http://replica:7475/v1/strand/verify \ -H "Authorization: Bearer $REPLICA_KEY" \ | python3 -m json.tool
# Healthy: chain_intact: true, signatures_invalid: 0
`
Challenge
Simulate a peer outage: stop the replica container. Write 20 records to the primary. Restart the replica. Trigger manual sync. Verify all 20 records arrived and chain integrity is intact.
Bonus: Set SAPIX_AUTH_CACHE_TTL_SECS=5. Write 100 records back-to-back to a Codios-protected agent. Compare write latency before and after the cache warms up.
---