Data/ML Engineer Blog

Data RDS

Designing for Global Reads on Aurora

Alex 0

Designing for Global Reads on Aurora: Latency Budgets, Read-After-Write, and DR Playbooks

Hook: Your EU customer clicks “Save,” hops to a dashboard, and… the number is wrong. Meanwhile an APAC user complains the page feels “heavy.” Welcome to the hard part of global scale: guaranteeing fast reads and read-after-write semantics across regions while keeping a tight RPO/RTO. This guide shows you exactly how to model latency, enforce read-your-writes, and drill your DR plan so failures are boring.

Why this matters (in 2 minutes)

  • Revenue & trust: Users churn fast when data looks stale after a write.
  • Physics is undefeated: Cross-ocean RTTs can blow through a 200 ms p95 budget if you’re careless.
  • Compliance & DR: Execs want numbers: “RPO ≤ 5 s, RTO ≤ 15 min.” You need a playbook that proves it.

We’ll focus on Amazon Aurora (MySQL/PostgreSQL) with Global Database and regional reader clusters.

Architecture at a glance

Baseline topology

  • Primary region: 1 writer + N readers in an Aurora cluster.
  • Secondary regions: Aurora Global Database secondary clusters (read-only).
  • Traffic routing: Latency-based DNS / Global Accelerator to nearest region.
  • App tier: Stateless services deployed per region; config/feature flags centralized; caches local.
  • Observability: Per-region SLOs, replication/lag metrics, synthetic probes.

Data flows

  1. Write path: Client → nearest region app → (if local region ≠ primary) forward to primary’s writer → commit → async storage-level replication to secondary regions.
  2. Read path: Client → nearest region app → nearest Aurora reader (same region) → return.

Key reality: Cross-region replication is asynchronous. You must design for read-after-write when the read hits a different region than the write.

End-to-end latency modeling (and how to keep it honest)

Model your p95 (or p99) latency as additive components:

T_total = T_edge + T_app + T_db + T_replica + T_cache_miss + T_tls + T_queue

Where:

  • T_edge: Client ↔ edge POP (CDN/Accelerator)
  • T_app: Regional service processing (including deserialization, auth, business logic)
  • T_db: Query execution time on the regional reader/writer
  • T_replica: (Only when enforcing read-after-write across regions) delay until the replica is at/after the write’s position
  • T_cache_miss: Extra round trips if you front reads with cache and miss
  • T_tls: Fresh TLS handshakes if connection reuse isn’t working
  • T_queue: Any queue/connection pool wait time under load

Practical budget for a “global read” SLO (example)

Budget itemTarget p95
T_edge (Anycast/Accelerator)10–20 ms
T_tls (HTTP/2 reused)~0–5 ms
T_app (simple read)10–20 ms
T_db (indexed point read)5–15 ms
Subtotal, same-region read25–60 ms
T_replica (cross-region RAW*)+ 0–200 ms (bursty)

* RAW = read-after-write. If the read is same region as write, T_replica ≈ 0; if read is in a different region immediately after a write, you pay replication catch-up.

Guardrails

  • Keep p95 query time for hot paths < 15 ms via appropriate indexes.
  • Keep app p95 < 20 ms (no blocking I/O in request thread, aggressive connection reuse).
  • Only enforce RAW when you must; otherwise accept eventual consistency to avoid waiting on replication.

Read-after-write (RAW): 4 workable patterns

You need determinism, not hope. Choose one (or mix) per endpoint:

1) Sticky post-write reads to the writer

  • How: After a successful write, mark the session/request context for N seconds to route reads to the primary writer (or same-AZ reader).
  • Pros: Simple, guaranteed RAW.
  • Cons: Adds cross-region latency when user is far from primary; increases writer load.

Implementation sketch

  • Set a header/claim like rw_sticky_until = now()+3s.
  • Your API gateway/service routes such requests to the writer endpoint.

2) Version fence (“wait-until at least”)

  • How: On write, return (entity_id, version|LSN|commit_ts). For the next read, the client sends min_version=.... The read path blocks or retries until the regional replica has applied ≥ that version.
  • Pros: Precise RAW without forcing all reads to writer.
  • Cons: You must fetch replica apply position and implement bounded waits with fallbacks.

Implementation sketch (Aurora-agnostic)

-- Table carries monotonically increasing version
UPDATE account SET balance = balance + :delta, version = version + 1 WHERE id = :id;

-- Client receives version V. On read:
SELECT version, balance FROM account WHERE id = :id;
-- If version < V, sleep/backoff 20–40ms and retry up to a cap.

Store V in your UI state for a few seconds; drop it after.

3) Dual-read with reconcile (fast-path eventual, slow-path RAW)

  • How: Read locally. If the version is stale, silently re-read from primary and reconcile.
  • Pros: Great perceived latency for most users; correctness when it matters.
  • Cons: More moving parts and duplicate traffic.

4) Cache-backed tokens

  • How: After commit, write a short-TTL token {key, commit_ts} to a global cache (e.g., Redis with CRDT/replication or regional Redis + global pub/sub). Readers check token; if present and replica_ts < commit_ts, they delay or route to writer.
  • Pros: Centralized logic; easy to expire.
  • Cons: Requires extra infra; cache consistency matters.

Aurora engine note: Aurora offers read routing constructs and consistency behaviors that evolve by engine/version. Treat any “session RAW on readers” setting as engine-specific; verify availability/latency trade-offs in your version before you bet your SLO on it.

Real example: budgeting a cross-region RAW read

Scenario: Primary in us-east-1, secondary read in eu-central-1. A user in Germany updates a profile and immediately opens their dashboard (read in EU).

  1. Write path (EU app → US writer):
    • App + TLS + DB: ~35 ms
    • EU↔US network: ~80–100 ms RTT → ~100–140 ms end-to-end write
  2. Replication apply to EU reader: typically small, but bursty under load. Budget 0–200 ms.
  3. Read path (EU app → EU reader): ~20–30 ms plus any wait-until delay for RAW.

Takeaway: To keep p95 < 200 ms including RAW, you either:

  • enforce RAW only on the fields the UI surfaces instantly and keep those narrow, or
  • route RAW reads to the writer (paying the transatlantic RTT), or
  • use version fences with bounded wait (e.g., cap at 120 ms) and degrade gracefully if exceeded.

DR strategy: RPO/RTO goals and how to drill them

Definitions

  • RPO (Recovery Point Objective): How much data you can afford to lose (seconds).
  • RTO (Recovery Time Objective): How long you can be down (minutes).

Target envelope (sane defaults to start)

  • RPO: ≤ 5 seconds (steady-state low traffic may be sub-second; design for bursts).
  • RTO: ≤ 15 minutes (automated failover + app traffic cutover).

DR playbook (global database)

Pre-reqs

  • Secondary regions hot (readers serving traffic).
  • Infra as code: cluster params, subnet groups, secrets, parameter groups reproducible.
  • App images & migrations deployable per region.
  • Runbooks codified in pipelines (not wikis).

Failover steps (primary region impaired)

  1. Freeze writes at the app edge (reject write intents or switch to read-only mode).
  2. Promote secondary: execute cluster promotion in the chosen region (automated or one-click).
  3. Rotate endpoints: update writer/reader endpoints and routing (DNS/Accelerator).
  4. Unfreeze writes in new primary region.
  5. Backfill: reconcile any out-of-band data (queues, caches).
  6. Post-incident validation: schema/version checks, synthetic write/read tests, data diff sampling.

RPO/RTO drill (quarterly)

  • RPO drill:
    1. Induce controlled replication lag (traffic + throttled IOPS).
    2. Measure delta between last commit on old primary and first commit in new primary after promotion.
    3. Verify meets ≤ target seconds; tune I/O, commit settings, and write burst patterns.
  • RTO drill:
    1. Start timer at “primary declared unhealthy.”
    2. Promote secondary via automation.
    3. Flip traffic, run smoke tests.
    4. Stop timer on “p95 latency back under SLO and writes enabled.”
    5. If > target, identify the long poles: DNS TTL, container warm-up, connection pool stabilization, migration locks.

Evidence pack for leadership

  • Before/after CloudWatch graphs (lag, commit latency).
  • Synthetic probe results (p95 end-to-end).
  • Checklist with timestamps per step.

Best practices & common pitfalls

Best practices

  • Design endpoints by consistency need. Mark each as RAW required vs eventual OK.
  • Keep write transactions skinny. Short locks, single partition/row hot paths, idempotent retries.
  • Precompute read models. Materialized views/tables for dashboards; keep reads O(1) with the right index.
  • Connection hygiene. Use pooled, long-lived connections; cap max concurrency to avoid head-of-line.
  • SLO-first monitoring. Track p95/p99 per region as seen by users and alert on SLO burn, not CPU.
  • Chaos/game days. Practice partial region loss, long-haul packet loss, and replica lag spikes.

Pitfalls

  • Global cache invalidation tied to DNS region: users flip regions and see stale data. Use keys that include version/fencing, or scoped TTLs.
  • “One size fits all” RAW: Forcing RAW everywhere crushes latency and writer capacity.
  • DNS TTL too long: Slow traffic cutover; keep TTLs low (but watch for resolver cache load).
  • Migrations during failover: Schema changes blocking promotion. Schedule and gate with feature flags.

Reference snippets

1) Version fence with bounded wait (app pseudocode)

def read_with_raw(entity_id, min_version=None, timeout_ms=120):
    start = now_ms()
    while True:
        row = db.query("SELECT version, payload FROM entity WHERE id = %s", [entity_id])
        if min_version is None or row.version >= min_version:
            return row.payload
        if now_ms() - start > timeout_ms:
            # Degrade: show stale with banner or route to writer
            return route_to_writer(entity_id)
        sleep(jitter(20, 40))  # backoff 20–40 ms

2) Simple post-write stickiness

# Response headers after write
X-RW-Sticky-Until: 2025-11-21T12:00:03Z

Your API gateway/service reads this header and pins subsequent GETs to the writer until expiry.

3) Lag SLO alerting (concept)

  • Alert when: p95_global_read > SLO AND replication_lag > threshold for 5+ minutes.
  • auto-mitigate: enforce stickiness or downgrade features that require RAW.

Comparison: RAW strategies

StrategyLatencyCost on writerComplexityWhen to use
Sticky to writerHigh (cross-region)HighLowPost-write confirmation screens
Version fenceMediumLowMediumProfile/orders immediately after change
Dual-read & reconcileLow (perceived)MediumHighCritical UX with high read volume
Cache tokensMediumLowMedium-HighMulti-service RAW coordination

Conclusion & takeaways

  • Write lanes and read lanes are different products. Give each endpoint an explicit consistency contract.
  • Model your SLO mathematically and defend the budget in code reviews.
  • Practice failover until your RPO/RTO are muscle memory, not slides.
  • RAW is a scalpel, not a hammer—apply only where the UX truly needs it.

Call to action:
Want a tailored latency budget and DR drill plan for your stack? Share your regions, target SLO, and hottest endpoints; I’ll turn this into a concrete runbook you can execute this quarter.

Internal link ideas

  • “Sharding Strategies 101: Range vs Hash vs Directory (with failure modes)”
  • “DynamoDB Transaction Tokens vs SQL Version Fences”
  • “Global Caching Patterns: TTL, Versioned Keys, and Invalidation”
  • “Observability for SLOs: RED/USE + Synthetic Probes for Databases”

Image prompt (for DALL·E/Midjourney)

A clean, modern data architecture diagram showing an Aurora Global Database: primary region with writer and readers, two secondary regions with read replicas, latency-based routing, and an inset timeline illustrating read-after-write version fencing and DR promotion. Minimalistic, high-contrast, isometric 3D style.

Tags

#Aurora #RDS #NoSQL #DataEngineering #GlobalScale #Latency #Consistency #DR #RPO #RTO

Related Story

Leave a Reply

Your email address will not be published. Required fields are marked *