Mall Entrance Occupancy Management System

This problem looks like a toy counter problem. It is not. The earned-secret insight is that a mall occupancy counter is a life-safety system with an asymmetric CAP choice baked into regulation, not engineering taste:

• Entry path is CP — over-admission is a regulatory & liability incident (fire code violation).
• Exit path is AP — under-counting exits is annoying (admits fewer people than legally allowed), but blocking an exit is a felony under most building codes. Software must never be able to prevent egress.

Everything else — Redis-vs-Spanner, sharded counters, lease models — is engineering that serves this asymmetry. If the candidate only designs a "distributed counter," they land at L5. L6 adds sharding and event sourcing. L7 recognizes that the two sides of the turnstile run on different CAP regimes wired to different hardware fail-safe modes, and that the software design cannot be the root of trust for exits because the fire marshal says so.

Restatement. Build a system that, for each shopping mall, (a) enforces a strict maximum occupancy by admitting or blocking entries at any of N gates in real time, (b) never prevents egress, (c) exposes a live count, (d) retains a complete audit trail of every entry/exit event with gate attribution, and (e) remains safe-by-default under partition or central-service failure.

Clarifying questions (ask these up front — ~3 min):

Non-questions (stated, not asked): we assume the mall network has some backhaul (fiber or LTE), but it can partition. We assume each gate has a local microcontroller (gate controller) with ~MB of RAM and persistent storage, and that no gate controller has >1s of un-persisted state. We assume time is approximately synchronized via NTP within ±50ms; we do not rely on wall-clock ordering for correctness.

In-scope (numbered, testable):

1. FR1 — Atomic entry admission. request_entry(gate_id) must return ADMIT only if the global occupancy strictly remains ≤ capacity after the increment. No two concurrent requests anywhere in the mall may both see the same "last slot" and both admit.
2. FR2 — Unrestricted exit. record_exit(gate_id) must always succeed logically; it may never be rate-limited, blocked, or gated on central-service reachability. If central is unreachable, exit is recorded locally and replayed.
3. FR3 — Live occupancy query. get_occupancy(mall_id) returns current count with bounded staleness (≤ 5s under normal ops, ≤ 60s under partition).
4. FR4 — Per-gate metrics & dashboard. Operations dashboard shows occupancy, per-gate admit/reject/exit rate, queue depth estimate, sensor health.
5. FR5 — Audit log. Every entry attempt (admit or reject) and every exit is persisted as an immutable event with event_id, mall_id, gate_id, type, ts, admission_decision, admission_reason, sensor_confidence.
6. FR6 — Configurable capacity. Operator can set capacity with effective_from timestamp. Capacity can be reduced at runtime (e.g. HVAC failure, police order) — system must honor the lower bound immediately without crashing.
7. FR7 — Emergency override. emergency_override(mode=EVACUATE | LOCKDOWN | FIRE) — EVACUATE flips all gates to exit-only mode, blocks entries; FIRE triggers hardware fail-open (all turnstiles release); LOCKDOWN is for security events and still honors fire-code exits.
8. FR8 — Capacity change audit. Capacity changes themselves are audit events (who, when, why, previous value).

Out-of-scope (explicitly stated to bound the interview):

• Identity, demographics, facial recognition, loyalty program tie-in.
• POS revenue correlation, footfall → sales forecasting.
• Occupancy prediction / ML crowd forecasting (v3 feature, §9).
• Per-zone occupancy (food court vs parking). Treat mall as a single bucket.
• Ticketing, reservations, paid entry.

3.1 NFRs

3.2 Back-of-the-envelope (all math shown)

Single-mall admission load.

• Capacity C = 10,000.
• Typical turnover: 3× full capacity per day ⇒ 30,000 admissions/day.
• Peak hour concentration: 25% of daily traffic in busiest hour ⇒ 7,500 admits/hr = 2.08 admits/s steady peak.
• Black-Friday / mall-opening burst: 10× peak-steady ⇒ ~20 admits/s sustained, peaks of 100/s for 30–60s when doors open.
• Per-gate: 20 entry gates ⇒ burst ~5/s/gate — one turnstile can physically cycle at most ~1 person/1.5s ≈ 0.67/s, so the 5/s/gate burst figure is actually bounded by human-throughput; the system must survive every gate cycling at full physical rate simultaneously = 20 × 0.67/s ≈ 13.4 admits/s sustained, bursts to ~20/s with brief queue absorption. ⇒ Design for 100/s admit QPS per mall (5× safety factor).

Fleet load.

• 500 malls × 20/s steady avg = 10,000 admits/s across fleet.
• Fleet-wide peak (all malls at once, e.g. Saturday 11am across one time zone): 500 × 100/s = 50,000 admits/s peak.

Event log sizing.

• Event row: event_id(16B) + mall_id(8B) + gate_id(4B) + type(1B) + ts(8B) + decision(1B) + reason(8B) + sensor_confidence(4B) + serialization overhead ≈ 120B wire, 200B on-disk with indexing & compression-offset.
• Per mall per day: (30,000 admits + 30,000 exits + ~1,000 rejects) × 200B ≈ 12.2 MB/day/mall.
• Fleet: 500 × 12.2 MB = 6.1 GB/day; ~2.2 TB/yr hot.
• 7-yr cold tier: ~15 TB — trivial for object storage, ~$350/mo at S3 Glacier Deep Archive rates.

Counter store sizing.

• Hot state: ~1 KB/mall (counter, version, lease table, config) × 500 malls = 500 KB active state. Fits in any single Redis/Spanner node trivially; the scale is in the write QPS, not the size.

Read load.

• get_occupancy from ops dashboard: polled every 5s per mall view × say 20 operator sessions/mall × 500 malls = 10K queries/5s = 2K QPS. Trivial.

Key takeaway of BoE: write contention, not volume, is the constraint. The hot partition is the single counter per mall. 100 QPS on one key is fine for Redis (100K+ INCR/s on one shard), fine for Spanner single-row (thousands of commits/s with conflict-free sequencer), but pathological for a naive RDBMS SELECT ... FOR UPDATE pattern (contention latency degrades super-linearly).

All APIs: gRPC primary (low-latency, streaming for sensor telemetry), HTTP/JSON gateway for dashboard and admin.

4.1 Gate-plane (gate controller ↔ entry coordinator)

service EntryCoordinator {
  // Called on every turnstile presentation. Must be fast and idempotent.
  rpc RequestEntry(EntryRequest) returns (EntryDecision);

  // Fire-and-forget stream from gate controller; at-least-once.
  rpc RecordExit(stream ExitEvent) returns (ExitAck);

  // Gate pulls more lease tokens when its local budget is low.
  rpc AcquireLease(LeaseRequest) returns (LeaseGrant);

  // Gate returns unused tokens on shift end / graceful shutdown.
  rpc ReleaseLease(LeaseRelease) returns (LeaseAck);

  // Heartbeat + reconciliation (gate reports local counters; coord reconciles).
  rpc Reconcile(stream ReconcileBatch) returns (ReconcileAck);
}

message EntryRequest {
  string event_id = 1;      // UUIDv7 from gate. Idempotency key.
  string mall_id = 2;
  string gate_id = 3;
  int64  gate_local_seq = 4; // Monotonic per-gate. Gap-detection on replay.
  int64  ts_gate_ms = 5;     // Gate-local NTP time.
  float  sensor_confidence = 6; // 0..1. <0.8 ⇒ flag for reconciliation.
}

message EntryDecision {
  enum Code { ADMIT = 0; REJECT_AT_CAPACITY = 1; REJECT_EMERGENCY = 2;
              ADMIT_LOCAL_LEASE = 3;  // decided from gate's local lease, not central
              ERROR_RETRY = 4; }
  Code   code = 1;
  int64  occupancy_after = 2; // best-effort; may be stale under fallback
  int64  capacity = 3;
  string decision_source = 4; // "central" | "gate_lease" | "cached_fallback"
  int64  decision_latency_us = 5;
}

4.2 Control-plane (operator / admin)

service MallOps {
  rpc GetOccupancy(GetOccupancyRequest) returns (Occupancy);
  rpc StreamOccupancy(GetOccupancyRequest) returns (stream Occupancy); // push
  rpc SetCapacity(SetCapacityRequest) returns (CapacityChange);
  rpc EmergencyOverride(OverrideRequest) returns (OverrideAck);
  rpc GetGateHealth(MallRef) returns (stream GateHealth);
  rpc QueryAuditLog(AuditQuery) returns (stream AuditEvent);
}

4.3 Semantics

• Idempotency. event_id is a UUIDv7 minted by the gate controller. The coordinator maintains a bounded-time dedup set (last 5 min) to absorb retries. Events older than the dedup window are accepted (gate was partitioned longer than 5 min) but marked late_arrival=true and reconciled; double-admission risk is bounded to gate_local_lease size (see §7.2).
• Idempotency key is per-gate and per-sequence, not just UUID. This lets us detect gaps on replay: a gate that reconnects with seq 103 when we last saw 97 must have 5 events in flight that we need to reconcile.
• No cross-mall calls on hot path. Each mall is an independent bulkhead.
• ADMIT_LOCAL_LEASE vs ADMIT — deliberately a distinct return code. Dashboards and audit surface the source of every decision. This makes the "is the system partitioned right now?" question answerable from the data plane, not just from ops monitoring.

5.1 Primary stores (authoritative)

occupancy_counter — one row per mall, strongly consistent. This is the hot row.

Engine choice (see §7.1 for full analysis): Spanner single-row with read-write transactions for the authoritative counter; Redis cluster with Lua script as a tier-1 cache for sub-10ms reads + optimistic-path writes, with async write-through to Spanner. Rejected engines documented in §7.1.

gate_lease — one row per live lease. Mall-sharded.

events — append-only audit log. Kafka primary (RF=3, min.insync.replicas=2, acks=all). Compacted topic with mall_id as partition key for per-mall ordering; sub-partitioned by mall_id||gate_id to prevent one mall's audit stream from head-of-line-blocking another.

Archive: Kafka → tiered to object store (Parquet files, daily partitions) via Kafka Connect. Cold tier = Glacier-class. Hot query via Druid or ClickHouse (§5.3).

capacity_policy — small OLTP table (Postgres or Spanner).

5.2 Rationale for the effective_capacity = capacity - leased_out column

Naive designs keep one counter count. Under leased slots, the central count is the last reconciled count, which lags. If the coordinator compares count < capacity to admit, it will over-admit by leased_out. Precomputing effective_capacity means central's own admit path (for gates that round-trip) respects the outstanding leases. The leases are phantom admits from central's perspective.

5.3 Operational dashboard

• ClickHouse or Druid fed from the Kafka audit stream.
• Rollups: 1s, 10s, 1min, 1hr buckets per gate × decision.
• Why not just Grafana over Prometheus? Because we also need ad-hoc queries ("who entered gate 7 between 14:32 and 14:47 on Apr 4" for a lost-child or theft case), and that's an OLAP-shape workload.

5.4 A note on CRDTs (and why not)

A PN-Counter CRDT gives partition-tolerant counting with eventual consistency. It is wrong for this problem because it permits unbounded over-admission during partition (each partition independently increments without bound), which violates the hard fire-code ceiling. CRDTs are AP; we need CP on the entry path. The leased-slot model is a CP-shaped answer that looks CRDT-ish (local increments, eventual merge) but with bounded divergence via the lease size.

6.1 Full topology

                                      PHYSICAL PLANE                          ┃          NETWORK PLANE                    ┃   SERVICE PLANE (per mall)              ┃   FLEET CONTROL PLANE
                                                                              ┃                                           ┃                                         ┃
   ┌────────────────────┐    mag-lock  ┌──────────────────┐                   ┃                                           ┃                                         ┃
   │  Turnstile #1..20  │◄────HW────►  │ Fire alarm bus   │───┐               ┃                                           ┃                                         ┃
   │  (IR + HW counter) │ fail-open   │ (NFPA hardwired) │   │ override      ┃                                           ┃                                         ┃
   └─────────┬──────────┘              └──────────────────┘   ▼               ┃                                           ┃                                         ┃
             │ RS-485 /                                      ┌─────────────┐ ┃                                           ┃                                         ┃
             │ GPIO (<5ms)                                   │ Hardware    │ ┃                                           ┃                                         ┃
             ▼                                               │ Key-switch  │ ┃                                           ┃                                         ┃
   ┌───────────────────────────┐                             │ @ sec. desk │ ┃                                           ┃                                         ┃
   │ Gate Controller (per      │                             └──────┬──────┘ ┃                                           ┃                                         ┃
   │   turnstile, embedded)    │─┐                                  │        ┃                                           ┃                                         ┃
   │ - 4KB dedup ring          │ │                                  │        ┃                                           ┃                                         ┃
   │ - lease cache (N tokens)  │ │                                  │        ┃                                           ┃                                         ┃
   │ - local SQLite audit WAL  │ │                                  │        ┃                                           ┃                                         ┃
   │ - fallback policy FSM     │ │                                  │        ┃                                           ┃                                         ┃
   └─────────────┬─────────────┘ │                                  │        ┃                                           ┃                                         ┃
                 │  gRPC/mTLS    │ mTLS                             │        ┃                                           ┃                                         ┃
                 │  <5ms LAN     │ telemetry                        │        ┃                                           ┃                                         ┃
                 ▼               ▼                                  │        ┃                                           ┃                                         ┃
   ┌──────────────────────────────────────────┐                     │        ┃                                           ┃                                         ┃
   │  Edge Gateway (per mall, HA pair)        │                     │        ┃                                           ┃                                         ┃
   │  - local circuit-breaker to WAN          │                     │        ┃                                           ┃                                         ┃
   │  - aggregation, mTLS termination         │                     │        ┃                                           ┃                                         ┃
   │  - local Redis (read-through cache)      │                     │        ┃                                           ┃                                         ┃
   │  - fallback arbiter (see §7.2)           │                     │        ┃                                           ┃                                         ┃
   │  - local Kafka producer buffer (24h)     │                     │        ┃                                           ┃                                         ┃
   └──────────┬───────────────────────────────┘                     │        ┃                                           ┃                                         ┃
              │   WAN (SD-WAN, dual-homed fiber+LTE)                │        ┃                                           ┃                                         ┃
              │   p50 20ms, p99 80ms, may partition                 │        ┃                                           ┃                                         ┃
              ▼                                                     │        ┃                                           ┃                                         ┃
══════════════════════════════════════════════════════════════════════════════┃═══════════════════════════════════════════┃═════════════════════════════════════════
              │                                                     │        ┃  REGION: MALL's home region               ┃   REGION: cross-region control
              │        ┌────────────────────────────────────────────┘        ┃                                           ┃
              │        │  override (dual-path: HW+SW)                        ┃                                           ┃
              ▼        ▼                                                     ┃                                           ┃
   ┌────────────────────────────────────────────────┐  (A)  CAS/Lua ┌────────────────────────────┐                       ┃
   │  Entry Coordinator Service (per mall, 3x HA)   │◄─────────────►│  Primary Counter Store     │                       ┃
   │  ┌──────────────────────────────────────────┐  │   <5ms        │  Spanner row (SR)          │                       ┃
   │  │ Admission FSM:                           │  │               │  - occupancy_counter       │                       ┃
   │  │  1. Normalize request                    │  │  (B) read     │  - gate_lease              │                       ┃
   │  │  2. Dedup on event_id                    │  │ through Redis │                            │                       ┃
   │  │  3. Check override mode                  │  │  Cluster (L1) │  + Redis Cluster (L1)      │                       ┃
   │  │  4. Choose path:                         │  │  ~1ms         │    - Lua CAS script        │                       ┃
   │  │     (a) central CAS (default)            │  │               │    - TTL'd lease snapshot  │                       ┃
   │  │     (b) grant lease to gate              │  │               │                            │                       ┃
   │  │     (c) fallback: reject if no budget    │  │               └──────────────┬─────────────┘                       ┃
   │  │  5. Emit audit event                     │  │                              │ async replicate to global audit    ┃
   │  │  6. Return decision                      │  │                              ▼                                      ┃
   │  └──────────────────────────────────────────┘  │                 ┌─────────────────────────┐                        ┃
   └──────────┬───────────────────────────────────┬─┘                 │  Kafka (audit bus)      │                        ┃
              │                                   │                   │  topic: mall-events     │                        ┃
              │ audit write                       │ exit path         │  RF=3, acks=all         │                        ┃
              │ (at-least-once)                   │ (AP)              │  partition: mall_id     │                        ┃
              ▼                                   ▼                   └──────┬────────┬─────────┘                        ┃
   ┌────────────────────┐                ┌────────────────────┐              │        │                                  ┃
   │ Kafka producer     │                │ Exit Sink          │              │        └──►  S3/GCS Parquet archive       ┃
   │ (local buffer      │                │ (async, writes     │              │              (7yr cold tier)              ┃
   │  on edge gateway)  │                │  count DECR +      │              ▼                                            ┃
   └────────────────────┘                │  audit event;      │      ┌──────────────────────┐                             ┃
                                         │  NEVER blocks exit)│      │ ClickHouse / Druid   │                             ┃
                                         └──────────┬─────────┘      │ (ops dashboard DB)   │                             ┃
                                                    │                └──────────┬───────────┘                             ┃
                                                    │                           │                                          ┃
                                                    └────► Spanner counter ◄────┘                                          ┃
                                                          (DECR is always                                                  ┃
                                                          accepted; may go                                                 ┃
                                                          transiently negative                                             ┃
                                                          under race, see §8)                                              ┃
                                                                                                                           ┃
                                                                                                                           ┃   ┌───────────────────────────────┐
                                                                                                                           ┃   │ Fleet Control Plane           │
                                                                                                                           ┃   │ - Multi-mall config svc       │
                                                                                                                           ┃   │ - Capacity policy DB          │
                                                                                                                           ┃   │ - Global ops dashboard        │
                                                                                                                           ┃   │ - PagerDuty / alert routing   │
                                                                                                                           ┃   │ - Tenant mgmt / RBAC          │
                                                                                                                           ┃   │ - Firmware rollout controller │
                                                                                                                           ┃   └───────────────────────────────┘

Every arrow carries (protocol, budget, QPS):

6.2 Sub-diagram: Entry Decision Flow (happy path + fallback)

                       EntryRequest(event_id, gate_id)
                                 │
                                 ▼
              ┌─────────────────────────────────────┐
              │ Gate Controller receives            │
              │ 1. dedup event_id in ring buffer?   │──yes──► return cached decision
              │ 2. emergency mode flag set?         │──FIRE─► OPEN (HW override)
              │    (read from local GPIO + SW)      │──EVAC─► REJECT (entry blocked)
              └──────────────┬──────────────────────┘
                             │
             local lease     │
              tokens > 0 ?   │
                   ┌─────────┴──────────┐
                  yes                   no
                   │                    │
      ┌────────────▼───────────┐       ▼
      │ LOCAL ADMIT:            │   ┌───────────────────────────┐
      │ - decrement local lease │   │ Call coordinator          │
      │ - write local WAL       │   │ RequestEntry              │
      │ - open turnstile        │   │ timeout: 80ms              │
      │ - async push audit      │   └──────┬──────────┬──────────┘
      │ - if lease < low_water, │          │          │
      │   schedule async refill │     success       timeout/error
      └─────────────────────────┘          │          │
                                           ▼          ▼
                              ┌───────────────┐  ┌──────────────────────┐
                              │ Apply         │  │ Fallback arbiter:    │
                              │ decision from │  │ - lease > 0? admit   │
                              │ coordinator   │  │ - else: pessimistic  │
                              │ (authoritative)│  │   REJECT & alert    │
                              └───────────────┘  └──────────────────────┘

6.3 Sub-diagram: Gate-Local Fallback State Machine

                     ┌──── central healthy (last_heartbeat < 5s) ────┐
                     │                                                │
                     ▼                                                │
               ┌──────────┐   heartbeat timeout 5s   ┌────────────┐   │
       ┌──────▶│  NORMAL  │────────────────────────▶ │ DEGRADED    │──┘ recovery
       │       └────┬─────┘                          └─────┬──────┘    (force reconcile)
       │            │                                      │
       │    lease refill ok                    lease depleted & no WAN
       │            │                                      │
       │            ▼                                      ▼
       │       ┌──────────┐                         ┌────────────┐
       │       │ LEASED   │──lease exhausted──────► │ PESSIMISTIC│──┐
       │       │ OPERATION│                         │ REJECT      │  │
       │       └──────────┘                         └────────────┘  │
       │                                                  │          │
       │              fire override at any state          │          │
       │              ┌───────────────────────────────────┘          │
       │              │                                              │
       │              ▼                                              │
       │       ┌─────────────┐   fire cleared + mgmt ack             │
       │       │ FIRE_OPEN   │────────────────────────────────────► NORMAL
       │       │ (HW-enforced│
       │       │  exit-only) │
       │       └─────────────┘
       └─────── (return after reconcile from any state) ──────

6.4 Asymmetry of exit path (critical L7 call-out)

The diagram deliberately shows exit as a separate data path:

1. Exit sensor → Gate Controller — no admission check.
2. Gate Controller writes exit to local WAL, opens turnstile.
3. Exit events stream async to Kafka, which decrements the counter eventually.
4. The exit path has no synchronous call to the coordinator. Even if the WAN is fully partitioned, exit is local and always works.
5. If the gate controller itself dies, the turnstile mag-lock fails open (hardware default) — exit is still physically possible.

This is the asymmetric CAP decision made physical. Entry = CP. Exit = AP.

7.1 Deep Dive 1 — Strongly-consistent counter without hot-key collapse

Why critical. The whole system centers on one row per mall. Naïvely, 100 writes/s on one key is easy; under the CP + audit requirement it becomes a contention battle.

Options considered (quantified):

Chosen approach — two-tier:

1. Authoritative layer: Spanner single-row per mall for occupancy_counter. Single-row RW transactions on a hot row in Spanner are actually a good match: Spanner's participant leader serializes conflicting commits on a single Paxos group, and the row is small enough that commit latency is dominated by WAN round-trip (10-15ms p99). At 100 QPS/mall we are nowhere near its ceiling (2-5K commits/s hot row).
2. Fast-path cache layer: Redis Cluster with Lua. The coordinator's hot path reads the effective_capacity from Redis (refreshed every 100ms from Spanner) and serves most requests from the lease model without hitting Spanner for every admit. Writes (central-path admits) go to Spanner authoritatively and then invalidate/update Redis.
3. Dedup buffer: Redis set with TTL=5min for event_ids, to absorb retries.

Why not just Redis? The business can tolerate a Redis failover dropping unreplicated admits (they would be re-issued lease tokens), but cannot tolerate a silent gap where the canonical count is wrong after failover. Spanner gives us externally-consistent source of truth; Redis is a performance cache in front of it. If we run Redis-only, we would need to implement our own Raft/Paxos replication of the counter to avoid post-failover ambiguity — that's rebuilding Spanner badly.

Production systems named:

• Ticketmaster "queue-it" and TicketSocket: use a similar two-tier model — a global token mint (authoritative) with local queue servers. See Andrea Leopardi / Livewire talks on Ticketmaster's high-contention ticket mint — they moved off Postgres FOR UPDATE to pre-allocated token pools precisely because lock-queue latency was unbounded.
• Disney Parks FastPass / Genie+: pre-allocates ride-slot tokens per-day and distributes them to kiosks/app clients; kiosks admit from local budget; central reconciles.
• Stripe idempotency keys: TTL'd dedup set, same pattern as our event_id dedup.
• Google Spanner single-row hot-key benchmarks (Jeff Dean, CIDR 2017): single row at thousands of commits/s is supported with minor caveats (participant-leader batching).

Failure modes + mitigations (of the chosen counter design):

• Spanner regional outage → failover is automatic within configured placement. During the ~30s failover window, coordinator falls into "leased-slot only" mode — gates admit from their existing leases, no new leases granted, no new central admits. When Spanner recovers, a reconciliation pass folds in buffered admits.
• Redis cache poisoning / staleness → bounded by 100ms refresh window; coordinator always falls back to Spanner for the authoritative admit.
• Hot mall (one mall at 500 QPS, 10× design) → single-row commit queue on Spanner starts building. Mitigation: increase lease size so more admits go local; lease grants are themselves admits from Spanner's perspective, so a gate taking a 50-token lease is 1 commit for 50 admits — this is the contention-amortization trick.

7.2 Deep Dive 2 — Leased-slot admission (earned-secret core)

Why critical. This is the model that makes the system both partition-tolerant and safe. It is the single most important technique in the design.

The model.

Each gate holds a local lease of L admission tokens (e.g. L=20). When a person presents:

1. If local_tokens > 0: decrement, admit, write local WAL. No network call.
2. If local_tokens == 0: synchronously call coordinator to AcquireLease(L'). Coordinator does a Spanner CAS: leased_out += L', effective_capacity -= L'. Returns a lease grant.
3. When gate goes idle or on graceful shutdown: ReleaseLease returns unconsumed tokens to central.

When local_tokens drops below a low-water mark (e.g. 5 of 20), the gate asynchronously requests a refill so the synchronous path rarely blocks on network.

CAP positioning. This is a bounded-divergence CP design. At any instant, actual_occupancy ≤ central_count + Σ(leased_out per gate) = capacity. The invariant is maintained by the central CAS at lease grant time, not at admit time. The lease is the commit.

Under partition (pessimistic):

• Gate's last-known lease is L_current. Gate admits up to L_current, then rejects. Worst-case over-admission during partition: zero (by definition — the leased tokens were already committed against capacity centrally).
• When the partition heals, gate sends Reconcile with its consumed count. Central sets leased_out -= unconsumed, count += consumed. Accounting is precise.

Under partition (optimistic) — rejected for this problem but worth analyzing:

• Gate continues admitting past lease, up to a safety cap L_current + Δ.
• Over-admission up to Σ Δ across gates, which is bounded. E.g. 30 gates × Δ=10 = 300 over-admits ≤ 3% of capacity.
• Acceptable for some venues (concert festival, where queue rage > safety margin). Not acceptable here because fire code is a hard ceiling enforced by inspection and liability.

Low-water mark math. Peak per-gate admit rate ~0.67/s. Network p99 to central = 80ms. Coordinator lease-grant p99 = 100ms total. Expected admits during one refill round-trip = 0.67 × 0.1 = ~0.07 — basically never block. Set low-water = ⌈p99_admits_during_refill × safety_factor⌉ = 3. Configure lease L=20 so refills happen once per ~30s/gate.

Why not just "gate_budget = capacity / gates"? Because traffic is not uniform. At the mall's main entrance gate, 90% of admissions happen; at a side gate, 5%. If we statically partitioned capacity, the main gate would reject prematurely while side gates sat on unused budget. The lease model is a dynamic reallocation — a form of work-stealing counter — where hot gates refill more often and cold gates let their leases expire.

Lease expiry handling. A gate's lease has a TTL (e.g. 60s). If the gate dies without returning the lease, central reclaims it via:

1. Heartbeats. Gate sends heartbeat with remaining lease every 5s.
2. On heartbeat miss × 3 (15s), coordinator marks lease SUSPECT.
3. At 60s TTL, coordinator reclaims: leased_out -= granted_slots, count += granted_slots × fudge_factor. Critically, the fudge_factor here is a safe assumption that ALL tokens were consumed — i.e. the system errs on the side of treating unreachable gates as having admitted their full lease. This prevents double-counting if the gate comes back online after reclamation. It means we under-count capacity (transiently treat more people as inside than might be true), which biases toward rejecting more entries, which is the safe direction.
4. When gate reconnects, it sends its actual consumption in Reconcile; central does a compensating adjustment.

This is the L7 move: the asymmetry of assumption under uncertainty is deliberately biased toward the safety-critical direction. Over-rejection is recoverable; over-admission is not.

Production analogs:

• AWS IAM / STS temporary credentials: pre-issued tokens with TTL, verified locally.
• Kubernetes kube-scheduler with resource reservations: nodes hold a reservation budget for pods; scheduler reconciles.
• Airlines' seat-map reservation locks: a travel agent holds N seat locks for 15 min; returns unused.
• CockroachDB's "SQL sequence caching": each SQL node caches a block of sequence values to avoid a round-trip per INSERT. Same idea, same edge case (cached values on a dead node are lost = gaps in sequence; acceptable for sequences, not for our counter — so we reclaim).
• Riot Games / Fortnite queue-token model: login tokens pre-minted in pools, consumed by region servers.

7.3 Deep Dive 3 — Asymmetric exit path (fire safety as architecture)

Why critical. A naive system wraps both entry and exit in the same "update counter" API. This is not just inefficient — it is illegal. Most building codes (NFPA 101 in US; equivalent EN, ISO) require that egress cannot be prevented by mechanical or electronic means except under active fire-marshal control. If software can block an exit, the building is not code-compliant.

The design.

1. Exit does not go through the admission FSM at all. Separate gate-controller code path.
2. Exit is fire-and-forget with durable local WAL. The gate controller writes the exit event to local SQLite (fsync) before releasing the turnstile. Even if it reboots mid-cycle, the event is preserved.
3. Exit batches are streamed async to Kafka via the edge gateway. Kafka's consumer group (the "Exit Sink") DECRs the Spanner counter. Since exits can never cause over-admission (they only reduce count), eventual replication is safe.
4. Hardware-level fail-open. The turnstile mag-lock is wired so that loss of power or loss of signal from the gate controller releases the lock. Software cannot hold the lock closed.
5. Fire alarm integration is a hardwired NFPA loop, not an API call. When the fire panel asserts alarm, all mag-locks drop simultaneously via dedicated copper. Our software also enters FIRE mode via a separate telemetry channel (so we record the event), but the software path is the belt; the hardware loop is the suspenders.

Race: exit before entry admit replicates. Scenario: person enters gate 1, coordinator admits, Kafka event in flight. Same person exits gate 2 before the entry event reaches Spanner. The counter would transiently go negative.

Mitigation: count is stored as int64 (room for negative), and the dashboard clamps display to max(0, count). Over a reconciliation window (5s), the entry event catches up and the counter recovers. The invariant we enforce is "never above capacity," not "never below zero." This is an explicit L7 decision: the ceiling is regulatory; the floor is cosmetic.

Race: double-exit. Sensor fires twice on a single person (tailgating variant in reverse). Gate controller dedups within a 500ms window at the sensor level (hardware debounce + software). Cross-gate dedup is impossible (person physically at one gate at a time), so this is a local concern.

Tailgating (the real correctness ceiling). Turnstiles allow tailgating at ~0.1–0.3% depending on design (tailgater slips through behind another person, sensor only counts one rotation). This means the sensor itself has error that software cannot fix. We address this by:

1. Overhead optical counter (secondary sensor) at each gate, independently counted.
2. Coordinator reconciles turnstile count vs optical count hourly; discrepancy > 0.5% triggers ops alert and a sensor calibration task.
3. Reporting: "Sensor-confident admissions" vs "total foot traffic" are separate metrics. Fire-code compliance is measured against the higher of the two (worst-case), so we under-capacity relative to sensor-counted admissions.

Why not ML tailgating detection on camera? v3 feature. Introduces privacy surface (see §10) and drifts accuracy under crowd load. For v1/v2 we rely on dual-sensor mechanical reconciliation and ops-driven calibration.

Production analogs:

• Airport boarding: boarding pass scan is the "admit"; exit (deplaning) is physically unstoppable.
• Amusement park ride queues (Disney LightningLane): entry is ticketed; exit is free.
• Every data-center badge reader: entry requires auth; emergency egress ("panic bar") is hardware-guaranteed.

v1 — "Make it correct, for one mall" (0–3 months)

• Single mall, single region.
• Centralized Redis with Lua CAS on the occupancy counter; Postgres for capacity_policy and audit.
• Gates are synchronous — every admit is a blocking coordinator call.
• No leases, no local fallback — if central is down, entries stop.
• Exits are async batched to Redis (still asymmetric from day 1 — fire code non-negotiable).
• Ops dashboard: Grafana over Postgres audit.
• Good-enough SLO: p99 100ms local mall network; central outage = entries stop (acceptable for single-tenant pilot).

v2 — "Make it partition-tolerant and fleet-ready" (3–9 months)

• Add leased-slot model (§7.2) — gates hold local budget, operate through short partitions.
• Migrate authoritative counter from Redis → Spanner (or FoundationDB); Redis becomes L1 cache.
• Add Kafka audit bus — move audit off the critical path; introduce ClickHouse for OLAP.
• Multi-mall (200–500): control plane for fleet, per-tenant isolation by region.
• Reconciliation job runs on lease expiry + on partition heal.
• SLO upgrade: 99.99% entry decision availability; exit is effectively 100% (hardware-backed).

v3 — "Make it smart, safe, and compliant at scale" (9–18 months)

• ML-assisted tailgating detection on overhead camera (privacy-bounded: counts only, no identity).
• Emergency system integration: two-way with building management (BMS/BACnet) — fire panel, HVAC, elevator recall.
• Predictive capacity — forecast crowd peaks to preload leases hot gates.
• Compliance automation: auto-generated monthly occupancy reports for landlord / insurer; SOC 2 + local fire-code audit evidence exports.
• Dynamic capacity based on HVAC (reduce capacity when CO2 sensors trigger), parking (reduce when full), public-safety events.
• Per-zone occupancy (food court, cinema, department stores) — sub-counter hierarchy with per-zone leases.
• Multi-region active-active per mall (true two-region dual-write with CRDT-bounded conflict resolution using the same bounded-lease technique — v3-level trick that only applies when regional WAN is itself flaky).

10.1 Hardware-level interlocks and why software cannot be the root of trust

Building fire codes (NFPA 101, IBC, EN 13637 for electric locks in escape routes) require that exits fail safe on:

• Loss of power
• Loss of signal from control panel
• Manual emergency release ("panic bar") at the door itself
• Fire alarm

No amount of clever software can satisfy the code — the inspector checks the physical wiring. Our software architecture must be designed as a cooperating peer to the hardware interlock, not as its manager. This frames many decisions (e.g., FIRE mode in software is for auditing and alerting, not for enforcement; enforcement is in copper).

10.2 Privacy of entry/exit counts

Even anonymous counting has surveillance implications:

• Aggregate hourly counts at an abortion clinic, a church, a synagogue, a political rally → identifies individual visit patterns when correlated with other data.
• Mitigations:
  • Differential privacy noise on public-facing counts (dashboards shown to tenants or published externally).
  • Coarse bucketing of audit logs (day-level for external, per-event only for operator-accessible and with retention limits).
  • No biometrics, no face matching in counting pipeline.
  • Minimize retention: raw per-event audit in hot tier 30 days, aggregated in cold tier; long-tail liability windows use aggregates unless incident triggers hot-tier retention.
• This is relevant to the candidate's current role (Meta Privacy Infra): the same techniques (anonymization, DP, k-anonymity on aggregate release) apply.

10.3 Legal record retention and e-discovery

• In US, statute of limitations for slip-and-fall personal injury in malls = 2–4 years depending on state; for structural/fire events = 7 years (varies). 7-year retention is the industry default.
• Audit log must be immutable for evidentiary value (append-only + hash-chained or WORM storage). Kafka with compaction disabled + object-storage retention lock satisfies this.
• Design must support legal hold — ability to tag specific events as "do not delete past retention" for ongoing litigation.

10.4 Integration with building management systems (BMS / BACnet / Modbus)

• BMS protocols are not modern REST — they are BACnet or Modbus over serial/IP with weak auth.
• Our integration is one-way pull for safety signals (HVAC, CO2, temperature) — never trust write paths from BMS to be authenticated, so the occupancy system reads signals and makes its own decisions.
• Fire panel integration is specifically via hardwired dry-contact relays, not protocol-level — this isolates software vulnerabilities from life-safety actuation.

10.5 Sensor calibration and tailgating measurement

• Quarterly calibration: walk-through test with known count, compare against turnstile + optical counter. Discrepancy logged; if >0.5%, sensor flagged for service.
• Tailgating rate is a slowly-varying mall characteristic — depends on turnstile geometry, staff attentiveness, crowd density. Some malls see 0.05%, some see 0.4%. Per-gate learned baseline; flag drift.

10.6 Probabilistic counting vs exact counting — two parallel pipelines

• Exact (CP pipeline) for capacity enforcement, as designed above.
• Probabilistic (AP pipeline) for marketing & foot-traffic analytics — WiFi-probe counting, Bluetooth beacons, camera-based crowd estimation. Much cheaper, less precise, explicitly non-regulatory.
• These pipelines must be kept organizationally separate to prevent the marketing team's approximate counter from being used for capacity decisions. (L7: data lineage matters for compliance.)

10.7 Observability as first-class SLO surface

Operator-facing SLOs (internal, not customer-facing):

Over-admission counter as an SLO: this is the anomaly. Most system SLOs measure performance; an over-admission SLO measures correctness against a regulatory ceiling. It is measured by independent sensor reconciliation, not by the system's own self-report. This defends against the failure mode where the bug is in the code that counts.

10.8 Why "just use Spanner" is tempting but incomplete

Spanner solves the authoritative-counter problem at the write layer. It does not solve:

• Local decision latency during partition (Spanner doesn't exist in the mall).
• Exit-never-blocked (that's a hardware decision).
• Bounded divergence on partition (that's the lease model).
• Audit lineage & data retention (that's Kafka + tiering).
• Sensor error (that's calibration + dual-sensor).

An L6 answer says "Spanner." An L7 answer names Spanner as the hot row but lays the lease model on top because the problem is really about a distributed semaphore with hardware fail-safes and regulated correctness, not a distributed counter.

10.9 Analog to distributed locking services

The lease model is isomorphic to how Chubby / Zookeeper / etcd hand out leases for leader election, but inverted: instead of leasing a single resource (leader slot) to one client, we lease N abstract "admission tokens" to many clients. Each consumption is unilateral (no central round-trip). The lease is the contract. If the lessor (gate) dies, the contract expires and the slots revert. This is one of the fundamental distributed patterns, and recognizing that our counter is a semaphore with lease-based distribution (not a shared variable) is the conceptual unlock.

Compare to Kubernetes: kubelet holds a resource reservation from kube-scheduler; pods scheduled locally are deducted from the local reservation. If the node partitions, the scheduler reclaims the reservation (pod eviction). Exactly our model with different nouns.

10.10 What I'd sketch on the whiteboard if given 5 more minutes

• A reconciliation invariant:

  For all time t, for each mall:
      actual_count(t) == spanner_count(t)
                         + Σ(consumed_tokens in gate g) for each gate g
                         - Σ(exits_in_kafka_lag)
  
  When partitions heal:
      spanner_count += Σ(consumed_tokens from reconciled gates)
      Σ(consumed_tokens on gate) -= reconciled_amount
      effective_capacity -= spread across re-granted leases
  
  Invariant throughout: spanner_count + leased_out ≤ capacity
  

  This invariant is the formal proof that over-admission cannot occur under our assumptions. The assumptions that can break it:

  • Byzantine gate reporting (malicious firmware): mitigated by signed firmware + dual-sensor reconciliation.
  • Sensor miscounting (tailgating): bounded by SLO, accepted.
  • Clock skew in TTL reclaim: mitigated by making reclamation strictly additive and using hybrid logical clocks for ordering if we ever need total-order reasoning (we don't for capacity).

1. SRE pager-carryable? ✅ — Ops can diagnose via (a) per-gate decision_source metric, (b) fallback state machine diagram in runbook, (c) reconciliation invariant as a check. Primary pagers: entry_decision_p99, over_admission_counter, exit_reject_rate (any >0 is page), Kafka_audit_lag, spanner_commit_latency.
2. Every diagram arrow → real API/data flow? ✅ — table in §6.1 maps each arrow to the API in §4 or the data plane in §5.
3. Deep-dive L7 or L6? ✅ — §7.2 leased-slot includes the bias-toward-safety reclamation rule (the L7 move); §7.3 maps asymmetric CAP onto regulated hardware. §7.1 rejects Spanner-only, Redis-only, and FoundationDB with quantified trade-offs.
4. Exits-never-blocked as asymmetric code path? ✅ — §6.1 shows exit bypassing admission FSM; §6.4 explicitly calls out the asymmetry; §7.3 is a whole deep dive on it.

End of solution.