Subscription Tracker
REST API for Subscription Lifecycle Management. REST API for subscription lifecycle management — tracking, automated reminders, and auditable state transitions.
THE PROBLEM
Modern compute workloads require reliable, multi-node execution across heterogeneous clouds without centralized bottlenecks. When compute instances experience transient network partitions or kernel freezes, distributed schedulers suffer from split-brain dual dispatch, worker starvation under load bursts, and unhandled task dropouts. Preserving deterministic state consistency and zero-loss recovery under real-world infrastructure failures is the core problem RunStack solves.
THE APPROACH
- // 01Centralized control plane decouples admission validation from physical execution.
- // 02Partitioned Kafka event log ensures durable, at-least-once workload buffering.
- // 03Concurrent min-heap scheduler computes placement using real-time node capacity scores.
- // 04Sliding-window heartbeats detect silent node stalls and trigger automatic requeueing.
- // 05Local Go host daemons isolate untrusted processes inside strict Linux cgroups v2 boundaries.
ARCHITECTURE & DATA FLOW
Decoupled cluster topology separating ingress validation, distributed message durability, concurrent placement scoring, and local kernel sandbox boundaries.
ENGINEERING CHALLENGES
The primary failure modes encountered when scaling autonomous execution loops across unpredictable cloud infrastructure.
Split-Brain & Lease Race Conditions
During transient network partitions, multiple scheduler instances could attempt to dispatch identical batch workloads to separate nodes, causing duplicate resource reservation and state corruption.
Sub-Second Node Eviction & Workload Rescue
Compute instances experiencing kernel stalls or silent network drops leave tasks orphaned indefinitely without notifying the control plane.
Goroutine Exhaustion Under Burst Traffic
Unchecked incoming dispatch streams spawn tens of thousands of goroutines, degrading Go garbage collection cycles and increasing latency.
Graceful Termination During Deployments
Restarting node daemons during maintenance cycles risks severing running container workloads mid-computation without proper state flushing.
TECHNICAL RIGOR
System guarantees enforced through disciplined runtime primitives, strict kernel constraints, and automated verification suites.
ENGINEERING CHALLENGES & SOLUTIONS
Step-by-step engineering reasoning: decomposing the problem, identifying the core invariant, implementing the fix, and measuring the outcome.
SPLIT-BRAIN & LEASE RACE CONDITIONS
Concurrent schedulers dispatched identical batch jobs to different nodes during transient network partitions.
Workload ownership must remain strictly mutually exclusive without central synchronous database locks.
Implemented distributed Redlock mutexes in Redis with deterministic 5000ms TTLs and conditional CAS state updates.
Zero duplicate executions recorded across 500 concurrent stress worker routines.
SUB-SECOND NODE EVICTION & WORKLOAD RESCUE
Frozen host kernels stalled assigned tasks indefinitely without emitting explicit failure signals.
Unhealthy nodes must be evicted quickly without false-positive flapping on momentary latency spikes.
Engineered a sliding-window failure detector on 1000ms ticks (3 misses = DEGRADED; 5000ms = EVICTED and requeued).
Sub-second node eviction window averaging 3.2s from silent crash to workload recovery.
GOROUTINE EXHAUSTION UNDER BURST TRAFFIC
Unchecked dispatch streams spawned tens of thousands of goroutines, degrading Go runtime garbage collection.
Scheduler memory footprint must remain constant regardless of incoming client ingestion bursts.
Adopted bounded worker pools with non-blocking channel selectors and drop-guard backpressure queues.
Constant memory envelope under peak RPS with zero goroutine leaks.
GRACEFUL TERMINATION DURING ROLLING RELEASES
Restarting node daemons during maintenance could terminate executing containers mid-computation, corrupting state.
Node supervisors must shut down cleanly without leaving orphaned container processes.
Trapped POSIX SIGTERM signals to enter a 30s drain mode: reject new leases, wait for tasks, flush receipts.
Zero-data-loss rolling deployments with deterministic process cleanup.
TECHNICAL DECISIONS
Why each technology was chosen, what operational trade-offs were accepted, and the verified result delivered in production.
Go (Golang)
CORE RUNTIMEPredictable low-latency garbage collection, zero-overhead binary packaging, and first-class goroutines/channels.
Explicit concurrency management requires disciplined bounded worker pool budgets and channel leak guards.
Sub-millisecond scheduling dispatch with a predictable, constant memory envelope under burst traffic.
Docker Engine API & cgroups v2
WORKER ISOLATIONHard kernel-level multi-tenant CPU throttling and memory ceilings preventing rogue customer workload exhaustion.
Requires local Unix socket permissions and strict host kernel cgroups v2 hierarchy support.
Rigid task sandboxes with automated POSIX SIGKILL runaway watchdog protection.
Concurrent Min-Heap Scheduler
PLACEMENT ENGINEO(log N) node capacity selection evaluating real-time memory envelopes and idle CPU metrics.
Scheduler min-heap requires reliable periodic heartbeat telemetry to maintain accurate scoring.
Instantaneous placement selection without head-of-line blocking or scheduling starvations.
Redis 7.x & Redlock Mutex
CONSENSUS & LEASESub-millisecond distributed lock acquisition with atomic SET NX EX semantics and deterministic 5000ms TTLs.
Demands tight clock synchronization (NTP) across nodes to prevent premature lease expiration.
Globally mutually exclusive task execution with zero split-brain duplicate dispatch.
Apache Kafka Event Bus
DURABILITY LOGPartitioned, persistent event log allowing historical replay during node recovery and guaranteed at-least-once delivery.
Higher operational overhead and broker infrastructure compared to ephemeral in-memory queues.
Zero dropped jobs during simulated 40% packet-loss network partition events.
SYSTEM INTERACTION STATES
Deterministic state transitions driving the workload execution loop. State is explicit, verifiable, and observable at every phase.
Job accepted by the control plane; SHA-256 idempotency signature verified against cache.
Buffered into partitioned Kafka log buffer; awaiting concurrent scheduler assignment.
Min-heap scores idle CPU and memory headroom, selecting optimal worker host.
Host daemon acquires Redis Redlock mutex lease (5000ms TTL) to lock execution.
Local agent spawns Docker container sandbox constrained by Linux cgroups v2 ceilings.
Container execution finishes; status code trapped; failures trigger exponential backoff retry.
Final receipt committed back to control plane and cached for 24-hour replay protection.
MEASURABLE RESULTS
Empirical reliability numbers derived from stress benchmarks, network failure injections, and race detection suites.
Zero race warnings across 500 simulated concurrent worker routines via `go test -race -count=100`.
Heartbeat sliding-window timeout identifies unavailable workers without blocking scheduler operation.
Zero dropped jobs recorded during simulated 40% packet-loss network partitions.
Deterministic request replay protection via SHA-256 signature cache in admission gateway.
FROM JOB TO RESULT
Summary lifecycle pipeline: end-to-end deterministic progression of every compute workload admitted to RunStack.
THE IMPLEMENTATION IS OPEN.
Inspect the Go source code, min-heap scheduler routines, Docker container hooks, and Redis Redlock lease implementation.
RunStack
Distributed Deployment & Job Orchestration Platform