RunStack

RunStack uses a trusted Control Plane as the authoritative source of state, while Agents remain thin execution workers.

Owns authoritative state, node registrations, deterministic scheduling, and application reconciliation loops.

Discovers the host node, maintains heartbeat, polls and claims assigned work, executes tasks, and reports results.

Provides strict process and container lifecycle isolation through Docker or Podman on the host.

RunStack couples an authoritative control plane with lightweight worker agents over HTTP. State, scheduling, and reconciliation remain strictly centralized, while agents manage local execution.

Provides REST endpoints for job admission, app deployment manifests, node registrations, polling, and heartbeat reporting.

Maintains authoritative in-memory state (V1) protected by sync.RWMutex: Node, Job, Application, Deployment, and Instance stores.

Dispatches jobs and instances to healthy online nodes using a sorted node ID ring and a persistent round-robin cursor.

Continuous control loop comparing desired instances from immutable deployments against actual runtime instance status.

Autonomous host daemons that report capabilities, pulse 1000ms heartbeats, claim work atomically, and run tasks locally.

Directly drives Docker or Podman on the host node for strict process isolation, CPU/memory quotas, and exit code extraction.

Jobs transition through strict, centrally validated states. The scheduler assigns pending jobs to online nodes, agents atomically claim the execution, and failures evaluate against a deterministic retry budget.

A JobID identifies the logical task, while an ExecutionID identifies a specific physical attempt. Separating logical intent from execution identity prevents zombie workers from corrupting cluster state after node recovery.

Physical exactly-once execution is theoretically impossible during network partitions. RunStack guarantees execution-aware result fencing: late results from severed nodes are safely dropped, preserving state correctness.

Four fundamental distributed systems challenges encountered when coordinating uncoordinated compute workers across network boundaries.

System guarantees enforced through disciplined concurrency primitives, explicit state guards, and automated verification suites.

Disciplined systems reasoning: Problem → Constraint → Design → Outcome applied across every failure boundary.

Nodes register on startup and maintain an active heartbeat loop with the Control Plane. Missed heartbeats trigger automated offline transition and workload recovery.

Architecture Note: Node capabilities are collected upon registration for cluster visibility, but the V1 scheduler uses deterministic round-robin placement rather than dynamic resource scoring.

RunStack orchestrates long-running services through declarative specs. An Application defines desired state, a Deployment captures an immutable snapshot, and Instances represent physical containers.

RunStack strictly separates Instance Status, Health Probes, and Host Node State. Infrastructure drops and application failures are evaluated independently by the Reconciler.

Deliberate engineering trade-offs: why each architectural path was chosen, the operational costs accepted, and the delivered outcome.

Combined cluster lifecycle linking job state machines to node heartbeat availability. Node disappearance directly feeds into job recovery and rescheduling loops.

RunStack extends beyond basic process spawning. The agent interfaces with the local container runtime (Docker or Podman) to drive the full container lifecycle, streaming logs, checking health, and feeding telemetry into the Reconciler.

RunStack isolates operator intent from worker execution. The Control Plane acts as the sole security boundary, enforcing ownership validation and rejecting untrusted state mutations.

Strict boundary between cluster operators (deployments, routing, secret references) and worker agents (claims, execution, heartbeat).

Only the node designated by the scheduler can claim and report on an assigned execution attempt.

Every attempt carries an ephemeral ExecutionID token. Stale results from severed or timed-out workers are rejected by the Control Plane.

Secret references are stored securely in the Control Plane and resolved just-in-time into agent container environment injection.

Commands are parsed directly via strings.Fields() into exec.Command arguments, eliminating arbitrary shell injection vulnerabilities.

Guarantees cluster state immutability once a job reaches terminal SUCCEEDED or FAILED status, or when an ExecutionID is superseded.

Verifiable engineering outcomes built directly into the system architecture, avoiding speculative or ungrounded benchmark claims.

Engineering rigor requires documenting what a system does not guarantee. Explicit V1 constraints directly guided the design of the PostgreSQL V2 persistence foundation.

Correctness in distributed systems requires automated enforcement. RunStack standardizes verification through a comprehensive make check pipeline.

Validates all registry sync.RWMutex locks and channel communications under simulated parallel access.

Tests agent SIGINT/SIGTERM cancellation, completing in-flight jobs and unregistering cleanly.

End-to-end execution testing verifying client submission, agent claim, Docker container run, and result receipt.

Verifies database migrations, transactional rollbacks, SELECT FOR UPDATE locks, and SQL fencing.

End-to-end execution pipeline: the lifecycle of a compute workload from HTTP admission, deterministic scheduling, and atomic agent claim through to fenced completion and event history.