AEGIS
Closed-loop, local-first SRE platform for detecting container failures, diagnosing incidents, and executing policy-controlled remediation.
THE PROBLEM
Modern containerized systems fail through OOM conditions, timeouts, crash loops, port collisions, memory leaks, and permission errors. Detecting the failure is only the first step — automated systems must safely diagnose, evaluate policy, remediate, and audit without human bottlenecks.
Understand what failed at the container runtime level.
Determine the probable failure cause via local semantic classification.
Determine whether automated remediation is safe under strict policy.
Perform a deterministic, controlled Docker action.
Preserve the complete end-to-end evidence trail in durable storage.
THE APPROACH
Aegis uses a closed-loop architecture. Rather than passively alerting human operators or granting unbounded shell execution to LLMs, it connects detection to policy-gated remediation.
Watchman catches container death and tails stdout/stderr logs.
Publishes raw incident payloads into decoupled Kafka KRaft topics.
Generates 384-dim log embeddings and runs FAISS + MLP classification.
Validates confidence, risk level, and blast radius against safety gates.
Executes approved enum actions directly through Dockerode API.
Stores full incident context, embeddings, and execution receipts in MongoDB.
ARCHITECTURE & DATA FLOW
Air-gapped distributed topology running entirely within the host environment. Events flow asynchronously through Kafka KRaft into local neural inference and policy-enforced actuation.
Coordinates Docker Watchman events, Kafka producers/consumers, safety policy gates, and actuator calls.
ZooKeeper-free distributed event log for reliable incident streaming, decoupled buffering, and replay.
Local neural network pipeline: all-MiniLM-L6-v2 embeddings, sub-ms FAISS lookup, and PyTorch MLP classifier.
Persistent incident store, raw log history, embedding vectors, remediation plans, and offline RL replay buffers.
Direct Unix socket client executing enum-only container actions (restart, stop, inspect) without shell scripts.
FROM CRASH TO RECOVERY
Terminal execution trace showing the step-by-step progression of an incident from container death to audited recovery.
Docker daemon emits lifecycle death signal across local Unix socket.
Watchman listener filters out intentional stops and flags unexpected termination.
Recent stdout and stderr streams extracted into raw incident diagnostic context.
Asynchronous incident payload committed into partitioned Kafka event backbone.
NestJS orchestrator consumes event and routes diagnostic payload to local AI engine.
Python service returns classification label, confidence score, and suggested action.
Deterministic safety policy validates diagnosis against blast radius rules.
Orchestrator invokes scoped Dockerode restart call directly on target container.
Complete audit record, diagnosis vector, and actuation receipt stored in MongoDB.
TECHNICAL RIGOR
Concrete systems guarantees enforced through decoupled streaming, immutable persistence, and deterministic safety boundaries.
AI DIAGNOSIS PIPELINE
Aegis avoids non-deterministic generative LLMs in favor of a fast, local neural pipeline combining semantic embedding, vector similarity, and multi-layer perceptron (MLP) classification.
Preprocesses raw container stderr/stdout logs and converts them into normalized 384-dimensional dense semantic vectors.
Executes sub-millisecond local vector distance search over historical incident embeddings to find nearest prior failures.
Classifies the 384-dim log embedding into supported failure categories (OOM, Timeout, Crash Loop, Port, Perm, Leak).
Emits failure classification, calibrated confidence score, blast radius risk level, and suggested enum remediation action.
ENGINEERING CHALLENGES
Key architectural challenges encountered when designing a self-healing SRE platform operating in sensitive, air-gapped container environments.
AUTOMATED REMEDIATION WITHOUT RCE
Giving an autonomous system unrestricted shell access creates an unacceptable vulnerability if arbitrary commands enter the execution path.
AI UNCERTAINTY & FALSE DIAGNOSIS
A neural diagnosis can be wrong or low-confidence. Automated remediation on incorrect assumptions can amplify cluster instability.
EVENT DECOUPLING & LATENCY SKEW
Docker container death events occur in microsecond bursts, whereas local model inference has completely different latency characteristics.
ABSOLUTE AUDITABILITY
Every automated remediation decision, raw diagnostic log tail, vector similarity score, and Docker call must be fully traceable post-mortem.
AIR-GAPPED OPERATION
The entire platform must function without outbound internet access, cloud AI endpoints, or external model hosting infrastructure.
INDEPENDENT FAILURE RECOVERY
Kafka brokers, consumer orchestrators, target microservices, or MongoDB storage can fail independently without dropping inflight incidents.
ENGINEERING CHALLENGES & SOLUTIONS
Step-by-step engineering reasoning: Problem → Constraint → Engineering Response → Result applied across every critical reliability boundary.
AI-DRIVEN REMEDIATION WITHOUT RCE
AI model output should never become arbitrary shell execution on the host container environment.
Automated actions must remain strictly deterministic, scoped, and bounded.
Used an enum-only action registry (RESTART_CONTAINER, STOP_CONTAINER, IGNORE) mapped directly to explicit Dockerode Unix socket calls.
Zero AI-generated shell scripts enter the execution path. Remote code execution is architecturally impossible.
LOW-CONFIDENCE DIAGNOSIS & FLAPPING
Inaccurate or low-confidence diagnosis can trigger unsafe automation loops and container thrashing.
Remediation should execute autonomously only when classification certainty is mathematically high.
Enforced strict threshold gates: requires confidence >= 0.85 AND risk == LOW AND action == RESTART_CONTAINER. Otherwise, automated action is skipped and flagged for operator review.
Unsafe or uncertain actions fail-safe into human oversight rather than causing cascading container failures.
EVENT DECOUPLING & BURST RESILIENCE
Docker event capture should not block on AI inference or drop events during container failure storms.
Watcher latency must remain sub-millisecond even when neural networks take longer to compute.
Kafka KRaft acts as the distributed event backbone between the Docker Watchman and the NestJS orchestrator.
High-throughput event ingestion with zero dropped incident signals and decoupled consumer processing.
EXPLAINABLE & AUDITABLE POST-MORTEMS
Automated remediation must remain completely explainable and auditable long after execution.
Every input, neural vector, decision gate, and API execution receipt must be immutably preserved.
Persisted structured incidents, raw log tails, FAISS vectors, remediation plans, and execution receipts in MongoDB.
Complete forensic audit trail for every automated SRE event across the infrastructure.
TECHNICAL DECISIONS
Compact decision records detailing why each core technology was selected to guarantee air-gapped autonomy, strict isolation, and high performance.
Kafka KRaft
EVENT STREAMINGDurable event streaming and high-throughput decoupling between container event capture and neural processing.
MongoDB
STATE & AUDITFlexible document persistence perfectly suited for semi-structured incident payloads, raw log arrays, high-dimensional vector embeddings, and offline RL replay buffers.
Dockerode
CONTAINER RUNTIMEDirect Docker Engine Unix socket communication without spawning shell subprocesses, eliminating shell injection and command interpretation risks.
FAISS
VECTOR SEARCHUltra-fast local nearest-neighbor similarity search over historical incident embeddings without cloud vector database latency or cost.
SentenceTransformers
LOCAL EMBEDDINGSCompact, efficient local model (all-MiniLM-L6-v2) generating 384-dimensional dense semantic vectors entirely offline on CPU/GPU.
NestJS
CONTROL PLANEEnterprise-grade TypeScript modular architecture with strict dependency injection, separating watchers, event buses, safety policies, and actuators.
AUTOMATION WITH GUARDRAILS
Aegis separates diagnosis from actuation. AI recommends an action, but a deterministic policy engine evaluates safety thresholds before a single container is touched.
Neural classification confidence must be strictly ≥ 0.85. Low-confidence ambiguity defaults to safe inaction.
Risk rating must be strictly LOW. Destructive or stateful services are blocked from autonomous intervention.
Any gate failure aborts automated execution, persists full audit telemetry, and escalates to human on-call engineers.
SECURITY BOUNDARIES
Aegis enforces rigid execution firewalls between the probabilistic AI pipeline, the NestJS control plane, and the Docker container host.
No bash, sh, or exec wrappers. Eliminates command injection vectors completely.
Zero outbound network calls to third-party AI APIs; fully isolated from public internet.
Guaranteed reproducibility with deterministic embeddings and calibrated MLP output.
Hardcoded enum values (RESTART, STOP, IGNORE) bounding actuator scope.
Internal Docker bridge network with no external ports exposed for telemetry.
High-risk, stateful, or uncertain incidents require explicit human sign-off.
SYSTEM INTERACTION STATES
Deterministic incident state machine tracking every container transition from failure detection to audited recovery. Hover or click any state to inspect its operational invariant.
NestJS Orchestrator tests confidence >= 0.85, risk == LOW, and action == RESTART.
THE AUDIT TRAIL
Aegis is not simply a container restart script. It maintains an immutable, multi-dimensional MongoDB audit history preserving full incident context, neural vectors, and execution logs.
Sequential incident-state-action tuples captured to train offline reinforcement learning policies without production interference.
Point-in-time CPU, RAM, and container network metrics recorded before, during, and after remediation actions.
Transactional outbox pattern guaranteeing that every detected container incident is eventually delivered to Kafka even if brokers temporarily disconnect.
OFFLINE LEARNING LOOP
Aegis incorporates an offline Reinforcement Learning (RL) replay pipeline to analyze historical incident episodes and evaluate future remediation heuristics without live cluster risk.
- The RL engine runs strictly offline in Python as a batch research tool.
- It does NOT have direct access to the Docker socket and cannot directly restart or stop containers.
- It can NEVER bypass or override the deterministic NestJS safety policy gate.
CHAOS TESTING
Aegis ships with a built-in chaos engineering suite to intentionally inject container failure modes and verify closed-loop detection, neural classification, and policy gating.
$ aegis chaos oomInjects heap exhaustion$ aegis chaos timeoutTriggers socket stall$ aegis chaos crashForces panic termination$ aegis chaos portInduces port collisionSpawns a memory-greedy worker allocating unconstrained RAM until Linux kernel OOM-killer fires.
Injects deadlocks and hung HTTP keep-alive loops that exceed service health-check deadlines.
Forces unhandled runtime panics and fatal SIGSEGV exit codes to test immediate recovery.
Revokes host volume filesystem permissions to evaluate permission-denied classification.
Attempts binding already-occupied host TCP ports to test address-in-use detection.
MEASURABLE RESULTS
Grounded engineering outcomes built directly into the codebase and validated via chaos testing suites rather than speculative marketing numbers.
LOCAL-FIRST
Zero runtime dependency on cloud AI or external endpoints; all model inference executes on host silicon.
AIR-GAPPED
All services operate inside an isolated Docker bridge network with zero outbound internet egress.
AUDITABLE
Every raw crash log, FAISS vector distance, neural classification, and remediation action is persisted in MongoDB.
POLICY-GATED
Autonomous container actions strictly restricted by confidence (>= 0.85) and LOW blast radius risk thresholds.
DETERMINISTIC
Fixed enum-only action registry mapping directly to Dockerode calls, eliminating arbitrary command execution.
CHAOS-TESTABLE
Built-in failure injection harness actively verifying OOM, timeout, crash loop, and permission scenarios.
THE CLOSED LOOP
The complete autonomous lifecycle: capturing Docker failure signals, streaming through Kafka, computing local neural classifications, verifying safety gates, executing remediations, auditing state, and compiling offline learning buffers.
WHAT I WAS SOLVING
“I designed a closed-loop remediation system where AI can recommend an action, but deterministic policy controls whether that action is allowed to execute.”
Autonomous systems in production cannot rely on probabilistic model outputs alone. True infrastructure safety demands decoupling recommendations from execution, bounding actuators to fixed APIs, and keeping the entire loop air-gapped.
EXPLORE THE IMPLEMENTATION
Inspect the NestJS control plane, Kafka KRaft streaming consumers, Python FAISS/MLP pipelines, and Dockerode actuation modules on GitHub.
CarePulse
Healthcare Operations & Appointment Workflow Engine