a Local-First AIOps Platform

Introduction

Building a local-first AIOps platform is not only about monitoring services. It is about creating an infrastructure layer that can observe system behavior, detect failures, understand crash patterns, and support recovery decisions without depending on external cloud platforms.

Project Aegis was designed with a distributed systems mindset. Instead of relying only on centralized dashboards, the platform streams telemetry events through Kafka, processes incidents asynchronously, and creates a modular recovery pipeline that can evolve over time.

The stack combines Kafka, Docker, NestJS, Redis, MongoDB, Python workers, and reinforcement learning concepts to create a practical foundation for crash detection, telemetry streaming, chaos testing, incident classification, audit logging, and controlled recovery workflows.

The Problem

Modern backend systems are rarely simple. A single platform may contain APIs, background workers, queues, databases, caches, containers, logs, and event streams. When something breaks, the real cause is not always visible immediately.

Traditional monitoring tools are useful, but many of them stop at alerting. They tell engineers that something is wrong, but they do not always explain why it happened, how it affected other services, or what recovery action should be taken next.

[ERROR] api-gateway timeout after 5000ms
[WARN] redis queue depth exceeded safe threshold
[ERROR] worker-service disconnected from kafka broker
[WARN] repeated restart detected for demo-crash-service
[INFO] recovery action requested: restart_container

In real systems, failures often move across services. Aegis was designed to make those failure chains visible, structured, and easier to act on.

System Design / Approach

Project Aegis follows an event-driven architecture. Kafka acts as the central communication layer for telemetry and crash events. Every important failure or system signal is converted into a structured event and published to a topic.

Independent services consume those events for classification, audit logging, operational state updates, and recovery planning. This keeps the system modular, replayable, and easier to extend without tightly coupling every component.

Failure Signal
    ↓
Telemetry Collector
    ↓
Kafka Topic
    ↓
Incident Consumer
    ↓
AI Classifier
    ↓
Audit Store
    ↓
Recovery Decision Layer
    ↓
Controlled Remediation
    ↓
Verification and Feedback

Architecture Overview

The architecture follows a failure-to-recovery pipeline. A service fails or a chaos event is triggered. The signal becomes telemetry, telemetry becomes an incident, and the incident moves through classification, decision-making, and recovery tracking.

┌─────────────────────┐
│ Demo Crash Service  │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│ Telemetry Collector │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│ Kafka Topic         │
│ aegis.telemetry.*   │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│ NestJS Orchestrator │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│ Python AI Worker    │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│ Recovery Decision   │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│ Audit and Feedback  │
└─────────────────────┘

This separation keeps the system flexible. Kafka handles event flow, NestJS manages orchestration, Python handles classification, MongoDB stores audit history, Redis keeps fast operational state, and Docker provides a reproducible local runtime.

Implementation

Step 1: Bootstrap Local Infrastructure

Docker Compose was used to run the full infrastructure locally. This made testing easier and created a reproducible environment for Kafka, Redis, MongoDB, the API, and demo crash services.

services:
  kafka:
    image: apache/kafka
    ports:
      - "9092:9092"

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

  mongodb:
    image: mongo:6.0
    ports:
      - "27017:27017"

  api:
    build: .
    depends_on:
      - kafka
      - redis
      - mongodb

Local infrastructure makes the platform easier to test, reset, debug, and run without external dependencies.

Step 2: Publish Structured Telemetry Events

Every crash, timeout, memory issue, or container anomaly should be represented as a structured event. Structured events are easier to classify, replay, search, and audit.

await producer.send({
  topic: "aegis.telemetry.crashes",
  messages: [
    {
      key: event.service,
      value: JSON.stringify({
        eventId: event.id,
        service: event.service,
        type: event.type,
        severity: event.severity,
        timestamp: new Date().toISOString(),
      }),
    },
  ],
});

Structured telemetry turns raw failure signals into useful operational data.

Step 3: Consume Kafka Events Independently

Consumers can subscribe to telemetry topics and handle events without blocking the producer. This allows classification, auditing, alerting, and recovery workflows to evolve separately.

await consumer.run({
  eachMessage: async ({ topic, message }) => {
    const event = JSON.parse(message.value?.toString() ?? "{}");

    await incidentService.handleCrashEvent({
      topic,
      event,
    });
  },
});

Independent consumers make the telemetry pipeline more scalable and easier to extend.

Step 4: Store Operational State in Redis

Redis is useful for fast, temporary operational state such as active incidents, retry counters, recent service health, and short-lived telemetry snapshots.

await redis.set(
  `incident:${event.eventId}`,
  JSON.stringify({
    service: event.service,
    type: event.type,
    severity: event.severity,
    status: "active",
  }),
  "EX",
  60 * 30
);

Redis keeps recent operational state fast to read without replacing the audit database.

Step 5: Store Audit History in MongoDB

Aegis needs traceability. Every incident, classification, recommendation, and recovery action should be stored for review, debugging, and future model improvement.

await auditModel.create({
  eventId: event.eventId,
  service: event.service,
  incidentType: event.type,
  severity: event.severity,
  decision: recommendation.action,
  createdAt: new Date(),
});

Audit storage makes the platform trustworthy because operational decisions can be reviewed later.

Step 6: Add Controlled Chaos Testing

Chaos testing endpoints simulate failures such as memory pressure, timeouts, service crashes, and port conflicts. This makes the telemetry and recovery pipeline testable under controlled conditions.

@Post("/chaos/:type")
async triggerChaos(@Param("type") type: string) {
  return this.chaosService.trigger({
    type,
    timestamp: new Date().toISOString(),
  });
}

Controlled chaos helps verify whether the system can detect, classify, and respond to realistic failures.

Step 7: Classify Incidents with Python Workers

Python workers process telemetry events and classify incidents into operational categories such as memory pressure, network delay, service disconnect, or unknown failure.

def classify_incident(event):
    if event["type"] == "timeout":
        return {
            "incident": "network_delay",
            "severity": "high",
            "recommended_action": "inspect_service_latency"
        }

    if event["type"] == "oom":
        return {
            "incident": "memory_pressure",
            "severity": "critical",
            "recommended_action": "restart_container"
        }

    return {
        "incident": "unknown",
        "severity": "medium",
        "recommended_action": "manual_review"
    }

Classification turns raw telemetry into operational meaning that the recovery layer can use.

Step 8: Recommend Recovery Actions

Recovery decisions should be controlled and explainable. The system should recommend actions based on incident type, severity, history, and confidence.

if (incident.severity === "critical") {
  return {
    action: "restart_container",
    service: incident.service,
    reason: "Critical incident detected with high recovery confidence",
  };
}

return {
  action: "manual_review",
  service: incident.service,
  reason: "Incident requires human inspection",
};

Recovery actions become safer when they include context, reason, and confidence instead of blindly executing remediation.

Step 9: Execute Remediation Safely

Automated remediation should be guarded. Restarting containers or changing service state should only happen when the action is allowed, logged, and recoverable.

if (decision.action === "restart_container") {
  await dockerService.restartContainer(decision.service);

  await auditService.recordRecovery({
    service: decision.service,
    action: decision.action,
    status: "executed",
    executedAt: new Date(),
  });
}

Safe remediation requires execution control, audit logs, and post-action verification.

Step 10: Add a Developer CLI

A custom CLI improves developer experience by turning repeated infrastructure workflows into clear commands. It can manage the cluster, trigger chaos tests, stream telemetry, and inspect system health.

aegis cluster up
aegis cluster status
aegis chaos timeout
aegis stream
aegis doctor

CLI commands reduce manual setup friction and make local infrastructure easier to operate.

Step 11: Stream Live Telemetry in the Terminal

A headless platform needs strong terminal visibility. Live telemetry streaming makes Kafka events, service state, crash patterns, and recovery actions visible without a frontend.

[aegis.telemetry.crashes]
service=auth-service type=oom severity=critical
service=api-gateway type=timeout severity=high
service=worker type=disconnect severity=medium

[recovery]
service=auth-service action=restart_container status=executed

Terminal streaming keeps the system observable while preserving the headless architecture.

Step 12: Verify Recovery Results

Recovery should not end after an action is executed. The system should verify whether the service actually became healthy again and record the result.

const health = await healthService.check(decision.service);

await auditService.recordVerification({
  service: decision.service,
  healthy: health.ok,
  checkedAt: new Date(),
});

Verification closes the loop and helps measure whether remediation actually improved the system.

Why Local-First Matters

A major design decision in Project Aegis was keeping the platform local-first. Many monitoring and AI-based tools depend on external cloud dashboards, hosted APIs, or third-party inference services.

Aegis is designed to run locally so telemetry, failure data, audit logs, and recovery experiments stay under direct control. This makes the platform better suited for restricted environments, academic experimentation, offline testing, and privacy-sensitive infrastructure research.

Trade-offs

Real-World Impact

What I Learned

• Monitoring becomes more powerful when telemetry is structured and event-driven.
• Kafka is useful when multiple services need to react to the same failure signal.
• Local-first infrastructure gives more control over telemetry, privacy, and experimentation.
• Chaos testing is important because recovery logic needs realistic failure input.
• Automated recovery must be controlled, auditable, and verified after execution.
• AI incident classification is only useful when event quality and context are strong.
• A headless infrastructure platform needs excellent CLI output and operational logs.

Conclusion

Project Aegis demonstrates how Kafka, Docker, NestJS, Redis, MongoDB, Python workers, and AI workflows can work together to create a resilient local-first AIOps platform.

The project combines observability, chaos engineering, telemetry streaming, incident classification, audit logging, and controlled recovery into a practical infrastructure experimentation system.

The key lesson is simple: self-healing systems should not blindly automate recovery. They should observe clearly, classify carefully, act safely, and leave a traceable record of every decision.

SYSTEM STATUS
─────────────
Kafka Broker        ✅ Online
Redis State Layer   ✅ Online
MongoDB Audit Store ✅ Connected
AI Worker           ✅ Running
Telemetry Stream    ✅ Active
Recovery Engine     ✅ Healthy