Observability for AI Systems — With AI as the On-Call Engineer

Explore an open-source lab observing an AI inference gateway. Custom agents act as on-call engineers, investigating issues and generating RCA without human intervention.

Claude Agent SDK Python Grafana Loki Tempo

Overview

This started as a side project. I wanted to actually understand how observability works in modern AI infrastructure, and reading about it wasn’t cutting it, so I built something I could break on purpose. It’s a fully local, open-source lab: a small AI inference gateway that’s observed end to end (logs, metrics, traces, data lineage), and five custom Claude agents built on the Claude Agent SDK that investigate whatever goes wrong in it. There’s an RCA assistant, an incident reporter, an auto-fixer, a dashboard generator, and a runbook executor. The other thing I wanted to find out was whether custom agents can actually pinpoint root causes fast enough to be useful in day-to-day engineering, or if it just looks good in demos.
For the live demo I’ll break the system on purpose. A Grafana alert fires, an agent spawns, queries the telemetry itself, and writes a postmortem naming the root cause. No human in the loop. I’ll show the working system, the repo, and the agents’ own traces.

Links

https://github.com/ShahriarBijoy/Agent-Driven-Observability
Docker-Compose lab demonstrating AI observability using Grafana, OpenLineage, and Claude agents.

Tech stack

Claude Agent SDK

The Claude Agent SDK (Python, TypeScript) transforms Claude into an autonomous, production-ready agent by granting it controlled access to a computer: file operations, Bash execution, and external tool integration via the Model Context Protocol (MCP).

This SDK provides the core agent harness that powers Claude Code, enabling developers to build sophisticated, general-purpose agents. It offers official TypeScript and Python libraries for seamless integration. Agents gain powerful, safe capabilities: reading and writing files, executing sandboxed Bash commands, and leveraging a rich tool ecosystem. The framework includes essential production features (e.g., automatic context management, granular permission modes like `acceptEdits`, and built-in cost tracking). Use the open Model Context Protocol (MCP) to connect custom tools, databases, and APIs, extending Claude’s utility far beyond simple chat to complex, multi-step workflows.

https://docs.claude.com/en/api/agent-sdk/overview

View projects
Python

Python: The high-level, general-purpose language built for readability, powering everything from web backends to advanced machine learning models.

Python is the high-level, general-purpose language prioritizing clear, readable syntax (via significant indentation), ensuring rapid development for any team . Its ecosystem is massive: use it for robust web development with frameworks like Django and Flask, or leverage its power in data science with libraries such as Pandas and NumPy . The Python Package Index (PyPI) provides thousands of community-contributed modules, offering immediate solutions for tasks from network programming to GUI creation . The language is actively maintained by the Python Software Foundation (PSF), with the stable release currently at Python 3.14.0 (as of November 2025) .

https://python.org

View projects
Grafana

Grafana is the open-source platform for operational observability: query, visualize, and alert on metrics, logs, and traces from any data source.

Grafana is your mission control for data visualization and monitoring. This open-source analytics platform connects directly to over 100 data sources (Prometheus, Loki, InfluxDB, MySQL, etc.) without requiring data ingestion. You build dynamic, flexible dashboards using customizable panels (graphs, heatmaps, tables) to transform raw metrics and logs into actionable, real-time insights. The built-in alerting engine allows you to define rules visually and send notifications to critical channels (Slack, PagerDuty, email). Use Grafana to consolidate your entire observability stack: see infrastructure performance, application health, and user behavior in a single pane of glass.

https://grafana.com/

View projects
Loki

Grafana Loki is a horizontally scalable, highly available log aggregation system designed to store and query application logs cost-effectively by indexing metadata instead of full text.

Loki takes a highly efficient approach to log management by indexing only the metadata (labels) of your log streams rather than parsing and indexing the full text of every log line. Inspired by Prometheus, this design allows engineering teams to reuse the same labels across metrics and logs, enabling seamless navigation between dashboards and raw logs within Grafana. By compressing log payloads and storing them in cheap object storage (like Amazon S3 or Google Cloud Storage), Loki eliminates the massive storage overhead and complex indexing schemes common in traditional logging stacks, delivering fast writes and cost-effective operations at petabyte scale.

https://github.com/grafana/loki

View projects
Tempo

An AI-powered visual editor for React that lets designers and developers build, style, and deploy production-ready web apps directly from the codebase.

Tempo bridges the gap between design and engineering by serving as a visual IDE for React. Instead of exporting static mockups or messy code, teams can use natural language prompts to generate clean components, edit layouts visually in a Figma-style canvas, and write production-ready code simultaneously. The platform integrates directly with tools like GitHub, Vercel, Supabase, and Tailwind CSS (allowing one-click deployments and seamless backend connections). By keeping design changes and code in sync, Tempo eliminates the traditional handoff bottleneck and helps teams ship functional web and mobile apps 10x faster.

https://www.tempo.new

View projects

Observability for AI Systems — With AI as the On-Call Engineer

Related talks