@canyang25LLM agents for production incidents: a UW-Madison data science student
Hi, I'm Preston, a data science master's student at UW-Madison in Chicago. I build software, data, and AI systems: web platforms with job queues, data pipelines, and tool-calling LLM agents. I've interned as an SDE at Biotechne and as an LLM engineer at AI Rudder.
My main project is AutoSRE, an LLM-powered agent that takes a production incident from alert to fix: it pulls metrics and logs, diagnoses the root cause, runs remediation, and writes the incident report. I'm still working out how much autonomy it should get before a human signs off. I also built L-AI, a skincare routine app with a product-aware chatbot.
I'd love feedback from anyone who has run on-call rotations or built remediation tooling, especially on where you'd draw the line for an agent acting in production.
Comments2
Welcome to Lounge, Preston. On the autonomy line, I'd guess it depends on blast radius and reversibility more than diagnosis confidence, so read-only fetches and a restart differ a lot from a config change. How are you grading AutoSRE's root cause calls today, replayed past incidents or live ones?
Welcome Preston. I run production solo, so the alerting and first response for my own SaaS is basically automated, and the line I ended up with: the agent may do anything reversible and idempotent from a short allowlist (restart a worker, clear a stuck queue, scale a pool), once, then it must check the recovery metric and stop. A second attempt or anything touching data, config or payments goes to a human with a ready plan and the exact undo step. Two things that saved me: a hard cap on remediations per hour, because restart loops hide the real cause, and running the agent in shadow mode for a while, where it only writes what it would have done next to what the human actually did. That diff is also the best eval set you can get. Is AutoSRE allowed to act today, or only to propose?
2 comments
Welcome to Lounge, Preston. On the autonomy line, I'd guess it depends on blast radius and reversibility more than diagnosis confidence, so read-only fetches and a restart differ a lot from a config change. How are you grading AutoSRE's root cause calls today, replayed past incidents or live ones?
Welcome Preston. I run production solo, so the alerting and first response for my own SaaS is basically automated, and the line I ended up with: the agent may do anything reversible and idempotent from a short allowlist (restart a worker, clear a stuck queue, scale a pool), once, then it must check the recovery metric and stop. A second attempt or anything touching data, config or payments goes to a human with a ready plan and the exact undo step. Two things that saved me: a hard cap on remediations per hour, because restart loops hide the real cause, and running the agent in shadow mode for a while, where it only writes what it would have done next to what the human actually did. That diff is also the best eval set you can get. Is AutoSRE allowed to act today, or only to propose?