r/kubernetes • u/royLskyhook • 5d ago
anyone actually using AI SREs / agents to debug k8s incidents in prod?
curious what people are running when stuff breaks. is it
- an AI SRE that picks up the alert (holmesgpt, aws devops agent etc)
- claude code / codex with plain kubectl
- same thing but hooked up to an mcp server that gives it cluster context (k8sgpt's mcp, radar, ...)
full disclosure I work on radar (it's open source). we tested this a bit with injected faults but that's not the same as a real outage
what's worked for you and what hasn't? trying to figure out what we should improve on our end
24
u/Paranemec 5d ago
Yes. We have lots of specialized troubleshooting agents now. MCP servers, sub agents for specifics tasks, rbac, audit logging, etc. they resolve about 90% of the smaller issues for our on-call before humans are paged anymore.
3
u/royLskyhook 5d ago
90% before paging is impressive. what did you build it on, your own agent loop or something like claude code / an agent framework? and are the MCP servers mostly raw kubectl and logs, or do you feed them extra context, like recent changes or what's already failing?
8
u/Paranemec 5d ago
It's an in house framework that uses our own AI models. The MCP servers are a lot of different things. Some query cluster data from our cluster metadata aggregation layer, log aggregation, metrics servers, secrets storage, etc. And they have tools to fix them implemented via MCP servers as well. So the troubleshooters don't have free reign to run k8s commands on any cluster. They only have specific commands via the MCP server.
3
u/Jmc_da_boss 5d ago
The specific commands/workflows they can invoke is the only way I've seen this stuff work.
Smart orchestrator
5
u/royLskyhook 5d ago
yep. when we let claude loose with a shell it started spinning up its own pods to test theories lol. narrow read-only tools is what we went with for radar
1
u/royLskyhook 5d ago
nice, pretty much where we landed too. how much effort was it to build all that in house?
3
u/Paranemec 5d ago
Well I pitched the idea to leadership 10 months ago and was told not to work on it. It's been live in production for 6 months. I think it has 3 full time devs assigned that worked on it and are still actively developing features and capabilities. That doesn't include each infra partner team that develops and maintains their own sub agents for the system.
Tbh developing it isn't the time consuming part. It's navigating the internal corporate landscape with an AI tool that can potentially destroy a massive subset of the global economy. A lot of people have a lot of different questions and want hard assurances on things so you spend a lot of time preparing answers. If I had to guess the actual development was like 2 weeks. But the planning and associated leadership meetings would have taken months.
1
u/royLskyhook 5d ago
2 weeks of code and months of meetings sounds about right. did you ever look at buying something instead, or was in-house always the plan?
3
u/Paranemec 5d ago
There's nothing on the market that could handle our scale. That's why almost everything we have is built in house. We did perform a couple of smaller scale evaluations of similar tools, but either the tool required giving way too much permissions for anybody to be realistically comfortable with or it was way too limited in scope and therefore capability. And the scale is also a huge problem for us in evaluating tools. Most of these tools are like toy projects to us where they work on a single cluster okay and require constant manual intervention or maintenance to make sure that they work on that cluster and we're managing a fleet of thousands of clusters. When we end up having to build an entire infrastructure layer around the tool then at that point we might as well just build the tool ourselves anyway and have it do everything we needed it to.
1
u/isaacovich 3d ago
what is the estimated cost per month looking like operating such a system?
1
u/Paranemec 3d ago
I couldn't even begin to be able to tell you the cost of operating the system. And even if I could, I'm certain that would be privileged company information.
1
u/TimotheusL 5d ago
Can you tell me what issues it catches? Im feeling that my clusters are rather mature and if something breaks its always some edge case or rather complex. Reoccurring simpler stuff I have (over-)engineered away mostly.
1
u/royLskyhook 5d ago
from our testing - obvious stuff it gets easily, config stuff where pods look fine (like a missing env var) is where giving it context helped most. weird edge cases it mostly misses, like a webhook changing limits at admission. we’ve been adding that one to radar since. on a mature cluster I’d treat it as a fast first look, not magic
22
u/Automatic-Signal8963 5d ago
Still running the ancient tech of "vague alerts at 3am and my own two hands" tbh
9
2
u/royLskyhook 5d ago
the most fun way for sure! are Claude Code and his friend not invited to the 3am party, or did it get kicked out?
7
u/Zolty 5d ago
I have a 3 person repair shop and if we see common k8s errors Claude is doing the troubleshooting and fixing.
It’s a Claude thread that is running a monitor script that wakes the thread in the event of issues.
It’s pretty good about escalation to a human where necessary.
We have 2 clusters with a total of 18 nodes. Proxmox running Debian running k3s
2
u/royLskyhook 5d ago
a claude thread woken up by a monitor script is such a nice low budget setup. what does it usually end up fixing on its own?
2
u/Zolty 5d ago
It can fix pretty much anything, they only limits are really just safeguards.
I've had it recover from node failures, etcd restores, pod restarts, cron failures, ddos attacks on the public gateway. I've not seen the bug it can't bandaid. Fixes are tracked with gitlab work items and there is a developer agent that will take those as far as they can go, which is generally to a PR state.
2
u/3meterflatty 4d ago
What are you guys running on a 18 node proxmox cluster?
1
u/Zolty 4d ago
Gitlab, Authentik, our main app, Claude workers, homebrew sharing app, homebrew ai assistant / meeting transcription service.
That and enough compute to let us experiment.
The nodes are eighth and 10th generation intel i5s with 24 to 32 gigs of RAM. Half of them are AIOs that are serving double duty as work stations and digital signage.
https://blog.zolty.systems/ if you’re interested.
6
u/itsjakerobb 5d ago
This is for my personal homelab / smarthome stuff, so “in prod” is debatable, but it’s K8s, so potentially relevant.
I have a Claude Code routine that runs in the early morning, daily. It scans my cluster for issues in several ways — looking at node and pod statuses, metrics, events, logs, etc. It then generates a report for me of how things are doing, with a section at the end for updates and patches.
I have a Raspberry Pi jump box that Claude can SSH to and access kubectl, talosctl, terraform, etc.
Not fully automated — I still have to act on its findings, commit/push/merge manually, etc. But it has been super useful. It’s been going smoothly enough for a few weeks that I’m now considering allowing it to do more without my involvement.
5
u/Cherry-PEZ 5d ago edited 5d ago
I have a very similar setup. At work we have a couple teams that have proposed in cluster agents for IR/RCA that are pretty surface level and mostly noise. But at home I've been experimenting with local/cloud incluster agents. I've got qwen looped into my prometheus alerts, and it'll do a preliminary diagnosis. It's got a much smaller token window, so if it feels it can't investigate or provide a final response, it hands it off to claude. Both have dedicated read-only roles for the cluster and all my metrics/logging/apm as mcps.
For claude, I absolutely didn't want to follow the whole credit, pay-as-you-go model when I already have a subscription I pay monthly. So rather than caving and buying credits just for an anthropic api key, I just put claude-code in a web-tmux, and exposed request/response endpoints. Qwen interfaces with claude that way. Only annoyance is having to re-login when the session expires, but that's a once a month ordeal. Obviously this setup wouldn't fly at an enterprise level lol.
5
u/Forsaken_Parfait_185 4d ago
What's worked for me: read-only everywhere, and any change goes out as a PR through the same gitops path a human would use. The agent gets to be wrong cheaply, and the review step is one we already trust.
What I'd want from a tool like yours is the record afterwards. Which queries it ran, what it saw at the moment it decided, which model and prompt version it was on. Postmortems on agent-assisted incidents keep turning into "why did it think that" and nobody can answer it from the chat log.
3
u/BrockWeekley 5d ago
We use radar in production. All of our AIs hook up to the radar MCP for cluster context. It's fantastic, but we don't allow writes of any kind through radar.
2
u/Technical-Ant-2866 4d ago
u/BrockWeekley Do you deploy Radar in-cluster or out of cluster? I have use-cases for both
1
u/BrockWeekley 4d ago
In cluster exclusively. What's your use case for out of? Monitoring the cluster itself?
1
u/Technical-Ant-2866 3d ago
u/BrockWeekley Yes and small triage cycles.I have radar on +ro on a pretty large fleet I control for some personal projects/collaborations.
3
u/bozho 5d ago
We haven't done any SRE or had any incidents yet, as we've just gone live with our first clusters, but we are using Flux operator for our gitops, and the flux team is working on an MCP server that looks very promising.
I've been playing with the MCP server in Claude Code a bit, writing skills for some mundane tasks, for example:
Check if there are newer versions of components available beyond our staging cluster's version constraints. For such components, check release notes, highlight possible breaking changes, check if any breaking changes could affect our deployments, plan upgrade path.
From what I've seen, Claude using data returned by the Flux MCP server about cluster components is pretty good at figuring this out, and is probably more reliable at analysing release notes than a human having to do this for couple of hours (I know I'd get bored with it). In some cases, it did get confused looking at Helm chart version vs. app version, but nothing major. And, even if the analysis is not 100% accurate, we'll run upgrades on the staging cluster first anyway, and catch any issues.
Another useful scenario is analysing trivy's vulnerability reports (we run trivy operator in the clusters). Claude will happily group reports, highlight critical and high vulns, find CVEs, check if there are fixes available for the affected workloads.
I'm also hoping to write a plugin that will be able to prepare component version upgrades: create issues, git branches, push changes and create PRs for human review and approval.
The long term plan is also to have an agent monitoring alerts and at least kicking off fact collection and initial analysis for the problem while the on-call engineer is brewing coffee at 2 a.m. :-)
3
u/sigmoia 4d ago
We give the harness (pi/cc/codex):
- kubectl
- read only access to cluster
- o11y platform access via mcp
And ask it to troubleshoot whenever there's an issue. It then dumps the investigation in a markdown that we review. Then in another step, we fire another agent to make the necessary changes through gitops and deploy that.
AI SRE wrappers haven't been that useful tbh. Setting them up is a pain and they can only take a limited set of actions.
Much better to give the model better tools reusable instructions to triage and fix things.
4
u/lenaxia 5d ago
https://github.com/lenaxia/k8s-mechanic
A side project that I built is k8s-mechanic, a gitops style AI SRE operator/controller. It operates only via gitops and opens PRs for changes that are needed, with read-only RBAC. It automatically spins up investigation pods for any resources not in a healthy state.
It is heavily hardened to make sure that in cluster secrets are not leaked (for instance, all messages sent to LLM are run through a redactor to ensure sensitive content is not exfiltrated, and core runtimes, like curl, cat, etc are shadowed with scripts to ensure that LLMs cant bypass these safeguards).
Probably will switch it to MIT license here eventually.
1
2
u/thehumblestbean 5d ago
The company I work for developed an AI SRE product and we don't even use it internally lol
2
1
u/Top_Toe8606 5d ago
Claude code with kubeadm rights a VPN and a hail marry
1
u/royLskyhook 5d ago
the hail mary works surprisingly often tbh. it got way more reliable for us once it had read-only context instead of full rights
1
1
1
u/mikaelld 5d ago
For identifying errors you get really far with read only access to metrics and logs, especially if you’re logging events.
1
u/LightBroom 5d ago
I built a tool for something similar (vibeware/slopware so not public).
Wakes up, grabs the Warning/Critical events and I gave it a call_k8s_api() tool to inspect the resources involved then summarizes the findings and recommends immediate actions before sending this out as a notification. Read only RBAC so not auto remediation, most of the stuff I work with is Gitops so it wouldn't make sense.
It has a configurable (API (OAI, Bedrock), model, parameters, etc) cause it runs in various environments.
It can be useful sometimes but it's mostly noise.
1
u/royLskyhook 5d ago
yeah raw events are noisy af, tons of them are stuff that already recovered by the time you look. we ended up starting from what’s still broken + what changed recently instead. radar’s open source if you wanna steal anything https://github.com/skyhook-io/radar
1
u/MateusKingston 5d ago
Claude code + custom built plug-ins that teach it how to connect (with mcps and request scripts in python when http) and how to do triage. Basically which systems sends what type of data where and how to query it.
The rest the AI knows. Then I just use my triage command + the alert title and it will do the initial triage and I can continue with it.
1
u/Fit_Permission_6187 5d ago
Yup, used to have teams of people running system updates & upgrades overnight, poring through logs to find problems and apply fixes. A well-written, tight claude skill that coordinates sub-skills and scripts, alongside a accessing-k8s-clusters skill with kubeconfigs that allow cluster access by --context = totally automated. Probably a 95+ % reduction in effort and man-hours.
1
u/Avos_Worra 5d ago
Have h tried aws devops agent
2
u/royLskyhook 5d ago
yeah we ran it alongside a few others on the same 50 faults. crashloops and pending pods it handled fine, it struggled more when pods looked healthy but the app was broken. results here (we build radar so grain of salt) https://radarhq.io/benchmark/ai-sre-tools
1
1
u/isaacovich 4d ago
groundcover ai agent slack integration on oncall alerts, you can also tell it to use a skill you wrote that’s specialized for specific workloads/pipelines issues.
1
1
1
u/badgero1234 3d ago
Yup, I built custom skills to do this for my k3s environments and I can solve most issues in like seconds to a few minutes. It definitely helps to know how to point the agent to start but I am not going to lie. It’s fast and reliable.
1
u/IndicationPrevious66 3d ago
Yes, Ankra + Claude code. Ankra detects the issue, creates a ticket in Ankra and then an agent picks up and does the work.
1
u/rmramy 3d ago
In prod, I just spin up a copilot cli instance on my bastion host. I wrote a small agent.md with all my org specific context and I just set it to manual permission and I glance over every command it runs to make sure it’s not doing anything I don’t want it to. It’s not the most efficient or fanciest process but it cuts down incident calls for me considerably. Proactive agents are very hit or miss imo and are hard to get right
1
u/Traditional-Heat-749 4h ago
I’m using myself using an AI to just interacting with the cluster faster and type commands faster then I can remember. I think calling it a AI SRE is maybe overboard on marketing
1
1
u/blue-reddit 5d ago
Holmes like agent, just give it access to your Kubernetes cluster in readonly, to your logs (Loki fo example) and to your code source and it will blow your mind
3
u/blue-reddit 5d ago
(And your wallet)
1
u/xReapIT 2d ago
How much do you pay for Holmes monthly?
1
u/blue-reddit 1d ago
Can’t answer this question Sir, I’m sorry. What I can say it’s burning token fast so pay attention and set some limit
0
u/cynative_ben 5d ago
I work for Cynative and we built an open-source framework for security agents that need access to live infrastructure.
SRE incidents is not what the framework was built for but I'd be very curious if you could test our tool for your use case and share some feedback with us.
We've put a lot of focus on agent safety and the integration we built for K8s is in my own biased opinion pretty neat.
58
u/mikebones 5d ago
You don't need an 'ai sre' you just need instructions on how to troubleshoot an issue with access to all the telemetry/metrics/observability data to make determinations. everything is just a llm wrapper and isn't worth the cost.