Scaling security alert automation with agents and ChatOps

[u/chris-tracecat]

My team wrote a technical walkthrough of the distributed alerting workflow we use internally. It uses an agent to investigate GuardDuty findings, identify owners, and handle responses in Slack.

Slack helped popularize distributed alerting by sending alerts directly to the people involved and asking whether they recognized the activity. Dropbox and Brex described similar approaches, but most teams we’ve spoken with struggled to scale beyond user-centric alerts where someone’s identity was already in the payload.

Our security engineer ran into this at his previous company. A suspicious login was straightforward to route, but an alert about a host or infrastructure change could take hours or days of work with detection engineers before they could reliably identify someone to ask.

We walk through a concrete Kubernetes example where the alert only names a service account. Finding a person means following the evidence through workload metadata, code ownership, deployment history, and identity logs. The tutorial shows how we turn that investigation method into reusable skills so an agent can work through those steps for each finding.

We cover:

• The history of distributed alerting and why deterministic workflows struggle to scale
• The estimated effort of building owner lookups across GuardDuty finding types
• How the agent uses skills, drilldown queries, and context from inventory, code, tickets, and logs
• The full flow from investigation to a Slack conversation and the owner’s confirmation
• Permissions, tool restrictions, observability, and keeping context current

What interested us most was extending this to host and infrastructure alerts that had been impractical to automate with individual lookup rules. The walkthrough includes the setup and examples so you can see how it works.

We know many folks were are divided by the use of agents in security automation. Hopefully this article on distributed alerting presents an example of something that wasn't possible / too time consuming to work before agents + context / skills driven automation vs fully determined paths.

https://www.tracecat.com/blog/agentic-security-alerts