What to Automate in Incident Response: Start From the Trigger

The short version

Rank the trigger first: how often is it wrong? A honeytoken trip almost never is. A CPU or cost anomaly usually is.
Then rank the action: can it be undone, and how much can it touch? Restarting one stateless task is cheap to get wrong. Deleting data isn't.
Automate where both are high. Ask for one-click approval where one is. Leave the rest to a person, and let automation gather the evidence for them.
Every automated action should take a snapshot first, check the result on the signal that fired, and write an audit record.
Most AI SRE agents today investigate and propose. The ones that act alone do it behind approvals or a read-only default.

1. Why "should we automate this alert?" is the wrong question

Teams usually decide what to automate by looking at the response. Is the runbook well understood? Is the action safe? Can we roll it back? Those are good questions, and the autonomous DevOps framework (verifiable, reversible, contained) is built on them. But they skip the step before: how often is the trigger wrong?

An automated response runs every time the trigger fires, including every time it fires for no reason. So the trigger's error rate becomes the action's error rate. For most alerts, that rate is high, for a reason that has a name: the base-rate fallacy, which Stefan Axelsson applied to intrusion detection in 2000.

Illustrative arithmetic, not data

Say 1 alert in 1,000 is a real attack. An automatic "isolate the workload" response to that alert is wrong 999 times for every time it's right, and each wrong run is an outage you caused. Even an alert that's right 9 times in 10 would make an automatic response wrong once in ten runs, which is more than most teams will accept for anything that touches production.

That's why a person stays in the loop, and why that person is often asleep. The problem isn't that the action is unsafe. It's that the trigger is a symptom: CPU, egress, error rate, cost. Symptoms fire against a baseline that legitimate work moves around in, and that attackers can hide in. A different kind of trigger measures intent: something that only happens when someone does a thing nobody legitimate does. Those triggers are rare, and they are what makes automation safe.

The Google SRE book makes the mirror-image point: "Every page should be actionable", and "If a page merely merits a robotic response, it shouldn't be a page." Some pages should become automation. This guide is about telling which.

2. Triggers ranked by certainty

"Certain" here means: when it fires, it's almost always the thing you think it is. Rank your own triggers this way before you look at a single runbook.

Trigger	Why it is (or isn't) certain	Certainty
A honeytoken is used	A decoy credential nobody legitimate holds. Any use means someone found it and chose to try it. The only false positives are your own tests and your own scanners, and you can name both. See AWS canary credentials.	High
Specific GuardDuty findings, such as EC2 instance credentials used from outside AWS	Narrow findings about something that shouldn't happen at all are close to certain. GuardDuty's anomaly-based findings are, by design, about what looks unusual, so treat them as medium. See Honeytokens vs GuardDuty.	High for narrow findings, medium for anomalies
A Falco rule, such as a terminal shell in a container	Certain only if nobody on your team ever opens a shell in that container. Falco's own description of the "Terminal shell in container" rule notes it's common with `kubectl exec` and may be best used as a generic auditing rule. Scope it to production workloads where exec is forbidden and it becomes certain.	Depends on your rules
A deterministic condition: disk full, certificate expiring, a failed health check on a stateless task	Certain about the fact, not always about the cause. Orchestrators already automate the obvious response (ECS and Kubernetes replace unhealthy tasks); the open question is what to do when it keeps happening.	High for the fact
Error rate or latency jumps within minutes of a deploy	The deploy is a strong suspect, not a certainty: traffic, a dependency or a coincident change can do the same. Progressive delivery tools that compare a canary to the baseline make this much more certain.	Medium
A threshold or anomaly alert on CPU, egress, cost or queue depth	A symptom against a moving baseline. Legitimate work trips it all the time, and an attacker can stay under it.	Low

Notice what's at the top: triggers you build to be certain. You can't make a CPU alert certain by tuning it, but you can plant a decoy that only an intruder will touch, or write a rule for something your own processes never do. If the trigger you want to automate on is low on this list, the better investment is often a new trigger, not a smarter response.

3. Actions ranked by reversibility and reach

The second axis is the one most teams already think about. Two questions: can the action be undone, and how much can it touch?

Action	Undo	Reach
Collect evidence: logs, task metadata, a snapshot, a memory dump	Nothing to undo	None
Restart or replace one stateless task or pod from a clean image	The service scheduler brings it back	One workload
Deactivate one access key, or revoke one role's active sessions	Reactivate the key; sessions can be re-established	One identity, and whatever depends on it
Isolate a workload with a deny-all security group	Restore the previous groups	One workload, but it's down while isolated
Roll back the last deploy	Roll forward again	A whole service; can break with schema changes
Fail over a region or a database	Fail back, often slowly	Everything behind it
Delete or modify data, rotate shared secrets, change IAM policies broadly	Often impossible or slow	Unbounded

The top of this list is where automation is cheap to get wrong. A task restarted for no reason costs seconds. A deleted table costs days.

4. The matrix: what to automate, approve or leave to a person

Put the two rankings together. The rule in each cell is the whole method.

Trigger certainty ↓ / Action →	Undoable, one resource	Undoable, wide reach	Hard to undo
High (honeytoken, narrow finding, forbidden behaviour)	Automate. Act at once, verify, then tell a person what was done.	Approve. Pre-stage the action; one click runs it.	Person. Automate the evidence, not the decision.
Medium (deploy-correlated errors, anomaly findings)	Automate only with a check that confirms the cause first (for example, canary analysis before a rollback); otherwise approve.	Approve.	Person.
Low (threshold and symptom alerts)	Person. Automate the diagnosis and the evidence.	Person.	Person.

Two things follow. First, the "Automate" cell is small, and that's fine: it's where the 3 a.m. pages for real attacks come from, and minutes matter most there. Second, the bottom row isn't "do nothing". Uncertain triggers are where automated investigation pays off: gathering logs, recent deploys and related alerts so the person who decides starts with the evidence rather than a dashboard. That's what most AI SRE agents do today, and it's the right job for them.

To move a response to the left of the matrix, make the action smaller (restart one task instead of the service). To move it up, find or build a more certain trigger. Tuning the response rarely helps if the trigger is still a symptom.

5. Recommendation vs autonomous execution

A recommendation is a proposed action a person accepts or rejects. Autonomous execution means the system acts alone, so it has to do the things the person would have done:

Re-check the trigger at execution time. The event that fired five minutes ago may already be explained.
Snapshot before acting: the task definition, the security groups, the key's status. That snapshot is your undo.
Act under a named identity, so the audit trail says which automation did what, on whose authority.
Verify on the signal that fired. If the honeytoken keeps being used after the task restarted, the source wasn't that task.
Write the record a person reads in the morning: what fired, what was done, what was checked, how to undo it.

Grant autonomy per action, not per agent. "May restart tasks in this service when a honeytoken planted there is used" is a grant you can review. "May remediate incidents" isn't. The autonomous DevOps guide goes deeper on the action side: what makes an operation verifiable, reversible and contained.

6. Tools that do this today

Descriptions are from each vendor's own pages, checked on 8 November 2026. "Acts" means the vendor says the product can change your systems; "proposes" means a person approves each change.

Runbook and workflow automation

PagerDuty Runbook Automation turns manual runbook procedures into automated, self-service tasks. Its self-hosted edition (formerly Process Automation) is built on Rundeck, which is still available as open source (Apache-2.0).
incident.io Workflows automate actions and behaviours based on triggers in an incident.
AWS Systems Manager Automation runs maintenance and remediation runbooks, and is a supported EventBridge target, so a finding or CloudTrail event can start a runbook. Its sibling, Incident Manager, stopped accepting new customers on 7 November 2025.
GuardDuty publishes findings to EventBridge, where rules can send them to Lambda, Step Functions or an ECS task for an automated response.
StackStorm is open-source, event-driven automation for auto-remediation and incident response, and still maintained.
Falco (a CNCF graduated project) is a detection engine, not a responder, but it's where many of the certain container-level triggers come from.

AI SRE agents

A new category: agents that investigate incidents and, in some cases, act. Their approval models differ more than their marketing does, so this is the column to read.

Agent	What the vendor says it does	Acts or proposes
PagerDuty Paige	An SRE agent that detects, diagnoses and resolves incidents across people, agents and automations	Acts, but every remediation waits for approval unless you allow it to act independently
Rootly AI SRE	Investigation and response built into Rootly's incident platform	Proposes: every change needs human sign-off
incident.io Investigations	Starts debugging as soon as an incident is declared	Proposes: its only change is a pull request you merge
Datadog Bits Investigation (formerly Bits AI SRE)	An always-on agent for troubleshooting and late-night alerts	Suggests actions and code fixes; Datadog says it can now start remediation through Datadog workflows
Resolve.ai	Goes on call for you and resolves incidents autonomously	Acts: says it mitigates and restores service. The public page doesn't describe its approval model
Cleric	Checks production changes and fixes regressions before customers notice	Read-only by default; opens a fix pull request for approval
Traversal	Triages alerts and finds root cause at enterprise scale	Says it diagnoses and fixes incidents; read-only by default

The pattern matches the matrix. Most of these agents work on the bottom row: uncertain triggers, where the useful job is investigation and a proposed fix. The ones that act do so behind approvals or a read-only default, which is the "Approve" cell. Before you turn on any agent's autonomous mode, ask the trigger question: what fires it, and how often is that wrong?

7. Where Compass fits

Disclosure: we make it. Compass is open source and is built around the top-left cell. It plants honeytokens as pull requests to your infrastructure repository, so the trigger is one you made certain. The only actions it takes on its own are responses to those decoys, such as deactivating the key or restarting the ECS task it was taken from, and only after a person has granted each action by merging the pull request that gives Compass the permission. Each granted action has a mode: off, approve, or autopilot, which runs it only when its risk score, set by fixed rules rather than by a model, is at or below the ceiling you choose. Everything else it changes is a pull request a person merges.

You don't need Compass to apply this guide. The matrix works with any of the tools above, and the free terraform-aws-honeytoken module gives you a certain trigger in an afternoon.

8. FAQ

What should be automated in incident response?

Automate responses whose trigger is almost never wrong and whose action can be undone and is limited to one thing: for example, restarting the container a honeytoken was taken from, or deactivating that honeytoken's key. Automate evidence gathering for everything else, and leave the decision to a person.

Can an ops agent take remediation actions in production without human approval?

It can, safely, for a narrow set of actions: ones triggered by a signal that is almost never wrong, that can be undone, and that touch one resource. Everything else should wait for one-click approval or a person. Grant actions one at a time, and have the agent snapshot before acting, verify on the signal that fired, and write an audit record.

What is the difference between recommendation and autonomous execution?

A recommendation is a proposed action that a person approves or rejects. Autonomous execution means the system acts on its own, so it also has to do what the person would have done: confirm the trigger, check the action is still safe, verify the result and leave a record. The trigger's certainty decides which one is appropriate.

Why not just automate the response to every alert?

Because of base rates. If 1 alert in 1,000 is a real attack, an automatic response is wrong 999 times out of 1,000, and each wrong response is an outage you caused. The Google SRE book makes the related point that a page that only merits a robotic response shouldn't be a page.

Do AI SRE agents fix incidents on their own?

Mostly not yet. Rootly's AI SRE requires human sign-off on every change, incident.io's Investigations can only open a pull request, and Cleric and Traversal are read-only by default. PagerDuty's Paige waits for approval unless you let it act independently, and Resolve.ai says it mitigates and restores service. Check each vendor's approval model before you rely on it.

Is AWS Systems Manager Incident Manager still available?

Not to new customers. AWS stopped accepting new Incident Manager customers on 7 November 2025 and isn't adding features. Existing customers can keep using it. AWS points to Systems Manager OpsCenter for operational issues and to partners for paging and response. Systems Manager Automation runbooks are unaffected and can still be triggered from EventBridge.

Written by Ilia Dubovskii, founder of Beamreach. Vendor descriptions were checked against each vendor's own pages on 8 November 2026. Products in this space change quickly; if something here is out of date, tell us and we'll fix it.