How do you investigate an AWS incident?

Start with CloudTrail — it is the source of truth for what changed. Correlate the incident start time against recent API calls: IAM policy updates, security group changes, deployments, autoscaling events, and parameter changes. Then check service health dashboards and CloudWatch metrics for the affected resources. The vast majority of AWS incidents trace back to a change made by a person, a pipeline, or an autoscaling policy, so the question 'what changed?' resolves faster than 'what's broken?'.

What is the first thing to check during an AWS outage?

Two things in parallel: the AWS Health Dashboard (to rule out a regional AWS-side event) and CloudTrail for recent changes in your own accounts. If AWS itself is healthy, your incident almost certainly follows a change you or your automation made — an IAM update, a security group edit, a deploy, or a scaling event. Establishing which of the two it is within the first five minutes determines the entire response path.

How should multi-account AWS environments handle incident response?

Centralize the signals: aggregate CloudTrail into an organization trail, centralize CloudWatch alarms and Health events, and give responders one place to see cross-account state. During an incident, per-account console-hopping is where response time dies. AI agents with read access across accounts remove this bottleneck by running the cross-account correlation automatically.

AWS Incident Response Best Practices Playbook

Q: What are the most common AWS incidents?

The eight most common AWS operational incident types are: IAM permission and policy failures, security group and network ACL misconfigurations, RDS performance degradation and failovers, EC2/ASG capacity and health issues, Lambda throttling and concurrency exhaustion, S3 policy and access failures, service quota and API rate limit exhaustion, and cost anomalies from runaway resources. Most production AWS incidents follow a configuration change rather than a hardware failure.

Q: Can AWS incident response be automated?

Yes, in two layers. AI agents automate investigation: correlating CloudTrail events, CloudWatch metrics, and recent deployments to identify the root cause across accounts and services. Deterministic, approval-gated workflows automate remediation for known failure modes: reverting security group changes, scaling capacity, rotating credentials, or rolling back a deployment. Keep a human at the approval step for anything with production blast radius.

The defining fact of AWS incident response is that the cloud rarely breaks — it gets changed. Hardware failures are AWS's problem and are mostly abstracted away from you; what pages your team is an IAM policy edited last Tuesday, a security group rule someone "temporarily" removed, a Lambda concurrency limit nobody knew existed, or an autoscaling policy doing exactly what it was told. Configuration, not computers.

That fact shapes everything in this playbook: which incidents to prepare for, why CloudTrail — not CloudWatch — is where investigations should start, and why so much of AWS incident response can be automated. This is the cloud-side companion to our Kubernetes incident response playbook; the process is shared, the failure modes are not. (We build Kestrel, an AI automation platform spanning incident response, cloud provisioning, CI/CD, security operations, and developer requests — noted where relevant.)

The eight incident types behind most AWS pages

Incident type	Most common cause	First diagnostic move
IAM permission failures	Policy change, role trust edit, expired credentials, SCP update	CloudTrail for recent IAM writes; decode the AccessDenied error
Security group / NACL misconfig	Rule removed or tightened, often as a "cleanup"	CloudTrail `AuthorizeSecurityGroup/Revoke` events around incident start
RDS degradation / failover	Connection exhaustion, storage full, unindexed query, maintenance failover	DB connections + CPU + storage metrics; recent parameter group changes
EC2 / ASG capacity issues	Failed health checks cascading, capacity unavailable, bad launch template	ASG activity history; instance status checks
Lambda throttling	Concurrency limit hit, downstream slowness inflating duration	Throttles + ConcurrentExecutions metrics vs. account/reserved limits
S3 access failures	Bucket policy change, Block Public Access toggles, KMS key policy	CloudTrail `PutBucketPolicy`/KMS events; test with the failing principal
Quota / rate limit exhaustion	Growth hitting invisible ceilings; retry storms amplifying throttles	Service Quotas console vs. current usage; throttle error rates by API
Cost anomalies	Runaway autoscaling, forgotten resources, data transfer surprises	Cost anomaly detection; group by service + usage type, find the inflection date

Notice the pattern in the third column: five of the eight start in CloudTrail. That's not a coincidence — it's the operational consequence of AWS incidents being change-driven. Which brings us to the process.

The response process: rule out AWS, then find your change

1. First five minutes: AWS's problem or yours?

Check the AWS Health Dashboard and your CloudTrail in parallel. If AWS is having a regional event, your job changes completely: you're executing your degradation plan (failover, shed load, communicate), not debugging. If AWS is healthy — the common case — your incident follows a change made by a person, a pipeline, or a policy, and the investigation is a search problem with a known shape.

2. Triage by blast radius, mind the dependencies

Cloud blast radius is sneakier than cluster blast radius because managed services hide their dependency graphs: one throttled DynamoDB table can back-pressure a Lambda fleet into concurrency exhaustion that takes out an unrelated API. Triage on two axes — user impact now, and which shared resources (IAM roles, NAT gateways, databases, quotas) sit in the failure path and could widen it.

3. Investigate: CloudTrail is the source of truth

CloudWatch tells you what's happening; CloudTrail tells you what changed— and in AWS, what changed is almost always the answer. Line up the incident start against write-mode API calls in the relevant accounts: IAM and policy writes, security group edits, deployments, parameter changes, scaling events. In multi-account environments this is exactly where response time dies in console-hopping, and exactly what an AI agent does in seconds across an organization trail. Kestrel's investigation correlates CloudTrail, CloudWatch, and deploy history — down to the pull request that introduced the change when the trail leads back to code.

4. Remediate: revert the change, gate the risky steps

The cloud equivalent of "prefer rollback" is revert the change: restore the security group rule, roll back the policy version, return the parameter group to its previous state. IAM deserves special caution in both directions — over-broad "fixes" (attaching AdministratorAccess to make the error go away) create tomorrow's security incident, so remediation workflows should restore known-good states, not grant new permissions. And as with Kubernetes: security and data incidents are the exception where containment beats reversion.

Click to expand

The practices that separate elite teams

Centralize signals before you need them

Multi-account AWS is the norm now, and incident response across accounts is only tractable if the signals are already centralized: an organization-wide CloudTrail, aggregated CloudWatch alarms and Health events, and one place where a responder (human or agent) can see cross-account state. If step one of your incident process is "log into the right account," you've already lost ten minutes.

Guard the invisible ceilings

Service quotas are the distinctive AWS failure mode: invisible until the exact moment growth makes them a production incident. Alarm on quota utilization (Service Quotas publishes usage metrics for many services) at 80%, and treat retry behavior as part of the incident surface — naive retries turn a brief throttle into a self-sustaining storm. This is also a place where a continuously running monitoring agent earns its keep, checking ceilings humans forget exist.

Convert recurring responses into governed workflows

The same rule as Kubernetes: the third time an incident type occurs, its response becomes an executable workflow. AWS makes the case even stronger because remediation is API-shaped — reverting a security group rule, scaling an ASG, rotating a credential, restoring a policy version are all deterministic API sequences begging to be codified with an approval gate in front. Kestrel Workflows builds these from plain English against AWS and 30+ other integrations — spanning infrastructure (Kubernetes, OCI), IaC (Terraform Cloud, Pulumi Cloud, OpenTofu, CloudFormation), CI/CD (GitHub, GitLab, ArgoCD, Helm), observability (Datadog, OpenTelemetry), on-call (PagerDuty, incident.io, Slack), and databases (Supabase, PlanetScale, Neon) — with the access control and approval model that makes them safe to hand to a wider team. The same workflow model reaches beyond incident remediation into provisioning, deploy operations, and security response — reverting that security group rule is one governed workflow away from provisioning the environment correctly in the first place.

Two ways to convert recurring responses into governed workflows — Suggested Workflows and the Workflow Assistant.

Automate the investigation, keep approval on the write path

Everything in stage 3 — CloudTrail correlation, metric analysis, deploy diffing, cross-account state — is read-only and mechanical, which makes it perfect agent work. The division of labor that works: agents investigate automatically when the alert fires; remediation runs as deterministic workflows with a human approving anything that writes to production. The general version of this argument is our AI agents guide; the AWS-specific payoff is that the investigation phase — usually the longest part of a cloud incident — finishes before a human opens the page.

Frequently asked questions

What are the most common AWS incidents?

Eight patterns cover most operational pages: IAM permission failures, security group/NACL misconfigurations, RDS degradation, EC2/ASG capacity issues, Lambda throttling, S3 access failures, quota exhaustion, and cost anomalies. Nearly all are change-driven rather than failure-driven.

What's the first thing to check during an AWS outage?

The AWS Health Dashboard and your CloudTrail, in parallel. AWS-side event → execute your degradation plan. AWS healthy → find your change; it's almost certainly there.

How is AWS incident response different from Kubernetes incident response?

The process is the same (detect, triage, find the change, revert); the substrate differs. AWS failures are config- and API-shaped with hidden dependency graphs and quota ceilings, and CloudTrail replaces "check the previous logs" as the investigation's starting point. See our Kubernetes playbook for the cluster side.

Can AWS incident response be automated?

Investigation, yes — read-only agents correlating CloudTrail, metrics, and deploys across accounts. Remediation, yes for known failure modes — deterministic API workflows behind approval gates. The human's job shifts from doing the correlation to approving the fix.

The bottom line

AWS incidents are configuration incidents: made by changes, found through CloudTrail, and fixed by reversion. Rule out AWS in the first five minutes, triage with dependency graphs in mind, centralize your signals before the incident, alarm on the invisible ceilings, and turn every recurring response into a governed workflow. What's left for humans is the novel — which is the only part that ever needed them.

Kestrel runs this playbook as a product: 24/7 detection across AWS and Kubernetes, AI investigation that correlates CloudTrail, metrics, and code changes, and approval-gated remediation workflows — plus the same automation engine for cloud provisioning, CI/CD, security operations, and developer requests. You can try it free or see the incident response quickstart.