Observability across 100 AWS accounts on a national platform

The problem

An AWS Landing Zone spanning roughly a hundred accounts. Some carried twenty to forty compute resources, others fewer than ten, and all of them changed week to week. Instances appear and disappear faster than anyone can maintain monitoring by hand.

And the account admins were not going to stop making mistakes. That isn't a criticism, it's a design constraint: at a hundred accounts, human diligence is not a control you can rely on. Whatever got built had to work without anyone remembering to switch it on.

The failure mode isn't a missing dashboard. It's a resource nobody is watching, running perfectly quietly, right up until it isn't.

What I built

Alarm provisioning that reacts to the estate instead of describing it. When anyone in any account creates an EC2 instance, an ECS or EKS workload, an application load balancer or a NAT gateway, the monitoring attaches itself. No ticket, no manual step, no dependency on the person who launched it.

Load balancers and NAT gateways are in that list deliberately. Most estates instrument compute and stop there, which is where the attention goes but not where the money leaks. An idle load balancer bills quietly forever, and NAT gateway data processing is one of the more reliable ways to be surprised by an AWS invoice.

Architecture

Hub and spoke. Each account carries only an EventBridge rule; the logic lives once, in the master account, and reaches back into each child through an assumed role. One place to change behaviour, one place to debug, and nothing to keep in step across a hundred copies.

Child account · one of ~100

EC2

ECS

EKS

ALB

NAT Gateway

created or deleted

1

EventBridge ruleforwards the lifecycle event out of the account

alarms written back in

4

CloudWatchalarms attached or removed, plus the account dashboard

Master account

2

Custom event busreceives lifecycle events from every account

invokes

3

Alarm managerassumes a cross-account role, then adds or removes alarms

on alarm

5

SNSemail notification

and

6

Slack notifierformats the same alarm for the team channel

1. Someone in a child account creates or deletes an EC2 instance, an ECS or EKS workload, an ALB or a NAT gateway.
2. An EventBridge rule in that account forwards the lifecycle event to a custom bus in the master account.
3. The bus invokes a Lambda, which assumes a cross-account role back into the originating account.
4. It attaches the alarms, or removes them if the resource is gone. Same path in both directions.
5. A firing alarm publishes to SNS, which sends the email.
6. A second Lambda formats the same event for Slack.

I owned the whole of it: the architecture, the Terraform, and the bash automation that deployed the solution consistently across every account in the estate.

The mechanism is simple by design. The difficulty was never the Lambda; it was doing this identically across a hundred accounts with their own configurations, permissions and quirks, in an environment where the people using it could not be relied upon to cooperate with it.

Deletion runs the same path

Removing a resource triggers the same flow and takes its alarms with it. That half is usually skipped, and skipping it is expensive in both directions: orphaned alarms are billed monthly forever, and they alarm on metrics that stopped reporting, so the channel fills with noise about things that no longer exist.

An alerting system nobody trusts gets muted, and a muted alerting system is worse than none, because everyone believes it is still watching.

Building the symmetric path is the difference between something that demos well and something you can leave running for a year.

Why those metrics

The alarms weren't generic hygiene. Sustained high CPU alongside high network throughput is the signature of compute being consumed by something nobody authorised: a leaked credential turned into a bill, which is the abuse pattern any large multi-account estate has to assume will eventually happen to it.

Those were the metrics chosen, and that was the reason. Monitoring designed around a specific threat model looks different from monitoring assembled out of defaults.

A workload behaving strangely looks the same in the metrics whether the cause is a bug, a misconfiguration or an intruder.

Which is the useful property. The same alarms that surface a runaway test environment surface an unauthorised one, because at the level of CPU and network they are the same event: something is running that shouldn't be. Cover the estate automatically and you don't have to know in advance which kind you're looking for.

The reporting came first

The consumption reporting was a separate tool on its own pipeline, built from cost and usage data, and it was built before the alarm automation. That order was deliberate. You cannot sensibly automate an estate you cannot yet see, and a detailed picture of where the money was going existed within days rather than at the end of the engagement.

It landed as a single daily message in the project supervisors' Slack channel rather than an emailed attachment. Spend reporting only changes behaviour if the people who can act on it actually read it, and that means arriving where they already are.

Handover

Before moving on I ran hands-on training for two colleagues covering the multi-account automation and the cross-account deployment workflow. Automation nobody else can operate is a liability dressed as an asset.

Stack