How not to get locked out of your AWS account

A few weeks ago, a nightmare-level situation began to unfold. A routine login attempt to the AWS console for fossable.org suddenly hit me with an email verification challenge:

So what email did I decide to use for the root account? Obviously one under fossable.org, hosted by an email server within the very same AWS account I'm trying to login to. The same one I setup on a tiny EC2 instance with emailwiz 3 years ago and forgot about.

You might see where I'm going with this. Something unspeakable happened to that EC2 instance and it was no longer receiving email or SSH connections. Since the fossable.org domain is hosted with Route53, I couldn't just redirect it to a new server either.

I was locked out. The reality sunk in that I would have to cancel the credit card associated with the account and wait for the domain to expire, only to buy it back again (would that even work?).

Customer service doesn't come to the rescue

I understand you don't have access to the email address on the account.

For security and privacy reasons, AWS Customer Service can't share or change the email address or password information on accounts.

If the email address on the account is part of your corporate email system, we recommend contacting the IT system administrator to discuss options
that may assist you with regaining access to the email address by setting up a catchall.

Definitely not written by an AI. Understandably, they wouldn't let me back in. I had almost nothing to prove I was the account holder other than my password.

CI saves the day

Fortunately, I found a glimmer of hope in one of my private Github repositories. There was an AWS keypair saved in the CI secrets that hadn't been used since 2022. At first I tried to echo them into the job log, but Github Actions redacts secrets even if you intentionally try to print them (thankfully).

So instead I did the next best thing:

# XD
echo "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" | base64

Which worked like a charm and ended up being a keypair with sufficient privileges to update my Route53 zone. Now that I've graduated from emailwiz to nixos-mailserver, I had a new instance running in minutes to accept the verification email from AWS.

Tips

Obviously, don't use a recovery email that's hosted within the same AWS account. If you absolutely have to, enable MFA for the root user which seems to prevent AWS from ever sending you an email verification challenge. Keeping a root keypair in a safe hiding spot might save your skin (or, equally likely, lead to your ultimate demise).