Picture this…

It's 2 AM, most people are asleep, and your phone starts shaking like crazy!

Oh no! It looks like the production database has been deleted (it should never happen like that, but you never know!). No worries, though! We can restore it from a snapshot, but you'll need the necessary AWS permissions. Typically, database creation/modification/deletion should be handled through our Infrastructure as Code (IaC) Continuous Integration/Continuous Deployment (CI/CD) tools, like Atlantis for Terraform, which have the required AWS permissions granted through an IAM role.

As pipelines take on a more prominent role in the software development lifecycle in a DevOps model, the necessity for extensive human access to environments decreases. Human users should be granted minimal access necessary for their role, which is usually read-only access that does not allow any modifications or access to sensitive data. For experimentation which is typically hands-on and exploratory, teams should be granted access to sandbox environments which are isolated from system workloads.

~ AWS Well Architected Framework: Limit human access with just-in-time access

However, IaC with CI/CD requires both a pull request (PR) creator and a PR approver - always the two, like a master and an apprentice.

If you are the only person awake at night, making fixes through CI/CD becomes impossible (no one can approve a PR, <panic-mode-on>). So, what can be done in this situation?

If an incident occurs in the middle of the night and we lack the necessary permissions and cannot reach anyone for assistance, a break-glass policy comes into play. It’s like breaking the glass to access a fire extinguisher during an emergency. In this case, "breaking the glass" means using an account or role with elevated privileges/permissions.

—

In AWS, "break glass" refers to a method of granting emergency access to resources when standard access methods are unavailable or insufficient. It's a security measure designed to provide temporary, elevated access during critical situations like security incidents, system failures, or unavailability of key personnel. This is typically achieved by assuming a predefined "break glass" role with specific permissions, often through a trust policy, that allows access to otherwise restricted resources. 

—

So you "broke the glass" and used this powerful role to restore the database. You're the hero, but what's next?

Post Mortem

It is essential to clearly explain the reasons for using this role and the actions performed with it. This is necessary for auditability, as this role can have the capabilities to DELETE data or CREATE additional access. Therefore, it should only be utilized in emergency situations.

It’s important to avoid using this role for everyday tasks. Mistakes can occur; for example, you might think you are working on the staging AWS account and accidentally delete a database, only to realize later that you were actually in the production environment (have you heard how Gitlab did it once?)

In this blog post, you will learn how to establish a break-glass policy and set up Slack alerts for its usage. The remainder of the blog post will delve into technical details.

It's worth noting that implementing Break Glass access is one of the foundational elements of the AWS Well-Architected Framework.

Emergencies or unforeseen circumstances might necessitate temporary access beyond regular permissions for day-to-day work. Having break-glass procedures helps ensure that your organization can respond effectively to crises without compromising long-term security. During emergency scenarios, like the failure of the organization's identity provider, security incidents, or unavailability of key personnel, these measures provide temporary, elevated access beyond regular permissions.

~ AWS Well-Architected Framework: Implement break-glass procedures

How to implement Break Glass in AWS

Case A: IAM Users

If you use IAM users to access your AWS accounts, stop using them right now. Configure SSO for your account, and get rid of long-lived credentials.

Case B: SSO

In the Single Sign-On (SSO) context, a Permission Set refers to an Identity and Access Management (IAM) role created on target AWS accounts with specific permissions. This role is intended for users on those AWS accounts.

For example, a single user may have the following roles available for use on the "production" account:

• devops

• devops-break-glass

Each role comes with a different set of permissions. For instance, the "devops" role may have read-only access to databases:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "rds:Describe*",
        "rds:List*",
        "rds:View*"
      ],
      "Resource": "*"
    }
  ]
}

The "devops-break-glass" permission allows users to create, update, or delete RDS databases:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "rds:*",
      "Resource": "*"
    }
  ]
}

Although removing the database in case of an emergency may not sound like something you would do, you can imagine any other scenario: EC2 SSM shell access, purging the SQS queue, removing a file from the S3 bucket, or even reading files from the S3 bucket.

Notifications

It's crucial to configure notifications when the break-glass role is used. Those notifications can be used to implement a process requiring a person using that role to explain the need to use it.

There are many ways to achieve that; I would like to show you one based on EventBridge and API Destinations.

1. Create Eventbridge Connection to Slack API

For Authorization key use Slack Bot Token. Eventbridge will store this key in AWS Secrets Manager.

2. Create API destination

For connection ARN, use connection created in step 1. It will automatically match the authorization method configuration in EventBridge Connection.

3. Create an EventBridge rule to catch all usages of Break Glass roles.

Event Pattern:

{
  "detail": {
    "eventName": ["AssumeRole", "AssumeRoleWithSAML"],
    "eventSource": ["sts.amazonaws.com"],
    "requestParameters": {
      "roleArn": [{
        "wildcard": "arn:aws:iam::*:role/aws-reserved/sso.amazonaws.com/eu-west-1/AWSReservedSSO_break-glass_*"
      }]
    }
  },
  "detail-type": ["AWS API Call via CloudTrail"],
  "source": ["aws.sts"]
}

Target:

• use API destination Type and select the API Destination created in step 2

Input Transformer (very important! Here we are preparing a message for Slack):

{
    "channel": "#break-glass-audit",
    "blocks": [
        {
            "type": "header",
            "text": {
                "type": "plain_text",
                "text": "🚨 Break Glass Role Assumed 🚨",
                "emoji": true
            }
        },
        {
            "type": "section",
            "fields": [
                {
                    "type": "mrkdwn",
                    "text": "*Account:*\n<account>"
                },
                {
                    "type": "mrkdwn",
                    "text": "*Environment:*\nREPLACE_ME"
                },
                {
                    "type": "mrkdwn",
                    "text": "*Region:*\n<region>"
                },
                {
                    "type": "mrkdwn",
                    "text": "*Role:*\n`<role>`"
                },
                {
                    "type": "mrkdwn",
                    "text": "*User:*\n<user>"
                },
                {
                    "type": "mrkdwn",
                    "text": "*Time:*\n<time>"
                }
            ]
        },
        {
            "type": "divider"
        },
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "📣 *This role grants elevated permissions and should only be used in emergencies.*"
            }
        },
        {
            "type": "context",
            "elements": [
                {
                    "type": "mrkdwn",
                    "text": "📝 *Action required:* <user>, please provide a brief justification for using this role."
                }
            ]
        }
    ]
}

And click save.

Now, whenever the "Break Glass" role is assumed, we will receive a Slack notification, which looks like this:

—

Thanks to using EventBridge, we also get monitoring of how many times a Break Glass role was assumed in the past: