How to do bake time in AWS CodePipeline using Step Functions

I really like AWS CodePipeline to handle my AWS deployments but there is one function that is missing from it in my opinion when you are doing multi region deployments which is the ability to let your changes bake for a proscribed amount of time while monitoring your service for any alarms to go off before deploying to the next region.

Sure you could relatively easily use CodeBuild and write a small script that waited while monitoring your alarms, but that always seemed really wasteful of compute to me. So instead I came up with a really small step function. All it does is take an input in minutes and for every minute it will monitor all alarms and if any of them go into an alarming state it will fail.

As you might notice there is no dedicate compute for this function and it does 3 step transitions per minute in total which means that an hour of bake time will cost you roughly $0.005 after you have used up the 4000 step transitions of perpetual free tier that Step Functions have. Below is the function definition in whole.

{
  "StartAt": "DescribeAlarms",
  "States": {
    "DescribeAlarms": {
      "Type": "Task",
      "Arguments": {
        "AlarmTypes": [
          "CompositeAlarm",
          "MetricAlarm"
        ]
      },
      "Resource": "arn:aws:states:::aws-sdk:cloudwatch:describeAlarms",
      "Output": {
        "Duration": "{% $states.input.Duration - 1 %}",
        "InAlarm": "{% $count($states.result.MetricAlarms[StateValue = 'ALARM']) + $count($states.result.CompositeAlarms[StateValue = 'ALARM']) %}"
      },
      "Next": "Choice"
    },
    "Success": {
      "Type": "Succeed"
    },
    "Choice": {
      "Type": "Choice",
      "Choices": [
        {
          "Next": "Fail",
          "Condition": "{% $states.input.InAlarm > 0 %}"
        },
        {
          "Next": "Success",
          "Condition": "{% $states.input.Duration < 0 %}"
        }
      ],
      "Default": "Wait"
    },
    "Fail": {
      "Type": "Fail"
    },
    "Wait": {
      "Type": "Wait",
      "Seconds": 60,
      "Next": "DescribeAlarms"
    }
  },
  "QueryLanguage": "JSONata",
  "Comment": "Waits for a proscribed \"Duration\" minutes and will fail if any alarm goes into alarming state."
}

You would invoke it with a payload looking like this.

{
  "Duration": 10
}

Where 10 is the number of minutes you wish for it to wait.

This version will alarm on any defined CloudWatch Alarm going into alarming state, but you could easily modify the DescribeAlarms state's InAlarm expression or Arguments above to exclude alarms you don't want to fail your bake time such as alarms used for auto scaling and similar which are not indicating service issues.

The only permission the function needs is to cloudwatch:DescribeAlarms in addition to the normal Step Function permissions. When using in CodePipeline you might also need to add the permission to run your StepFunction to the pipeline execution role.

How to disable AWS Lambda recursion detection through code instead of support

As of August 2024 AWS Lambda has now introduced a property called RecursiveLoop that allows you to control the loop detection in a more straight forward manner through the API.

AWS Lambda recently introduced loop detection which it will shut down certain kinds of recursion (For more info see this article). The problem with this is that there are certain kinds of chained batched processing in Lambda that will trigger this incorrectly. For instance, if you use a Lambda that consumes a paginated API and invokes itself with information of the next page through an SQS queue event it will now only process the first 16 pages of data before it gets shut down.

To make this worse, even though you can turn off this behavior by contacting AWS support you can only do this if you are paying for an AWS support subscription (At the time of writing $29 / month minimum). Also, this will disable it for all Lambda's in your account. I've come up with a very simple snippet that solves this in code for Node.js on a per Lambda basis without any AWS support interaction.

    export function disableLoopDetection() {
        // This little piece of magic disables the loop detection in the AWS SDK
        if (process.env._X_AMZN_TRACE_ID) {
            process.env._X_AMZN_TRACE_ID = process.env._X_AMZN_TRACE_ID.replace(/:\d+$/, ":1");
        }
    }

The loop detection uses the AWS X-Ray trace header. The very last number in the header that looks like this.

    X-Amzn-Trace-Id:Root=1-645f7998-4b1e232810b0bb733dba2eab;Parent=5be88d12eefc1fc0;Sampled=1;Lineage=43e12f0f:5

The very last number in that string is the invocation count. This code snippet changes the environment variable that contains the X-Raw trace header and changes the invocation count back to 1 with every invocation. You need to make sure you call this method in every call to the handler since each call will set this environment variable to a new value.

Optimized database layout for Underscore Backup

Tonight I spent a few hours optimizing the storage of objects in the back end of Underscore Backup. I expect more than 99% of the DynamoDB storage for this application to be a single table that contains every object that any source has stored with the service. Each source is identified by a unique UUID and each object contains a field depicting which source it belongs to. The optimization I realized is that I was storing the UUID as a string, including the '-' characters of the UUID which is 36 bytes long instead of the 16 bytes it would take to store the UUID in binary form. Based on my estimates, this will likely reduce the overall storage requirements in DynamoDB for the service by over 20%. Not to mention the smaller size will also mean it consumes fewer read units when querying and scanning the table.

This was a breaking change that required migration of old data so I am glad I did this now instead of later when the data volume would have made this a much harder problem to solve.