Saturday, December 2, 2023

How to disable AWS Lambda recursion detection through code instead of support

AWS Lambda recently introduced loop detection which it will shut down certain kinds of recursion (For more info see this article). The problem with this is that there are certain kinds of chained batched processing in Lambda that will trigger this incorrectly. For instance, if you use a Lambda that consumes a paginated API and invokes itself with information of the next page through an SQS queue event it will now only process the first 16 pages of data before it gets shut down.

To make this worse, even though you can turn off this behavior by contacting AWS support you can only do this if you are paying for an AWS support subscription (At the time of writing $29 / month minimum). Also, this will disable it for all Lambda's in your account. I've come up with a very simple snippet that solves this in code for Node.js on a per Lambda basis without any AWS support interaction.


    export function disableLoopDetection() {
        // This little piece of magic disables the loop detection in the AWS SDK
        if (process.env._X_AMZN_TRACE_ID) {
            process.env._X_AMZN_TRACE_ID = process.env._X_AMZN_TRACE_ID.replace(/:\d+$/, ":1");
        }
    }

The loop detection uses the AWS X-Ray trace header. The very last number in the header that looks like this.


    X-Amzn-Trace-Id:Root=1-645f7998-4b1e232810b0bb733dba2eab;Parent=5be88d12eefc1fc0;Sampled=1;Lineage=43e12f0f:5

The very last number in that string is the invocation count. This code snippet changes the environment variable that contains the X-Raw trace header and changes the invocation count back to 1 with every invocation. You need to make sure you call this method in every call to the handler since each call will set this environment variable to a new value.

0 comments:

Post a Comment