Sunday, February 26, 2023

Automating testing of signup

The main trick of testing the sign-up process of most sites is to handle the email verification since that requires you testing framework to parse an incoming email amd follow a link to complete it. I solved this by using SES incoming email functionality whixh through SNS passes the email to a Lambda. This Lambda then stores the parsed verification link in a DynamoDB table indexed by the recipient.

I then use the same Lambda set up with a simple function URL to do an HTTP redirect when you do a get to the verification link. That way you can automate your verification easily by just loading the Lamba URL. Below is the entire Lambda function code for receiving the email, storing it in a DynamoDB table called EmailLinks and also implements a HTTP responder where you call it with link {functionUrl}?email={email to confirmation to redirect to}.

const AWS = require('aws-sdk')
const findLinkRE = /(whatever you need to find the confirmation link from your email in the first match group)/;
const dbClient = new AWS.DynamoDB.DocumentClient();

exports.handler = async (event) => {
    console.log(event);
    if (event.Records) {
        for (let i = 0; i < event.Records.length; i++) {
            const message = JSON.parse(event.Records[i].Sns.Message);
            
            const content = message.content;
            const match = findLinkRE.exec(content)
            if (match) {
                for (let j = 0; j < message.mail.destination.length; j++) {
                    console.log(`Set ${message.mail.destination[j]}: ${match[1]}`);
                    await new Promise(function(resolve, reject) {
                        dbClient.put({
                            TableName: "EmailLinks",
                            Item: {
                                "Email": message.mail.destination[j],
                                "Link": match[1]
                            }
                        }, function(err, data) {
                            if (err) reject(err); else resolve(data);
                        });
                    });
                }
            }
        }
    } else if (event.rawQueryString.startsWith("email=")) {
        const item = await new Promise(function(resolve, reject) {
            dbClient.get({
                TableName: "EmailLinks",
                Key: { "Email": decodeURIComponent(event.rawQueryString.substring(6)) }
            }, function(err, data) {
                if (err) reject(err); else resolve(data);
            });
        });
        if (item && item.Item) {
            const response = {
                statusCode: 301,
                headers: { "location": item.Item.Link, "Content-Type": "text/html" },
                body: JSON.stringify(item.Item),
            };
            return response;
        } else {
            return {
                statusCode: 404,
                headers: { "Content-Type": "text/html" },
                body: "Not found"
            };
        }
    } else {
        return {
            statusCode: 400,
            headers: { "Content-Type": "text/html" },
            body: "Bad request"
        };
    }
};

Friday, February 24, 2023

Optimized database layout for Underscore Backup

Tonight I spent a few hours optimizing the storage of objects in the back end of Underscore Backup. I expect more than 99% of the DynamoDB storage for this application to be a single table that contains every object that any source has stored with the service. Each source is identified by a unique UUID and each object contains a field depicting which source it belongs to. The optimization I realized is that I was storing the UUID as a string, including the '-' characters of the UUID which is 36 bytes long instead of the 16 bytes it would take to store the UUID in binary form. Based on my estimates, this will likely reduce the overall storage requirements in DynamoDB for the service by over 20%. Not to mention the smaller size will also mean it consumes fewer read units when querying and scanning the table.

This was a breaking change that required migration of old data so I am glad I did this now instead of later when the data volume would have made this a much harder problem to solve.

Improved security option for private key recover in Underscore Backup

Yesterday night I came up with a new way of handling subscriptions in Underscore Backup that now gives the option of not storing the email of an account anywhere in the service as long as you are ok with not receiving billing emails. This will improve the security for private key recovery since the service will now not have the information required to decrypt the key stored anywhere in the service as long as you disable billing emails about charges (You can still see your invoices on the website). By default this is disabled because I think most people will prefer the billing emails over the increased security this provides.

Strarting up this little micro blog

Going to start a dev diary planning to do smaller posts with just minor updates compared to my main blog at my normal website.