Calling third-party API on AWS

We wanted to call third-party API on AWS. Since they are third-party API, we should call them one by one and dodge to call multiple time at the same time. Otherwise, we might be blocked by providers.

SQS + Lambda

SQS + lambda should be easy to implement, but not sure if it fits with our purpose.
Therefore, we studied how SQS + Lambda works.

Configure SQS

The type of SQS is standard. The default visibility timeout is 30 sec.

Configure Lambda

Add trigger to lambda

Set concurrency to 1

Timeout is 5 for testing

Timeout is 5 sec.

Source code

This lambda will be timeout, but it is on purpose.

from time import sleep
import json

def lambda_handler(event, context):

    print("start python lambda");
    print(event)

    sleep(10)

    print("10 sec latter")

    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Now, ready to go

We sent messages to SQS via AWS console.

Expected result

Since lambda's timeout is 1 sec and the lambda sleeps 5 sec, every lambda must be timeout.
We assumes that SQS doesn't run all lambda in the queue concurrently, so each lambda should be executed one by one without 30 sec interval which is the default visibility timeout.

Cloudwatch

When we sent messages in a row like "1" -> "2" -> "3", then the log showed up like below.

The red block is "1", the blue block is "2" and the yellow block is "3". The first red block is executed at 17:37:05, the first blue block is executed at 17:37:10. Also, the second red block is executed almost 30 sec later. It seems that the default visibility timeout (30 sec) works well.

What this log shows?

If SQS tried to run lambda concurrentry, those message should be invisible. The invisible messages have to wait 30 sec because of the default visibility timeout. However, the blue block is executed at 17:37:10. That means that SQS waited until the first lambda had finished.

Conclustion

It is worth to consider SQS + lambda for calling third-party API.

*本記事は @qualitia_cdevの中の一人、伊藤さんに書いていただきました。