Why and how I use Amazon Step Functions

Why I use Amazon Step Functions Service

We have weekly releasing by using Jenkins, but we don't want to schedule RDS instance restoring and take snapshots by building server. So I decided to use Amazon Lambda do those tasks.

And the reason post is because I am looking forward more better way to do this task, if you have any idea please kindly share with me.

Prepare Jobs

• You already tired to operate RDS console manually each week?
• You never wrote any Python Code just like me?
• Not familiar with Amazon Cloud Watch like me?
• Not familiar with Amazon Step Function like me?
• Not familiar with Amazon Lambda like me?
• Only have 16 hours to finish this task like me?
• Two different security group already been created.
• Two RDS instance of any database; In this example I am using mySQL.
• You already know how to create IAM role.
• You know how RDS instance status and snapshot works.

Serval Amazon Services will be used

• Amazon RDS
• Amazon Lambda
• Amazon Step Functions
• Amazon Cloud Watch

Roadmap

At the beginning of story always be simple and easy...like this

Environment of RDS instance

• Dev RDS instance named DEV with SGA(security group A)
• Staging RDS instance named STAGING SGB(security group B)

Work Flow that I want to do

1. Take Dev's snapshot.
2. Delete Staging's instance with final snapshot.
3. Restore Dev's snapshot that we just took and rename it to Staging's instance name and setting preferences for it.
4. Replace SGA by modifying Staging's instance.

Let's Start!

First let's write a super easy Python on Amazon Lambda!!

Create IAM Role

Let's create a new IAM role only for your Lambda function use. If you need more help with creating IAM role, please see this in Japanese version or English version

The role I created here is just for demonstration, you do not necessary to attach full access like me. You should try those access rules and find out more accurately accessing rules.

And this is my Trust Relationship settings.

TrustRelationship.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create Lambda

After created a lambda function by using services panel. There're few steps introductions will lead you finish your function.

Select Blueprint

Select Python 2.7 and type "hello" in the text filter, we are going to use hello-world-python to write it.

Configure Triggers

We want to organize those lambda functions by using Amazon Step Function, therefore we just pass this trigger for now.
Click Next!

Configure Function

Let name your function in this step, make sure the Runtime should be Python 2.7

Lambda Function Code

It's really a very tiny code to use Python 2.7 and Boto3 ver.1.4.4, so let's use Edit code inline here.

dev-snapshot.py

import boto3
from botocore.client import ClientError
from datetime import datetime, timedelta, tzinfo

rds = boto3.client('rds')

# db_source_instance is a varible that your insance name
db_source_instance='dev'

# dev_snapshot_prefix is the snapshot name after execute this function
dev_snapshot_prefix = "dev-to-staging-snapshot"

# You can skip this class if you don't need convert to your timezone
class TWST(tzinfo):
    def utcoffset(self, dt):
        return timedelta(hours=8)
    def dst(self, dt):
        return timedelta(0)
    def tzname(self, dt):
        return 'JST'

# Lambda function entry
def lambda_handler(event, context):
    # Before we take snapshot for DEV, I prefer to delete the snapshots that I took in the same day.
    delete_snapshots(dev_snapshot_prefix)
    create_dev_snapshot(dev_snapshot_prefix, db_source_instance)

# Delete snapshots
def delete_snapshots(prefix):
    print '[delete_snapshots] Start'
    snapshots = rds.describe_db_snapshots()

    for snapshot in snapshots['DBSnapshots']:
        if snapshot['DBSnapshotIdentifier'].startswith(prefix):
            rds.delete_db_snapshot(DBSnapshotIdentifier=snapshot['DBSnapshotIdentifier'])

    print '[delete_snapshots] End'

# Create snapshot for DEV
def create_dev_snapshot(prefix, instanceid):
    print '[create_dev_snapshot] Start'
    snapshotid = "-".join([prefix, datetime.now(tz=TWST()).strftime("%Y-%m-%d")])

    for i in range(0, 5):
        try:
            snapshot = rds.create_db_snapshot(
                DBSnapshotIdentifier=snapshotid,
                DBInstanceIdentifier=instanceid
            )
            return
        except ClientError as e:
                print(str(e))
        time.sleep(1)

    print '[create_dev_snapshot] End'

Lambda Function Handler and Role

Keep the Handler and Role, select the IAM role you just created by your own, it will appear the name you gave it.

Advanced Settings

There are serval things that I hit this time.

At begin I wanted all those tasks in single one lambda function, then I realized waiting time like snapshot creating or restoring instance cause me timeout problem.

I tried to increase Timeout limit in Advanced Settings, but I can only increase it until 5 mins only. That's also make sense to me, because AWS Lambda charge service fee by request times, execution time and memory size.

時間
時間はコードの実行が開始された瞬間からコードが返されるか中止されるまでで計算され、>100 ミリ秒単位で切り上げられます。料金は関数に割り当てたメモリ量により異なります。1 >GB-秒の使用につき 0.00001667 USD の料金が発生します。

You can see more detail if you like.

Therefore I separated my function to several functions and using default advanced settings only. Again Timeout and Memory is depends on how much resources that your function need.

Test Your Lambda Function

After you click Next and Create Function, you will see this testing page.

Go test it and see does it works fine.
After you click the Test button, it will show a Input test event and wait your response for Sample event template.
Type "event" in that text filter like this.

When you select this template, you can see that content

Click Save and test
You should use those Execution result, Summary and Log output to fix your function until it's perfect to you!

Congratulations!!

Until here you already completed your first Python Lambda Function on Amazon Lambda.