目的
AWS Labmdaの本格利用を検討して、調査・検証を行ったので、知見をまとめる。
書いてあること
- 運用関連のこと
- コードの実装関連のこと
- cost関連のこと
管理
AWS Lambaの運用周りの私見をまとめる。
コードで管理する前から運用してるものがあるので、それらも管理対象としたければ、
aws cli使うか、スクリプト書いて、自前でCICD組むか、lambrollを使っちゃうのが良さそう。
AWS SAM
公式ではAWSSAMを使うことを推奨している模様。
https://github.com/aws/serverless-application-model
CloudFormationを利用するので、設定ファイルの書き方が独特で、調べにくいので、学習コストが高い。
既に運用済みのLambdaがあった場合、それをコードで管理できるようにする(terraform import的な)ことができなさそう。
テストコードはない
ローカルで立ち上げて動作を確認することになりそう。
定義の間違いは事前に確認できる
サンプル載せとく
nested application patternの確認
下記のような設定で、1つのconfigでfuncは増やせる。
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sam-test-mukai
xxxx omit xxx
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello_world/
Handler: app.lambda_handler
Runtime: python3.9
Architectures:
- x86_64
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
PiyoFunction: ######### ABLE TO ADD second func HERE #########
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: fuga/
Handler: app.lambda_handler
Runtime: python3.9
Architectures:
- x86_64
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /fuga
Method: get
xxxx omit xxx
削除するときは上記の箇所をけして、build → deploy で関連リソースとともに一緒に消してくれる。
$ sam build --profile assumed-profile-name
xxx omit xxx
CloudFormation stack changeset
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation LogicalResourceId ResourceType Replacement
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add ServerlessRestApiDeployment47fc2d5f9d AWS::ApiGateway::Deployment N/A
* Modify ServerlessRestApiProdStage AWS::ApiGateway::Stage False
* Modify ServerlessRestApi AWS::ApiGateway::RestApi False
- Delete PiyoFunctionHelloWorldPermissionProd AWS::Lambda::Permission N/A
- Delete PiyoFunctionRole AWS::IAM::Role N/A
- Delete PiyoFunction AWS::Lambda::Function N/A
- Delete ServerlessRestApiDeployment52d2abd56c AWS::ApiGateway::Deployment N/A
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Changeset created successfully. arn:aws:***
2024-02-07 19:56:09 - Waiting for stack create/update to complete
CloudFormation events from stack operations (refresh every 5.0 seconds)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus ResourceType LogicalResourceId ResourceStatusReason
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
UPDATE_IN_PROGRESS AWS::CloudFormation::Stack sam-test-piyo User Initiated
UPDATE_IN_PROGRESS AWS::ApiGateway::RestApi ServerlessRestApi -
UPDATE_COMPLETE AWS::ApiGateway::RestApi ServerlessRestApi -
CREATE_IN_PROGRESS AWS::ApiGateway::Deployment ServerlessRestApiDeployment47fc2d5f9d -
CREATE_IN_PROGRESS AWS::ApiGateway::Deployment ServerlessRestApiDeployment47fc2d5f9d Resource creation Initiated
CREATE_COMPLETE AWS::ApiGateway::Deployment ServerlessRestApiDeployment47fc2d5f9d -
UPDATE_IN_PROGRESS AWS::ApiGateway::Stage ServerlessRestApiProdStage -
UPDATE_COMPLETE AWS::ApiGateway::Stage ServerlessRestApiProdStage -
UPDATE_COMPLETE_CLEANUP_IN_PROGRESS AWS::CloudFormation::Stack sam-test-piyo -
DELETE_IN_PROGRESS AWS::Lambda::Permission PiyoFunctionHelloWorldPermissionProd -
DELETE_IN_PROGRESS AWS::ApiGateway::Deployment ServerlessRestApiDeployment52d2abd56c -
DELETE_COMPLETE AWS::Lambda::Permission PiyoFunctionHelloWorldPermissionProd -
DELETE_COMPLETE AWS::ApiGateway::Deployment ServerlessRestApiDeployment52d2abd56c -
DELETE_IN_PROGRESS AWS::Lambda::Function PiyoFunction -
DELETE_COMPLETE AWS::Lambda::Function PiyoFunction -
DELETE_IN_PROGRESS AWS::IAM::Role PiyoFunctionRole -
DELETE_COMPLETE AWS::IAM::Role PiyoFunctionRole -
UPDATE_COMPLETE AWS::CloudFormation::Stack sam-test-piyo -
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
xxx omit xxx
1つのfuncだけ修正したら、１だけ更新される。
$ sam deploy --guided --profile assumed-profile-name
xxx omit xxx
CloudFormation stack changeset
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation LogicalResourceId ResourceType Replacement
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
* Modify PiyoFunction AWS::Lambda::Function False
* Modify ServerlessRestApi AWS::ApiGateway::RestApi False
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Changeset created successfully. arn:aws:***
Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y
2024-02-07 20:51:47 - Waiting for stack create/update to complete
CloudFormation events from stack operations (refresh every 5.0 seconds)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus ResourceType LogicalResourceId ResourceStatusReason
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
UPDATE_IN_PROGRESS AWS::CloudFormation::Stack sam-test-piyo User Initiated
UPDATE_IN_PROGRESS AWS::Lambda::Function PiyoFunction -
UPDATE_COMPLETE AWS::Lambda::Function PiyoFunction -
UPDATE_COMPLETE_CLEANUP_IN_PROGRESS AWS::CloudFormation::Stack sam-test-piyo -
UPDATE_COMPLETE AWS::CloudFormation::Stack sam-test-piyo -
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
xxx omit xxx
serverless framework
serverless の管理まわりの元祖っぽい
SAMの構成と似ていて、SAMはこれを元に作られたんじゃないかという話があった。
メリット・デメリット・使い方もSAMと同じような感じ
terraform import みたいなことができないのがネック。
❯ sls
(node:46239) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Creating a new serverless project
? What do you want to make? (Use arrow keys)
❯ AWS - Node.js - Starter
AWS - Node.js - HTTP API
AWS - Node.js - Scheduled Task
AWS - Node.js - SQS Worker
AWS - Node.js - Express API
AWS - Node.js - Express API with DynamoDB
AWS - Python - Starter
AWS - Python - HTTP API
AWS - Python - Scheduled Task
AWS - Python - SQS Worker
AWS - Python - Flask API
AWS - Python - Flask API with DynamoDB
Other
❯ cat serverless.yml
service: aws-node-project
frameworkVersion: '3'
provider:
name: aws
runtime: nodejs18.x
region: ap-northeast-1
profile: assumed-role-name
functions:
function1:
handler: index.handler
assumed-role 指定できる
https://github.com/serverless/serverless/issues/3833#issuecomment-389739007
❯ AWS_SDK_LOAD_CONFIG=1 sls deploy --aws-profile=assumed-role-name
(node:41964) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Deploying aws-node-project to stage dev (ap-northeast-1)
✔ Service deployed to stack aws-node-project-dev (106s)
functions:
function1: aws-node-project-dev-function1 (1.4 kB)
Need a faster logging experience than CloudWatch? Try our Dev Mode in Console: run "serverless dev"
色々な使い方
❯ AWS_SDK_LOAD_CONFIG=1 sls invoke --function function1
(node:46864) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
{
"statusCode": 200,
"body": "{\n \"message\": \"Go Serverless v3.0! Your function executed successfully!\",\n \"input\": {}\n}"
}
test/sls-py-tuto/aws-node-project via v21.1.0 on ☁️ (ap-northeast-1) [10h28m19s] on ☁️ took 4s
❯ AWS_SDK_LOAD_CONFIG=1 sls logs --function function1
(node:46887) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
INIT_START Runtime Version: nodejs:18.v20 Runtime Version ARN: arn:aws:**
START
2024-02-08 12:41:31.521 INFO hogefuga
END Duration: 3.05 ms (init: 170.44 ms) Memory Used: 66 MB
test
Then, when you run sls test (your function must already have been deployed) it will make an HTTP request against the hello function and pass if the response has a status code of 200:
monitor
❯ AWS_SDK_LOAD_CONFIG=1 serverless metrics
(node:51784) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Service wide metrics
February 7, 2024 2:00 PM - February 8, 2024 2:00 PM
invocations: 2
throttles: 0
errors: 0
duration (avg.): 3.5ms
Nx
CI/CDに特化していそうな印象。
ただ、これもSAMに似たような感じになっていたので、ちゃんとは検証してない。
また、aws lambdaのサポートが終わりそう？
AWS SAM のwrapperで、cacheとかを使うっぽい？
lambroll
terraform import みたいなことができるのでこれ一択。
$ lambroll init --function-name=mukai-test --download --profile=assumed-profile
2024/02/08 22:14:32 [info] lambroll v0.14.7 with function.json
2024/02/08 22:14:34 [info] function mukai-test found
2024/02/08 22:14:34 [info] downloading function.zip
2024/02/08 22:14:34 [info] creating .lambdaignore
Overwrite existing file .lambdaignore? (y/n) [n]: y
2024/02/08 22:14:36 [info] creating function.json
import json
Overwrite existing file function.json? (y/n) [n]: y
2024/02/08 22:14:37 [info] completed
cat function.json
{
"Architectures": [
"x86_64"
],
"Description": "",
"EphemeralStorage": {
"Size": 512
},
"FunctionName": "hoge-test",
"Handler": "lambda_function.lambda_handler",
"LoggingConfig": {
"LogFormat": "Text",
"LogGroup": "/aws/lambda/hoge-test"
},
"MemorySize": 128,
"Role": "arn:aws:**",
"Runtime": "python3.12"
xxx
$ ls
function.json function.zip
$ unzip function.zip
Archive: function.zip
extracting: lambda_function.py
$ lambroll deploy --function=function.json --src="." --profile=assumed-profile
2024/02/08 22:16:33 [info] lambroll v0.14.7 with function.json
2024/02/08 22:16:33 [info] starting deploy function mukai-test
2024/02/08 22:16:35 [info] creating zip archive from .
2024/02/08 22:16:35 [info] zip archive wrote 307 bytes
2024/02/08 22:16:35 [info] updating function configuration
2024/02/08 22:16:35 [info] State:Active LastUpdateStatus:Successful
2024/02/08 22:16:35 [info] updated function configuration successfully
2024/02/08 22:16:35 [info] updating function code
2024/02/08 22:16:35 [info] State:Active LastUpdateStatus:InProgress
2024/02/08 22:16:35 [info] waiting for LastUpdateStatus Successful
2024/02/08 22:16:36 [info] State:Active LastUpdateStatus:Successful
2024/02/08 22:16:37 [info] update function code request was accepted
2024/02/08 22:16:38 [info] State:Active LastUpdateStatus:InProgress
2024/02/08 22:16:38 [info] waiting for LastUpdateStatus Successful
2024/02/08 22:16:39 [info] State:Active LastUpdateStatus:InProgress
2024/02/08 22:16:39 [info] waiting for LastUpdateStatus Successful
2024/02/08 22:16:41 [info] State:Active LastUpdateStatus:InProgress
2024/02/08 22:16:41 [info] waiting for LastUpdateStatus Successful
2024/02/08 22:16:45 [info] State:Active LastUpdateStatus:Successful
2024/02/08 22:16:45 [info] updated function code successfully
2024/02/08 22:16:45 [info] deployed version 1
2024/02/08 22:16:45 [info] updating alias set current to version 1
2024/02/08 22:16:45 [info] alias current is not found. creating alias
2024/02/08 22:16:45 [info] alias updated
2024/02/08 22:16:45 [info] completed
実装
AWS Lambda自体を調査し、処理の高速化につながるものや注意点等をまとめる。
The function's class stays in memory, so clients and variables that are declared outside of the handler method in initialization code can be reused. To save processing time on subsequent events, create reusable resources like AWS SDK clients during initialization. Once initialized, each instance of your function can process thousands of requests.
LambdaFunction class はメモリに残り続けるので、初期化時になるべく、使いまわせるものをメモリ上に残す実装にすると、次回の実行時により早くなる。
The 10-second timeout doesn't apply to functions that are using provisioned concurrency or SnapStart. For provisioned concurrency and SnapStart functions, your initialization code can run for up to 15 minutes. The time limit is 130 seconds or the configured function timeout (maximum 900 seconds), whichever is higher.
初期化されたコードは最大で15min実行され続ける。
Timeoutの設定で管理すると良い。
For example, if your account has a concurrency limit of 1,000, you cannot reserve all 1,000 units of concurrency to a single function.
アカウントが作成しているLambdaFunction全体の同時実行数を予約することになるので、それを考慮する必要がある。
Reserved concurrency – This represents the maximum number of concurrent instances allocated to your function. When a function has reserved concurrency, no other function can use that concurrency. Configuring reserved concurrency for a function incurs no additional charges.
Provisioned concurrency – This is the number of pre-initialized execution environments allocated to your function. These execution environments are ready to respond immediately to incoming function requests. Configuring provisioned concurrency incurs additional charges to your AWS account.
Reserved concurrencyはただ、台数を割り当てるだけ。
Provisioned concurrencyは事前に初期化された環境を用意するので、リクエストをすぐに捌ける。
今回のような動的リサイズにはProvisioned concurrencyの設定を入れる方が良い。
リソースの割り当て大きくして、Provisioned concurrencyあげると、料金が高くなる。
With Application Auto Scaling, you can set your own scaling schedule according to predictable load changes.
オートスケールの設定を入れられる。
Subsequent invocations processed by the same instance of your function can reuse these resources. This saves cost by reducing function run time.
リソースを使い回すことがあるので、初期化の際に適切にオンメモリにしとくと良い
To maintain your persistent connection, use the keep-alive directive associated with your runtime.
keep-alive を設定すると良い
This will reduce the amount of time that it takes for your deployment package to be downloaded and unpacked ahead of invocation.
packageを小さくすると起動が早い
AWSCodeDeployやAWS SAMを使わないとRolloing Deploymentができないので、デプロイ時に起動が遅くなる可能性がある。
Larger increase requests will take time to review, process, approve, and deploy. You can track your request case in the AWS Support console. Requests to increase service quotas don't receive priority support. If you have an urgent request, contact AWS Support.
同時実行数の上限の増加申請はAWS Support consoleから
最後に
検証させてくれたチームと会社に感謝