AWSの一番好きなサービスはAWS Lambda な人です(再掲)
Provisioned Concurrency for Lambda Functions ちょっと試してみた。
の続きです。
Apache Bench使って、負荷かけてみたので、その辺まとめます。
あと、12/20にAWS Loft TokyoでAWS re:Invent 2019 サーバーレス re:Cap (以下サーバーレスre:Cap)があり、そこで聞いた話を織り交ぜます。
あと、クラスメソッドさんのブログでも検証されていますので、こちらも。
LambdaのProvisioned Concurrencyを使って、コールドスタート対策をしてみた #reinvent
安い？それとも高い？Provisioned Concurrencyを有効化したLambdaのコストに関する考察 #reinvent
課金でいうと、ほぼ一ヶ月間、設定だけして、起動していたLambdaがあるんですが、$15ぐらいになってました(おおっ、課金されてる！その2)。
ご利用は計画的にwって感じでしょうか。
実行パターン
以下のパターンのLambdaを準備して、Apache Benchを使って、10並列1000リクエストと20並列2000リクエストを3回ずつ実行しています。
- 何も設定してないLambda関数(Non VPC)
- Provisioned Concurrency=10を設定したLambda関数(Non VPC)
- Reserved Concurrency=10、Provisioned Concurrency=10を設定したLambda関数(Non VPC)
- 何も設定してないLambda関数(VPC)
- Provisioned Concurrency=10を設定したLambda関数(VPC)
- Reserved Concurrency=20、Provisioned Concurrency=10を設定したLambda関数(VPC)
※下記記載の3. の事象があったので、数変更
Provisioned Concurrency設定時
設定すると、ログストリームが多めに作られる、つまりコンテナが余計に起動するのは、一緒です。
サーバーレスre:CapでSAの方に聞いてみたのですが、どうも多めに起動させて、徐々に減らして、適正な数にしてるんじゃないかとのことでした。
実行
まず先に書くと、3. Reserved Concurrency=10、Provisioned Concurrency=10を設定したLambda関数(Non VPC)ですが、APIから実行すると、全てのリクエストがスロットリングになりました。
API Gatewayは500になってますね。
ログ抜粋
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Extended Request Id: FMkOVExVtjMFmzQ=
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Verifying Usage Plan for request: 79217ec3-8791-4e43-b667-54a35612b6d7. API Key: API Stage: aabbccddee/dev
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) API Key authorized because method 'GET /api3' does not require API Key. Request will not contribute to throttle or quota limits
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Usage Plan check succeeded for API Key and API Stage aabbccddee/dev
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Starting execution for request: 79217ec3-8791-4e43-b667-54a35612b6d7
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) HTTP Method: GET, Resource Path: /api3
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Method request path: {}
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Method request query string: {count=10000}
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Method request headers: {Accept=*/*, CloudFront-Viewer-Country=JP, CloudFront-Forwarded-Proto=https, CloudFront-Is-Tablet-Viewer=false, CloudFront-Is-Mobile-Viewer=false, User-Agent=ApacheBench/2.3, X-Forwarded-Proto=https, CloudFront-Is-SmartTV-Viewer=false, Host=aabbccddee.execute-api.ap-northeast-1.amazonaws.com, X-Forwarded-Port=443, X-Amzn-Trace-Id=Root=
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Method request body before transformations:
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Endpoint request URI: https://lambda.ap-northeast-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-northeast-1:123456789012:function:usual-lambda-dev-index3:pbLatest/invocations
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Endpoint request headers: {x-amzn-lambda-integration-tag=79217ec3-8791-4e43-b667-54a35612b6d7...
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Endpoint request body after transformations: {"resource":"/api3","path":"/api3","httpMethod":"GET","headers":{"Accept":"*/*","CloudFront-Forwarded-Proto":"https","CloudFront-Is-Desktop-Viewer":"true","CloudFront-Is-Mobile-Viewer":"false","CloudFront-Is-SmartTV-Viewer":"false","CloudFront-Is-Tablet-Viewer":"false","CloudFront-Viewer-Country":"JP","Host":"vr05c
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Sending request to https://lambda.ap-northeast-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-northeast-1:123456789012:function:usual-lambda-dev-index3:pbLatest/invocations
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Received response. Status: 429, Integration latency: 33 ms
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Endpoint response headers: {Date=Tue, 24 Dec 2019 06:45:09 GMT, Content-Type=application/json, Content-Length=104, Connection=keep-alive, x-amzn-RequestId=988d5bb2-6e0d-4b81-beb0-650aae7e5713, x-amzn-ErrorType=TooManyRequestsException}
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Endpoint response body before transformations: {"Reason":"ReservedFunctionConcurrentInvocationLimitExceeded","Type":"User","message":"Rate Exceeded."}
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Lambda invocation failed with status: 429. Lambda request id: 988d5bb2-6e0d-4b81-beb0-650aae7e5713
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Execution failed due to configuration error: Rate Exceeded.
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Method completed with status: 500
自分な勝手な想像としては、普通にリクエストを受け付けてくれると思ったんですけどね。10以上きたら、スロットリング起きるのはしょうがないとして。
まさか全リクエストエラーになるとは思いませんでした。。。
メトリクスを見たところ、Invocations = ProvisionedConcurrencySpilloverInvocations になっていました。
ProvisionedConcurrencySpilloverInvocations ってなんだろと探したところ、
AWS Lambda のメトリクス
に以下の記載がありました。
すべてのプロビジョニングされた同時実行数が使用中のときに、プロビジョニングされていない同時実行数を使用して実行される呼び出しの数。プロビジョニングされた同時実行数を使用するように設定されたバージョンまたはエイリアスの場合、Lambda は、プロビジョニングされていない同時実行数を使用して実行される呼び出しごとに、1 回カウントをインクリメントします。
別途コンテナを起動して処理した数っていう認識（違っていたらすみません）ですが、
すでに予約済み10で、10個（おそらく10個以上）コンテナが確保済みなので、別途起動しようとしたんですが、
予約済みなので、受け付けない状態になってるんでしょうか。。。とりあえず謎です。
仕様通りっていう話であれば、全然OKなんですが。。。
Provisioned Concurrency=9など、Reserved Concurrency以下(そもそも設定時にReserved Concurrency設定値以上は登録不可)にすれば、とりあえずリクエストは受け付けてくれます。
なので、併用する場合は、Reserved Concurrency > Provisioned Concurrency にする必要がありそうです。
同じくサーバーレスre:Capでも聞いてみたんですけどね。こういうことする人がいないwのか、そういう話は聞いたことがないとのことで、
とりあえず書いておいてくださいってことなので、書いておきますw
12パターンについてログと、一部メトリクスを添付します。
ごめんなさい。抜粋してますけど、長いです。
ログの見方は以下を参照してください。
Apache Benchでサクッと性能テスト
パターン1 10並列
これが今までのLambdaです。
# 1回目
Concurrency Level: 10
Time taken for tests: 53.154 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 18.81 [#/sec] (mean)
Time per request: 531.537 [ms] (mean)
Time per request: 53.154 [ms] (mean, across all concurrent requests)
Transfer rate: 9.48 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 0.8 11 20
Processing: 393 507 94.9 514 1243
Waiting: 393 506 94.8 514 1242
Total: 404 517 95.1 524 1253
Percentage of the requests served within a certain time (ms)
50% 524
66% 560
75% 571
80% 578
90% 584
95% 618
98% 675
99% 1115
100% 1253 (longest request)
# 2回目
Concurrency Level: 10
Time taken for tests: 51.709 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 19.34 [#/sec] (mean)
Time per request: 517.092 [ms] (mean)
Time per request: 51.709 [ms] (mean, across all concurrent requests)
Transfer rate: 9.75 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 0.7 11 19
Processing: 389 499 82.1 508 1154
Waiting: 389 499 82.1 507 1153
Total: 399 510 82.2 518 1167
Percentage of the requests served within a certain time (ms)
50% 518
66% 558
75% 568
80% 574
90% 584
95% 603
98% 649
99% 757
100% 1167 (longest request)
# 3回目
Server Software:
Server Hostname: vr05cvqij1.execute-api.ap-northeast-1.amazonaws.com
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128
Server Temp Key: ECDH P-256 256 bits
TLS Server Name: vr05cvqij1.execute-api.ap-northeast-1.amazonaws.com
Document Path: /dev/api?count=10000
Document Length: 33 bytes
Concurrency Level: 10
Time taken for tests: 51.894 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 19.27 [#/sec] (mean)
Time per request: 518.940 [ms] (mean)
Time per request: 51.894 [ms] (mean, across all concurrent requests)
Transfer rate: 9.71 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 7.2 11 237
Processing: 332 502 82.3 510 1143
Waiting: 331 502 82.3 509 1142
Total: 342 513 82.9 520 1157
Percentage of the requests served within a certain time (ms)
50% 520
66% 560
75% 571
80% 577
90% 583
95% 600
98% 650
99% 730
100% 1157 (longest request)
パターン1 20並列
安定して処理してますねw
# 1回目
Concurrency Level: 20
Time taken for tests: 53.007 seconds
Complete requests: 2000
Failed requests: 0
Total transferred: 1032000 bytes
HTML transferred: 66000 bytes
Requests per second: 37.73 [#/sec] (mean)
Time per request: 530.066 [ms] (mean)
Time per request: 26.503 [ms] (mean, across all concurrent requests)
Transfer rate: 19.01 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.1 11 22
Processing: 352 505 94.6 517 1195
Waiting: 352 505 94.6 516 1195
Total: 363 516 95.1 527 1213
Percentage of the requests served within a certain time (ms)
50% 527
66% 561
75% 568
80% 574
90% 587
95% 616
98% 673
99% 1111
100% 1213 (longest request)
# 2回目
Concurrency Level: 20
Time taken for tests: 51.902 seconds
Complete requests: 2000
Failed requests: 0
Total transferred: 1032000 bytes
HTML transferred: 66000 bytes
Requests per second: 38.53 [#/sec] (mean)
Time per request: 519.025 [ms] (mean)
Time per request: 25.951 [ms] (mean, across all concurrent requests)
Transfer rate: 19.42 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.2 10 23
Processing: 387 501 82.1 510 1172
Waiting: 387 501 82.1 509 1171
Total: 398 511 82.6 520 1192
Percentage of the requests served within a certain time (ms)
50% 520
66% 559
75% 569
80% 574
90% 583
95% 598
98% 647
99% 698
100% 1192 (longest request)
# 3回目
Concurrency Level: 20
Time taken for tests: 52.264 seconds
Complete requests: 2000
Failed requests: 0
Total transferred: 1032000 bytes
HTML transferred: 66000 bytes
Requests per second: 38.27 [#/sec] (mean)
Time per request: 522.641 [ms] (mean)
Time per request: 26.132 [ms] (mean, across all concurrent requests)
Transfer rate: 19.28 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.2 11 32
Processing: 317 504 83.4 515 1189
Waiting: 317 504 83.4 515 1189
Total: 327 515 83.8 526 1204
Percentage of the requests served within a certain time (ms)
50% 526
66% 563
75% 573
80% 578
90% 590
95% 611
98% 658
99% 736
100% 1204 (longest request)
パターン2 10並列
若干ではありますが、MAXの値が高くなってますね。
# 1回目
Concurrency Level: 10
Time taken for tests: 53.185 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 18.80 [#/sec] (mean)
Time per request: 531.846 [ms] (mean)
Time per request: 53.185 [ms] (mean, across all concurrent requests)
Transfer rate: 9.47 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 0.8 10 25
Processing: 391 509 84.2 517 1022
Waiting: 391 509 84.2 517 1022
Total: 401 520 84.5 527 1036
WARNING: The median and mean for the initial connection time are not within a normal deviation
These results are probably not that reliable.
Percentage of the requests served within a certain time (ms)
50% 527
66% 562
75% 575
80% 580
90% 593
95% 618
98% 683
99% 986
100% 1036 (longest request)
# 2回目
Concurrency Level: 10
Time taken for tests: 52.592 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 19.01 [#/sec] (mean)
Time per request: 525.920 [ms] (mean)
Time per request: 52.592 [ms] (mean, across all concurrent requests)
Transfer rate: 9.58 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 0.7 11 21
Processing: 396 505 76.7 517 1014
Waiting: 395 505 76.7 517 1014
Total: 406 516 76.8 528 1028
Percentage of the requests served within a certain time (ms)
50% 528
66% 563
75% 576
80% 579
90% 588
95% 602
98% 638
99% 770
100% 1028 (longest request)
# 3回目
Concurrency Level: 10
Time taken for tests: 52.384 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 19.09 [#/sec] (mean)
Time per request: 523.835 [ms] (mean)
Time per request: 52.384 [ms] (mean, across all concurrent requests)
Transfer rate: 9.62 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.3 11 38
Processing: 329 504 74.7 513 983
Waiting: 329 503 74.7 512 983
Total: 340 514 74.9 523 999
Percentage of the requests served within a certain time (ms)
50% 523
66% 561
75% 576
80% 580
90% 587
95% 600
98% 634
99% 732
100% 999 (longest request)
パターン2 20並列
足りない分はコンテナ起動が走り、あと、スロットリングも発生してます。
1回目のメトリクスですが、
これを見ると、
ProvisionedConcurrencyInvocations(プロビジョニングされた同時実行数を使用して実行される呼び出しの数)が744、
ProvisionedConcurrencySpilloverInvocationsが697
なので、起動済みコンテナで744、別途起動したコンテナで697リクエストそれぞれ呼び出され、残り559がスロットリングになっている感じですね。
何も設定してないLambdaではスロットリング発生しなかったので、この辺は謎ですが、ちゃんと負荷試験して、適切な数を設定しないとダメだよっていう戒めでしょうかw
# 1回目
Concurrency Level: 20
Time taken for tests: 39.292 seconds
Complete requests: 2000
Failed requests: 559
(Connect: 0, Receive: 0, Length: 559, Exceptions: 0)
Non-2xx responses: 559
Total transferred: 1006286 bytes
HTML transferred: 67677 bytes
Requests per second: 50.90 [#/sec] (mean)
Time per request: 392.923 [ms] (mean)
Time per request: 19.646 [ms] (mean, across all concurrent requests)
Transfer rate: 25.01 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.2 11 43
Processing: 13 373 231.9 434 1642
Waiting: 12 373 231.9 434 1642
Total: 23 384 232.1 445 1662
Percentage of the requests served within a certain time (ms)
50% 445
66% 538
75% 560
80% 567
90% 584
95% 601
98% 658
99% 762
100% 1662 (longest request)
# 2回目
Concurrency Level: 20
Time taken for tests: 44.707 seconds
Complete requests: 2000
Failed requests: 358
(Connect: 0, Receive: 0, Length: 358, Exceptions: 0)
Non-2xx responses: 358
Total transferred: 1015532 bytes
HTML transferred: 67074 bytes
Requests per second: 44.74 [#/sec] (mean)
Time per request: 447.070 [ms] (mean)
Time per request: 22.353 [ms] (mean, across all concurrent requests)
Transfer rate: 22.18 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.6 11 48
Processing: 13 423 205.6 461 1173
Waiting: 13 423 205.6 461 1173
Total: 24 434 206.0 472 1195
Percentage of the requests served within a certain time (ms)
50% 472
66% 557
75% 568
80% 574
90% 586
95% 613
98% 681
99% 988
100% 1195 (longest request)
# 3回目
Concurrency Level: 20
Time taken for tests: 44.839 seconds
Complete requests: 2000
Failed requests: 311
(Connect: 0, Receive: 0, Length: 311, Exceptions: 0)
Non-2xx responses: 311
Total transferred: 1017694 bytes
HTML transferred: 66933 bytes
Requests per second: 44.60 [#/sec] (mean)
Time per request: 448.390 [ms] (mean)
Time per request: 22.419 [ms] (mean, across all concurrent requests)
Transfer rate: 22.16 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.8 11 65
Processing: 13 430 187.8 477 1004
Waiting: 13 430 187.8 477 1004
Total: 24 441 187.9 488 1024
Percentage of the requests served within a certain time (ms)
50% 488
66% 553
75% 566
80% 574
90% 584
95% 598
98% 647
99% 723
100% 1024 (longest request)
パターン3 10並列
上記した通り全リクエストエラーになりました。
Concurrency Level: 10
Time taken for tests: 3.179 seconds
Complete requests: 1000
Failed requests: 0
Non-2xx responses: 1000
Total transferred: 470000 bytes
HTML transferred: 36000 bytes
Requests per second: 314.59 [#/sec] (mean)
Time per request: 31.787 [ms] (mean)
Time per request: 3.179 [ms] (mean, across all concurrent requests)
Transfer rate: 144.39 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 0.8 11 19
Processing: 10 20 5.7 19 90
Waiting: 9 20 5.7 19 90
Total: 21 31 5.8 30 101
Percentage of the requests served within a certain time (ms)
50% 30
66% 32
75% 33
80% 35
90% 39
95% 41
98% 45
99% 46
100% 101 (longest request)
パターン3 20並列
これも全滅です。
# 1回目
Concurrency Level: 20
Time taken for tests: 3.258 seconds
Complete requests: 2000
Failed requests: 0
Non-2xx responses: 2000
Total transferred: 940000 bytes
HTML transferred: 72000 bytes
Requests per second: 613.92 [#/sec] (mean)
Time per request: 32.577 [ms] (mean)
Time per request: 1.629 [ms] (mean, across all concurrent requests)
Transfer rate: 281.78 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 12 1.6 12 30
Processing: 10 20 5.5 19 91
Waiting: 9 19 5.5 19 91
Total: 20 32 5.9 31 101
Percentage of the requests served within a certain time (ms)
50% 31
66% 32
75% 34
80% 35
90% 38
95% 42
98% 46
99% 49
100% 101 (longest request)
次からは真価を発揮するはずのVPC Lambdaです（こっちだけでいいんじゃね説あり）
パターン4 10並列
これが今までのLambdaです。
初回はちょっと時間かかってるかな。
とはいえ、ms単位なので、ほんと速くなりました！
# 1回目
Concurrency Level: 10
Time taken for tests: 53.362 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 18.74 [#/sec] (mean)
Time per request: 533.621 [ms] (mean)
Time per request: 53.362 [ms] (mean, across all concurrent requests)
Transfer rate: 9.44 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.0 11 34
Processing: 389 507 107.8 517 1413
Waiting: 389 507 107.8 517 1413
Total: 400 518 108.1 528 1424
Percentage of the requests served within a certain time (ms)
50% 528
66% 560
75% 570
80% 577
90% 587
95% 619
98% 673
99% 1263
100% 1424 (longest request)
# 2回目
Concurrency Level: 10
Time taken for tests: 51.580 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 19.39 [#/sec] (mean)
Time per request: 515.800 [ms] (mean)
Time per request: 51.580 [ms] (mean, across all concurrent requests)
Transfer rate: 9.77 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 0.8 10 24
Processing: 356 497 69.0 507 809
Waiting: 355 497 69.0 507 809
Total: 366 508 69.0 518 819
WARNING: The median and mean for the initial connection time are not within a normal deviation
These results are probably not that reliable.
Percentage of the requests served within a certain time (ms)
50% 518
66% 559
75% 565
80% 571
90% 581
95% 590
98% 619
99% 709
100% 819 (longest request)
# 3回目
Concurrency Level: 10
Time taken for tests: 51.879 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 19.28 [#/sec] (mean)
Time per request: 518.788 [ms] (mean)
Time per request: 51.879 [ms] (mean, across all concurrent requests)
Transfer rate: 9.71 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 0.7 11 16
Processing: 359 500 71.4 509 937
Waiting: 359 499 71.4 509 937
Total: 370 510 71.4 520 948
Percentage of the requests served within a certain time (ms)
50% 520
66% 560
75% 571
80% 578
90% 586
95% 598
98% 625
99% 693
100% 948 (longest request)
パターン4 20並列
こちらも安定して処理してますねw
# 1回目
Concurrency Level: 20
Time taken for tests: 53.248 seconds
Complete requests: 2000
Failed requests: 0
Total transferred: 1032000 bytes
HTML transferred: 66000 bytes
Requests per second: 37.56 [#/sec] (mean)
Time per request: 532.478 [ms] (mean)
Time per request: 26.624 [ms] (mean, across all concurrent requests)
Transfer rate: 18.93 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.2 11 22
Processing: 376 506 107.8 517 1374
Waiting: 375 506 107.8 517 1374
Total: 386 517 108.5 528 1394
Percentage of the requests served within a certain time (ms)
50% 528
66% 560
75% 568
80% 573
90% 587
95% 613
98% 673
99% 1299
100% 1394 (longest request)
# 2回目
Concurrency Level: 20
Time taken for tests: 51.571 seconds
Complete requests: 2000
Failed requests: 0
Total transferred: 1032000 bytes
HTML transferred: 66000 bytes
Requests per second: 38.78 [#/sec] (mean)
Time per request: 515.709 [ms] (mean)
Time per request: 25.785 [ms] (mean, across all concurrent requests)
Transfer rate: 19.54 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.2 10 24
Processing: 382 498 70.4 505 980
Waiting: 381 498 70.4 505 980
Total: 392 509 70.4 515 1001
Percentage of the requests served within a certain time (ms)
50% 515
66% 558
75% 567
80% 574
90% 583
95% 593
98% 616
99% 750
100% 1001 (longest request)
# 3回目
Concurrency Level: 20
Time taken for tests: 51.633 seconds
Complete requests: 2000
Failed requests: 0
Total transferred: 1032000 bytes
HTML transferred: 66000 bytes
Requests per second: 38.74 [#/sec] (mean)
Time per request: 516.326 [ms] (mean)
Time per request: 25.816 [ms] (mean, across all concurrent requests)
Transfer rate: 19.52 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.2 11 25
Processing: 376 498 70.6 510 818
Waiting: 375 498 70.5 509 818
Total: 386 509 70.7 520 838
Percentage of the requests served within a certain time (ms)
50% 520
66% 557
75% 567
80% 574
90% 586
95% 595
98% 616
99% 670
100% 838 (longest request)
パターン5 10並列
3リクエストとも似たような値で、真価発揮しまくりですね。
# 1回目
Concurrency Level: 10
Time taken for tests: 53.120 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 18.83 [#/sec] (mean)
Time per request: 531.199 [ms] (mean)
Time per request: 53.120 [ms] (mean, across all concurrent requests)
Transfer rate: 9.49 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 0.8 11 26
Processing: 392 508 86.0 516 1042
Waiting: 391 508 85.9 515 1042
Total: 402 519 86.2 527 1053
Percentage of the requests served within a certain time (ms)
50% 527
66% 564
75% 576
80% 580
90% 594
95% 616
98% 697
99% 976
100% 1053 (longest request)
# 2回目
Concurrency Level: 10
Time taken for tests: 51.891 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 19.27 [#/sec] (mean)
Time per request: 518.911 [ms] (mean)
Time per request: 51.891 [ms] (mean, across all concurrent requests)
Transfer rate: 9.71 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 0.8 11 18
Processing: 391 501 69.0 513 815
Waiting: 390 500 69.0 513 814
Total: 401 511 69.1 524 827
Percentage of the requests served within a certain time (ms)
50% 524
66% 559
75% 569
80% 573
90% 582
95% 595
98% 611
99% 771
100% 827 (longest request)
#3回目
Concurrency Level: 10
Time taken for tests: 52.026 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 19.22 [#/sec] (mean)
Time per request: 520.264 [ms] (mean)
Time per request: 52.026 [ms] (mean, across all concurrent requests)
Transfer rate: 9.69 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 0.8 10 23
Processing: 389 502 72.4 517 1003
Waiting: 389 502 72.4 517 1003
Total: 400 513 72.5 528 1017
WARNING: The median and mean for the initial connection time are not within a normal deviation
These results are probably not that reliable.
Percentage of the requests served within a certain time (ms)
50% 528
66% 560
75% 572
80% 578
90% 590
95% 601
98% 635
99% 683
100% 1017 (longest request)
パターン5 20並列
やはりスロットリングが発生してますね。
1回目はLambdaの起動が走っているので、若干MAXの値が高いです。
ごめんなさい。3回目のログ欠落してます。。。
メトリクスは以下になります。
ProvisionedConcurrencyInvocations(プロビジョニングされた同時実行数を使用して実行される呼び出しの数)が833、
ProvisionedConcurrencySpilloverInvocationsが805
なので、起動済みコンテナで833、別途起動したコンテナで805リクエストそれぞれ呼び出され、残り362がスロットリングになっている感じですね。
# 1回目
Concurrency Level: 20
Time taken for tests: 44.721 seconds
Complete requests: 2000
Failed requests: 362
(Connect: 0, Receive: 0, Length: 362, Exceptions: 0)
Non-2xx responses: 362
Total transferred: 1015348 bytes
HTML transferred: 67086 bytes
Requests per second: 44.72 [#/sec] (mean)
Time per request: 447.212 [ms] (mean)
Time per request: 22.361 [ms] (mean, across all concurrent requests)
Transfer rate: 22.17 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.2 11 27
Processing: 12 424 208.9 463 1714
Waiting: 12 424 208.9 463 1714
Total: 23 435 209.1 473 1733
Percentage of the requests served within a certain time (ms)
50% 473
66% 556
75% 569
80% 578
90% 589
95% 609
98% 673
99% 737
100% 1733 (longest request)
# 2回目
Concurrency Level: 20
Time taken for tests: 44.973 seconds
Complete requests: 2000
Failed requests: 305
(Connect: 0, Receive: 0, Length: 305, Exceptions: 0)
Non-2xx responses: 305
Total transferred: 1017970 bytes
HTML transferred: 66915 bytes
Requests per second: 44.47 [#/sec] (mean)
Time per request: 449.731 [ms] (mean)
Time per request: 22.487 [ms] (mean, across all concurrent requests)
Transfer rate: 22.10 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.2 11 23
Processing: 13 430 183.8 474 799
Waiting: 13 430 183.9 473 799
Total: 24 441 183.9 485 820
Percentage of the requests served within a certain time (ms)
50% 485
66% 556
75% 569
80% 576
90% 587
95% 600
98% 617
99% 658
100% 820 (longest request)
パターン6 10並列
値を変えたので、安定して処理しています。
パターン5と傾向似てます。
# 1回目
Concurrency Level: 10
Time taken for tests: 53.350 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 18.74 [#/sec] (mean)
Time per request: 533.498 [ms] (mean)
Time per request: 53.350 [ms] (mean, across all concurrent requests)
Transfer rate: 9.45 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 0.6 10 19
Processing: 397 510 85.3 519 1059
Waiting: 397 510 85.3 519 1059
Total: 407 521 85.5 529 1070
WARNING: The median and mean for the initial connection time are not within a normal deviation
These results are probably not that reliable.
Percentage of the requests served within a certain time (ms)
50% 529
66% 567
75% 577
80% 580
90% 595
95% 617
98% 677
99% 981
100% 1070 (longest request)
# 2回目
Concurrency Level: 10
Time taken for tests: 52.095 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 19.20 [#/sec] (mean)
Time per request: 520.947 [ms] (mean)
Time per request: 52.095 [ms] (mean, across all concurrent requests)
Transfer rate: 9.67 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 0.8 10 25
Processing: 393 503 69.5 513 879
Waiting: 393 503 69.5 513 878
Total: 403 514 69.6 523 889
WARNING: The median and mean for the initial connection time are not within a normal deviation
These results are probably not that reliable.
Percentage of the requests served within a certain time (ms)
50% 523
66% 561
75% 572
80% 578
90% 586
95% 598
98% 619
99% 680
100% 889 (longest request)
# 3回目
Concurrency Level: 10
Time taken for tests: 52.404 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 516000 bytes
HTML transferred: 33000 bytes
Requests per second: 19.08 [#/sec] (mean)
Time per request: 524.041 [ms] (mean)
Time per request: 52.404 [ms] (mean, across all concurrent requests)
Transfer rate: 9.62 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 0.8 11 23
Processing: 396 506 71.8 517 817
Waiting: 395 506 71.8 517 817
Total: 406 516 71.8 527 831
Percentage of the requests served within a certain time (ms)
50% 527
66% 565
75% 578
80% 581
90% 597
95% 603
98% 622
99% 752
100% 831 (longest request)
パターン6 20並列
これもスロットリングおきてますね。。。
# 1回目
Concurrency Level: 20
Time taken for tests: 42.204 seconds
Complete requests: 2000
Failed requests: 442
(Connect: 0, Receive: 0, Length: 442, Exceptions: 0)
Non-2xx responses: 442
Total transferred: 1011668 bytes
HTML transferred: 67326 bytes
Requests per second: 47.39 [#/sec] (mean)
Time per request: 422.038 [ms] (mean)
Time per request: 21.102 [ms] (mean, across all concurrent requests)
Transfer rate: 23.41 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.2 11 24
Processing: 12 404 221.6 445 1386
Waiting: 12 404 221.6 445 1386
Total: 23 415 221.9 456 1409
Percentage of the requests served within a certain time (ms)
50% 456
66% 550
75% 569
80% 576
90% 590
95% 609
98% 673
99% 761
100% 1409 (longest request)
# 2回目
Concurrency Level: 20
Time taken for tests: 42.836 seconds
Complete requests: 2000
Failed requests: 395
(Connect: 0, Receive: 0, Length: 395, Exceptions: 0)
Non-2xx responses: 395
Total transferred: 1013830 bytes
HTML transferred: 67185 bytes
Requests per second: 46.69 [#/sec] (mean)
Time per request: 428.356 [ms] (mean)
Time per request: 21.418 [ms] (mean, across all concurrent requests)
Transfer rate: 23.11 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.4 10 30
Processing: 13 410 201.1 452 837
Waiting: 13 409 201.2 452 837
Total: 24 420 201.2 463 858
Percentage of the requests served within a certain time (ms)
50% 463
66% 549
75% 566
80% 573
90% 587
95% 599
98% 620
99% 667
100% 858 (longest request)
# 3回目
Concurrency Level: 20
Time taken for tests: 44.347 seconds
Complete requests: 2000
Failed requests: 328
(Connect: 0, Receive: 0, Length: 328, Exceptions: 0)
Non-2xx responses: 328
Total transferred: 1016912 bytes
HTML transferred: 66984 bytes
Requests per second: 45.10 [#/sec] (mean)
Time per request: 443.467 [ms] (mean)
Time per request: 22.173 [ms] (mean, across all concurrent requests)
Transfer rate: 22.39 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 10 11 1.2 11 27
Processing: 13 426 189.0 469 949
Waiting: 13 425 189.0 469 949
Total: 24 436 189.1 481 959
Percentage of the requests served within a certain time (ms)
50% 481
66% 554
75% 570
80% 576
90% 589
95% 601
98% 623
99% 711
100% 959 (longest request)
VPC Lambda速くなったなーというのは素直な感想。
また、Provisioned concurrencyを設定すると、さらに安定して処理してくれる感じですね。
ただ、Provisioned concurrency設定以上の同時リクエストがあった場合にスロットリング起きるのがちょっと意外でした。
未設定なLambdaではおきなかったので。
もしかしたら、設定してすぐにテストを実施したので、
「うも多めに起動させて、徐々に減らして、適正な数にしてるんじゃないか」っていうあたりが絡んでたりするのかな。。。
来年早々に、設定して、しばらく放置してから、どうなるかを検証してみたいと思います。
サーバーレスre:Capでも説明あったんですが、
Provisioned concurrencyは、定量的に同時にリクエストがあるならば、Lambdaを起動しておいてくれるのはもちろん、
事前にバーストさせておくことができるのが真価の一つとのこと。
つまり、
同時接続数を上限緩和して、1500等にした場合、従来だと、
ベースの同時接続数の1000までは即時バーストしますが、
そのあとの上限までは30秒に250ずつ（１分間に500ずつ）スケールする仕様ですが、
Provisioned concurrencyは1500って指定すれば、設定時に、1500個のコンテナ(+α)が起動します。
※同時接続数割り当てのうち、100は他のLambdaの起動用に必要らしいので、 1500を指定したい場合は、同時接続数は1600以上にしておく必要があります。
このテストしたいんですけど、個人検証用アカウントなんで上限緩和通るかな。。。w
最後に
サーバーレスre:Capでも話ありましたが、
VPC Lambdaはアンチパターンと言われ続けてきましたが、
今年のLambdaのアップデート(ENI作成タイミング変更)と、 この Provisioned concurrency と Data Proxy でもはやアンチパターンではなくなったのではないでしょうかとのこと。
RDSなどを使いたいはもちろんのこと、Lambdaから他のAPIを呼ぶ際に、セキュリティ要件でIP縛りをしたいという際にVPC Lambdaにして、外への通信をNAT Gateway経由にするという用途にも使いやすくなったようです。