※本記事は IDCF Cloud Advent Calendar 2016 に参加しています。

昨日、IDCFクラウドを使ってさくっと負荷試験という記事を書きました。

負荷試験をされるサーバ側のチューニングの話は色々転がってそうなんで、
ab する側のマシンの性能でどのくらい変わるものなのかを実験してみたいと思います。

実際に色々試してみましたが、何かすごい内容があるわけでもないので、時間のない方は読み飛ばしてください。

いや、実験してみて、いい感じの -c の値を導き出せればと思ったんですが、対象サーバの環境なんかも依存するし、何をしたいかでも変わってくるので、実験だけしてまとまらないまま終了。

ちなみに、ab かけてるマシンは light.S2 です。ファイルディスクリプタいじった程度で大してチューニングしてません。nginx 使ってます。

先に結論と反省点と今後

abする側のマシンの性能で結構差は出ました。当たり前か・・・。

実際 ab する場合、-c 1000 程度なら lightタイプでもabしてる方のマシンがネックになることは少ないと思われます(サイズの大きいファイルだと別)。

それ以上にしたい場合は、スペックのいいタイプを選べばいいと思います。すぐ消してしまえばそんなにお金もかかりません。

反省点は、対象サーバの設定が思った以上にてきとうだったので、ある程度は設定しておくべきでした(途中でしました)。

あと、もうちょっとテンプレートのカーネル周りの設定をいじった方がよかったかもしれない。ネットワーク周りがデフォルトのままだったんで。ここは今後時間が出来たら修正したいです。

idcf-ab-bench もちょっと修正したので、変更を反映させておきたいと思います。

以下実験結果です。参考になるようなならないような。また実験したら追記します。タイプ名のあとのUUIDは serviceOfferingId です。

実験1

ab -n 10000 -c 1000 URL
をいろんなタイプのVM、5つからだいたい同時に投げてみる
URL は 423byte の画像

light.S1 (bd226b3b-6ae7-454d-b53d-c886f7eebe42)

Time taken for tests:   3.331 seconds
Failed requests:        13
   (Connect: 0, Receive: 0, Length: 13, Exceptions: 0)
Requests per second:    3002.50 [#/sec] (mean)

Time taken for tests:   4.927 seconds
Failed requests:        278
   (Connect: 0, Receive: 0, Length: 139, Exceptions: 139)
Requests per second:    2029.49 [#/sec] (mean)

Time taken for tests:   2.976 seconds
Failed requests:        10
   (Connect: 0, Receive: 0, Length: 10, Exceptions: 0)
Requests per second:    3360.73 [#/sec] (mean)

Time taken for tests:   3.053 seconds
Failed requests:        8
   (Connect: 0, Receive: 0, Length: 8, Exceptions: 0)
Requests per second:    3275.96 [#/sec] (mean)

Time taken for tests:   4.245 seconds
Failed requests:        4
   (Connect: 0, Receive: 0, Length: 4, Exceptions: 0)
Requests per second:    2355.77 [#/sec] (mean)

light.S2 (6ee2b663-3739-4c41-8e37-2fc3e830729f)

Time taken for tests:   3.034 seconds
Failed requests:        77
   (Connect: 0, Receive: 25, Length: 27, Exceptions: 25)
Requests per second:    3296.00 [#/sec] (mean)

Time taken for tests:   4.242 seconds
Failed requests:        15
   (Connect: 0, Receive: 0, Length: 15, Exceptions: 0)
Requests per second:    2357.45 [#/sec] (mean)

Time taken for tests:   3.039 seconds
Failed requests:        12
   (Connect: 0, Receive: 0, Length: 12, Exceptions: 0)
Requests per second:    3290.03 [#/sec] (mean)

Time taken for tests:   3.047 seconds
Failed requests:        22
   (Connect: 0, Receive: 0, Length: 22, Exceptions: 0)
Requests per second:    3281.62 [#/sec] (mean)

Time taken for tests:   4.070 seconds
Failed requests:        0
Requests per second:    2457.07 [#/sec] (mean)

standard.M8 (7ed510fa-f62e-4b4c-9172-4a29b423d674)

Time taken for tests:   3.526 seconds
Failed requests:        42
   (Connect: 0, Receive: 8, Length: 26, Exceptions: 8)
Requests per second:    2836.07 [#/sec] (mean)

Time taken for tests:   4.281 seconds
Failed requests:        56
   (Connect: 0, Receive: 13, Length: 30, Exceptions: 13)
Requests per second:    2335.86 [#/sec] (mean)

Time taken for tests:   4.249 seconds
Failed requests:        42
   (Connect: 0, Receive: 6, Length: 30, Exceptions: 6)
Requests per second:    2353.50 [#/sec] (mean)

Time taken for tests:   4.254 seconds
Failed requests:        209
   (Connect: 0, Receive: 46, Length: 117, Exceptions: 46)
Requests per second:    2350.81 [#/sec] (mean)

Time taken for tests:   0.428 seconds
Failed requests:        10121
   (Connect: 0, Receive: 1188, Length: 5334, Exceptions: 3599)
Requests per second:    23372.19 [#/sec] (mean)

最後の途中でめちゃくちゃ失敗してますね。

てか、そこまで有意な差もないし、対象マシンの load average が全然上がって無かったです。

実験2

ab -n 50000 -c 5000 URL
対象サーバが実際どのくらいまで処理可能なのかよくわからないので、1台だけでちょっと試してみる。

light.S1 (bd226b3b-6ae7-454d-b53d-c886f7eebe42)

Time taken for tests:   6.554 seconds
Complete requests:      50000
Failed requests:        4979
   (Connect: 0, Receive: 404, Length: 2425, Exceptions: 2150)
Requests per second:    7628.88 [#/sec] (mean)

light.S2 (6ee2b663-3739-4c41-8e37-2fc3e830729f)

Time taken for tests:   3.358 seconds
Complete requests:      50000
Failed requests:        6958
   (Connect: 0, Receive: 165, Length: 3536, Exceptions: 3257)
Requests per second:    14888.98 [#/sec] (mean)

standard.M8 (7ed510fa-f62e-4b4c-9172-4a29b423d674)

Time taken for tests:   4.443 seconds
Complete requests:      50000
Failed requests:        2042
   (Connect: 0, Receive: 133, Length: 1115, Exceptions: 794)
Requests per second:    11253.27 [#/sec] (mean)

なんでこんなにエラーが？
と思って nginx のログ見たら

2016/12/21 16:39:33 [alert] 64771#0: 768 worker_connections are not enough
2016/12/21 16:39:33 [alert] 64771#0: 768 worker_connections are not enough
2016/12/21 16:39:33 [alert] 64771#0: 768 worker_connections are not enough
2016/12/21 16:39:33 [alert] 64771#0: 768 worker_connections are not enough
2016/12/21 16:39:33 [alert] 64771#0: 768 worker_connections are not enough
2016/12/21 16:39:33 [alert] 64771#0: 768 worker_connections are not enough
...

私が悪かった。worker が 768 だかそのくらいだったので 32768 に変更。
軽く流してみてエラーが出なくなったのを確認。

実験3

ab -n 50000 -c 5000 URL
をいろんなタイプのVM、5つからだいたい同時に投げてみる
URL は 423byte の画像

対象サーバがさばける同時アクセス数が10000程度みたいなのもあって、有意な結果は出ず。むしろ、ん？という感じ。まぁ仕方ない。

light.S1 (bd226b3b-6ae7-454d-b53d-c886f7eebe42)

Failed requests:        12849
   (Connect: 0, Receive: 4243, Length: 4363, Exceptions: 4243)
Requests per second:    217.24 [#/sec] (mean)

Failed requests:        8685
   (Connect: 0, Receive: 2894, Length: 2897, Exceptions: 2894)
Requests per second:    384.97 [#/sec] (mean)

Failed requests:        10541
   (Connect: 0, Receive: 3490, Length: 3561, Exceptions: 3490)
Requests per second:    216.05 [#/sec] (mean)

Failed requests:        10038
   (Connect: 0, Receive: 3346, Length: 3346, Exceptions: 3346)
Requests per second:    365.49 [#/sec] (mean)

5台目は20分くらい待っても終わらなかったので諦めた。
nginxは元気でした。

light.S2 (6ee2b663-3739-4c41-8e37-2fc3e830729f)

Failed requests:        2355
   (Connect: 0, Receive: 545, Length: 1265, Exceptions: 545)
Requests per second:    383.52 [#/sec] (mean)

Failed requests:        6472
   (Connect: 0, Receive: 2152, Length: 2168, Exceptions: 2152)
Requests per second:    355.96 [#/sec] (mean)

Failed requests:        4137
   (Connect: 0, Receive: 1376, Length: 1385, Exceptions: 1376)
Requests per second:    383.61 [#/sec] (mean)

Failed requests:        7485
   (Connect: 0, Receive: 2493, Length: 2499, Exceptions: 2493)
Requests per second:    370.31 [#/sec] (mean)

また5台目は失敗。STDERR 出すようにしてあとで見てみよう・・・

standard.M8 (7ed510fa-f62e-4b4c-9172-4a29b423d674)

Time taken for tests:   138.987 seconds
Failed requests:        5547
   (Connect: 0, Receive: 1831, Length: 1884, Exceptions: 1832)
Requests per second:    359.74 [#/sec] (mean)

Time taken for tests:   243.874 seconds
Failed requests:        14276
   (Connect: 0, Receive: 4751, Length: 4774, Exceptions: 4751)
Requests per second:    205.02 [#/sec] (mean)

Time taken for tests:   130.334 seconds
Failed requests:        11653
   (Connect: 0, Receive: 3872, Length: 3908, Exceptions: 3873)
Requests per second:    383.63 [#/sec] (mean)

4台目失敗

Time taken for tests:   131.817 seconds
Failed requests:        11325
   (Connect: 0, Receive: 3772, Length: 3781, Exceptions: 3772)
Requests per second:    379.31 [#/sec] (mean)

highcpu.L8 (b603b7b2-d609-43da-b8fe-e4ec5ec501ab)

Time taken for tests:   127.926 seconds
Failed requests:        6743
   (Connect: 0, Receive: 2243, Length: 2257, Exceptions: 2243)
Requests per second:    390.85 [#/sec] (mean)

Time taken for tests:   136.940 seconds
Failed requests:        8372
   (Connect: 0, Receive: 2788, Length: 2796, Exceptions: 2788)
Requests per second:    365.12 [#/sec] (mean)

Time taken for tests:   273.500 seconds
Failed requests:        19566
   (Connect: 0, Receive: 6482, Length: 6602, Exceptions: 6482)
Requests per second:    182.82 [#/sec] (mean)

Time taken for tests:   153.623 seconds
Failed requests:        11650
   (Connect: 0, Receive: 3881, Length: 3888, Exceptions: 3881)
Requests per second:    325.47 [#/sec] (mean)

Time taken for tests:   287.897 seconds
Failed requests:        18816
   (Connect: 0, Receive: 6222, Length: 6370, Exceptions: 6224)
Write errors:           0
Requests per second:    173.67 [#/sec] (mean)

実験4

ab -n 100000 -c 10000 URL x 1台ずつ

ただし、-c 10000 は対象サーバが若干さばききれない時がある感じ(他にも色々動いてるので)。

light.S1 (bd226b3b-6ae7-454d-b53d-c886f7eebe42)

2回やってみました

Time taken for tests:   42.288 seconds
Failed requests:        915
   (Connect: 0, Receive: 305, Length: 305, Exceptions: 305)
Requests per second:    2364.74 [#/sec] (mean)

Time taken for tests:   48.977 seconds
Failed requests:        3450
   (Connect: 0, Receive: 1150, Length: 1150, Exceptions: 1150)
Requests per second:    2041.78 [#/sec] (mean)

light.S2 (6ee2b663-3739-4c41-8e37-2fc3e830729f)

Time taken for tests:   17.714 seconds
Failed requests:        1863
   (Connect: 0, Receive: 621, Length: 621, Exceptions: 621)
Requests per second:    5645.24 [#/sec] (mean)

Time taken for tests:   26.811 seconds
Failed requests:        2016
   (Connect: 0, Receive: 672, Length: 672, Exceptions: 672)
Requests per second:    3729.85 [#/sec] (mean)

2回で終わりにするつもりだったんですが、差が結構あるのでもうちょっと

Time taken for tests:   9.815 seconds
Failed requests:        0
Requests per second:    10188.09 [#/sec] (mean)

Time taken for tests:   14.282 seconds
Failed requests:        399
   (Connect: 0, Receive: 133, Length: 133, Exceptions: 133)
Requests per second:    7001.71 [#/sec] (mean)

よくわからなくなってきたので次いこう。

standard.M8 (7ed510fa-f62e-4b4c-9172-4a29b423d674)

Time taken for tests:   8.139 seconds
Failed requests:        0
Requests per second:    12286.78 [#/sec] (mean)

Time taken for tests:   14.695 seconds
Failed requests:        528
   (Connect: 0, Receive: 176, Length: 176, Exceptions: 176)
Requests per second:    6805.24 [#/sec] (mean)

Time taken for tests:   17.448 seconds
Failed requests:        2364
   (Connect: 0, Receive: 788, Length: 788, Exceptions: 788)
Requests per second:    5731.28 [#/sec] (mean)

なぜどんどんエラーが増えていくのか・・・

Time taken for tests:   16.555 seconds
Failed requests:        1569
   (Connect: 0, Receive: 523, Length: 523, Exceptions: 523)
Requests per second:    6040.30 [#/sec] (mean)

highcpu.L8 (b603b7b2-d609-43da-b8fe-e4ec5ec501ab)

Time taken for tests:   8.707 seconds
Failed requests:        0
Requests per second:    11484.69 [#/sec] (mean)

Time taken for tests:   15.383 seconds
Failed requests:        0
Requests per second:    6500.82 [#/sec] (mean)

Time taken for tests:   14.858 seconds
Failed requests:        723
   (Connect: 0, Receive: 241, Length: 241, Exceptions: 241)
Requests per second:    6730.47 [#/sec] (mean)

Time taken for tests:   0.660 seconds
Failed requests:        0
Requests per second:    15149.93 [#/sec] (mean)

テストしすぎて nginx のエラーログが大変なことになってきた tailf 出来ないワロス・・・

実験5

ab -c 10000 -n 100000 2台だいたい同時

highcpu.L8 (b603b7b2-d609-43da-b8fe-e4ec5ec501ab)

Time taken for tests:   34.098 seconds
Failed requests:        4262
   (Connect: 0, Receive: 1420, Length: 1420, Exceptions: 1422)
Requests per second:    2932.71 [#/sec] (mean)

Time taken for tests:   33.634 seconds
Failed requests:        7476
   (Connect: 0, Receive: 2492, Length: 2492, Exceptions: 2492)
Requests per second:    2973.23 [#/sec] (mean)

もっかい

Time taken for tests:   31.841 seconds
Failed requests:        3282
   (Connect: 0, Receive: 1094, Length: 1094, Exceptions: 1094)
Requests per second:    3140.63 [#/sec] (mean)

Time taken for tests:   32.033 seconds
Failed requests:        5698
   (Connect: 0, Receive: 1899, Length: 1899, Exceptions: 1900)
Requests per second:    3121.83 [#/sec] (mean)

実験6

ab -c 3000 -n 30000 3台だいたい同時

highcpu.L8 (b603b7b2-d609-43da-b8fe-e4ec5ec501ab)

Time taken for tests:   51.861 seconds
Failed requests:        0
Requests per second:    578.47 [#/sec] (mean)

Time taken for tests:   51.861 seconds
Failed requests:        3
   (Connect: 0, Receive: 1, Length: 1, Exceptions: 1)
Requests per second:    578.47 [#/sec] (mean)

Time taken for tests:   13.233 seconds
Complete requests:      30000
Failed requests:        120
   (Connect: 0, Receive: 40, Length: 40, Exceptions: 40)
Requests per second:    2267.08 [#/sec] (mean)

実験7

ab -c 500 -n 5000 10台だいたい同時

light.S2

Time taken for tests:   4.475 seconds
Failed requests:        0
Requests per second:    1117.44 [#/sec] (mean)

Time taken for tests:   3.485 seconds
Failed requests:        0
Requests per second:    1434.60 [#/sec] (mean)

Time taken for tests:   4.714 seconds
Failed requests:        0
Requests per second:    1060.69 [#/sec] (mean)

Time taken for tests:   4.086 seconds
Failed requests:        0
Requests per second:    1223.61 [#/sec] (mean)

Time taken for tests:   4.484 seconds
Failed requests:        0
Requests per second:    1115.17 [#/sec] (mean)

Time taken for tests:   4.258 seconds
Failed requests:        0
Requests per second:    1174.36 [#/sec] (mean)

Time taken for tests:   4.682 seconds
Failed requests:        0
Requests per second:    1067.85 [#/sec] (mean)

Time taken for tests:   4.691 seconds
Failed requests:        0
Requests per second:    1065.94 [#/sec] (mean)

Time taken for tests:   3.672 seconds
Failed requests:        0
Requests per second:    1361.64 [#/sec] (mean)

10台目は、ポートフォーワードの設定が終わる前にコマンド投げてしまって取れず・・・

highcpu.L8

Time taken for tests:   3.085 seconds
Failed requests:        0
Requests per second:    1620.68 [#/sec] (mean)

Time taken for tests:   2.993 seconds
Failed requests:        0
Requests per second:    1670.81 [#/sec] (mean)

Time taken for tests:   3.348 seconds
Failed requests:        0
Requests per second:    1493.23 [#/sec] (mean)

Time taken for tests:   2.745 seconds
Failed requests:        0
Requests per second:    1821.66 [#/sec] (mean)

Time taken for tests:   3.347 seconds
Failed requests:        0
Requests per second:    1493.89 [#/sec] (mean)

Time taken for tests:   3.509 seconds
Failed requests:        0
Requests per second:    1424.76 [#/sec] (mean)

Time taken for tests:   2.678 seconds
Failed requests:        0
Requests per second:    1867.35 [#/sec] (mean)

Time taken for tests:   3.560 seconds
Failed requests:        0
Requests per second:    1404.46 [#/sec] (mean)

Time taken for tests:   2.912 seconds
Failed requests:        0
Requests per second:    1717.19 [#/sec] (mean)

Time taken for tests:   0.823 seconds
Failed requests:        0
Requests per second:    6078.38 [#/sec] (mean)

最後のは実行中に他のが終わって空きが出来たから早かったのだと思われます。

この程度だと全然エラー出ませんね。

さいごに

日付変わったら何円使ったかチェックして追記します。

追記

前日に5円使ってるので、280円使ったということになります。
スペック高いの使うと結構5円チョコ56個分もかかるのか・・・