Edited at

IDCFクラウドで色んなタイプのVM作ってab砲発射してみた

More than 1 year has passed since last update.

※本記事は IDCF Cloud Advent Calendar 2016 に参加しています。

昨日、IDCFクラウドを使ってさくっと負荷試験 という記事を書きました。

負荷試験をされるサーバ側のチューニングの話は色々転がってそうなんで、

ab する側のマシンの性能でどのくらい変わるものなのかを実験してみたいと思います。


実際に色々試してみましたが、何かすごい内容があるわけでもないので、時間のない方は読み飛ばしてください。


いや、実験してみて、いい感じの -c の値を導き出せればと思ったんですが、対象サーバの環境なんかも依存するし、何をしたいかでも変わってくるので、実験だけしてまとまらないまま終了。

ちなみに、ab かけてるマシンは light.S2 です。ファイルディスクリプタいじった程度で大してチューニングしてません。nginx 使ってます。


先に結論と反省点と今後

abする側のマシンの性能で結構差は出ました。当たり前か・・・。

実際 ab する場合、-c 1000 程度なら lightタイプでもabしてる方のマシンがネックになることは少ないと思われます(サイズの大きいファイルだと別)。

それ以上にしたい場合は、スペックのいいタイプを選べばいいと思います。すぐ消してしまえばそんなにお金もかかりません。

反省点は、対象サーバの設定が思った以上にてきとうだったので、ある程度は設定しておくべきでした(途中でしました)。

あと、もうちょっとテンプレートのカーネル周りの設定をいじった方がよかったかもしれない。ネットワーク周りがデフォルトのままだったんで。ここは今後時間が出来たら修正したいです。

idcf-ab-bench もちょっと修正したので、変更を反映させておきたいと思います。

以下実験結果です。参考になるようなならないような。また実験したら追記します。タイプ名のあとのUUIDは serviceOfferingId です。


実験1

ab -n 10000 -c 1000 URL

をいろんなタイプのVM、5つからだいたい同時に投げてみる

URL は 423byte の画像27043.png


light.S1 (bd226b3b-6ae7-454d-b53d-c886f7eebe42)

Time taken for tests:   3.331 seconds

Failed requests: 13
(Connect: 0, Receive: 0, Length: 13, Exceptions: 0)
Requests per second: 3002.50 [#/sec] (mean)

Time taken for tests:   4.927 seconds

Failed requests: 278
(Connect: 0, Receive: 0, Length: 139, Exceptions: 139)
Requests per second: 2029.49 [#/sec] (mean)

Time taken for tests:   2.976 seconds

Failed requests: 10
(Connect: 0, Receive: 0, Length: 10, Exceptions: 0)
Requests per second: 3360.73 [#/sec] (mean)

Time taken for tests:   3.053 seconds

Failed requests: 8
(Connect: 0, Receive: 0, Length: 8, Exceptions: 0)
Requests per second: 3275.96 [#/sec] (mean)

Time taken for tests:   4.245 seconds

Failed requests: 4
(Connect: 0, Receive: 0, Length: 4, Exceptions: 0)
Requests per second: 2355.77 [#/sec] (mean)


light.S2 (6ee2b663-3739-4c41-8e37-2fc3e830729f)

Time taken for tests:   3.034 seconds

Failed requests: 77
(Connect: 0, Receive: 25, Length: 27, Exceptions: 25)
Requests per second: 3296.00 [#/sec] (mean)

Time taken for tests:   4.242 seconds

Failed requests: 15
(Connect: 0, Receive: 0, Length: 15, Exceptions: 0)
Requests per second: 2357.45 [#/sec] (mean)

Time taken for tests:   3.039 seconds

Failed requests: 12
(Connect: 0, Receive: 0, Length: 12, Exceptions: 0)
Requests per second: 3290.03 [#/sec] (mean)

Time taken for tests:   3.047 seconds

Failed requests: 22
(Connect: 0, Receive: 0, Length: 22, Exceptions: 0)
Requests per second: 3281.62 [#/sec] (mean)

Time taken for tests:   4.070 seconds

Failed requests: 0
Requests per second: 2457.07 [#/sec] (mean)


standard.M8 (7ed510fa-f62e-4b4c-9172-4a29b423d674)

Time taken for tests:   3.526 seconds

Failed requests: 42
(Connect: 0, Receive: 8, Length: 26, Exceptions: 8)
Requests per second: 2836.07 [#/sec] (mean)

Time taken for tests:   4.281 seconds

Failed requests: 56
(Connect: 0, Receive: 13, Length: 30, Exceptions: 13)
Requests per second: 2335.86 [#/sec] (mean)

Time taken for tests:   4.249 seconds

Failed requests: 42
(Connect: 0, Receive: 6, Length: 30, Exceptions: 6)
Requests per second: 2353.50 [#/sec] (mean)

Time taken for tests:   4.254 seconds

Failed requests: 209
(Connect: 0, Receive: 46, Length: 117, Exceptions: 46)
Requests per second: 2350.81 [#/sec] (mean)

Time taken for tests:   0.428 seconds

Failed requests: 10121
(Connect: 0, Receive: 1188, Length: 5334, Exceptions: 3599)
Requests per second: 23372.19 [#/sec] (mean)

最後の途中でめちゃくちゃ失敗してますね。

てか、そこまで有意な差もないし、対象マシンの load average が全然上がって無かったです。


実験2

ab -n 50000 -c 5000 URL

対象サーバが実際どのくらいまで処理可能なのかよくわからないので、1台だけでちょっと試してみる。


light.S1 (bd226b3b-6ae7-454d-b53d-c886f7eebe42)

Time taken for tests:   6.554 seconds

Complete requests: 50000
Failed requests: 4979
(Connect: 0, Receive: 404, Length: 2425, Exceptions: 2150)
Requests per second: 7628.88 [#/sec] (mean)


light.S2 (6ee2b663-3739-4c41-8e37-2fc3e830729f)

Time taken for tests:   3.358 seconds

Complete requests: 50000
Failed requests: 6958
(Connect: 0, Receive: 165, Length: 3536, Exceptions: 3257)
Requests per second: 14888.98 [#/sec] (mean)


standard.M8 (7ed510fa-f62e-4b4c-9172-4a29b423d674)

Time taken for tests:   4.443 seconds

Complete requests: 50000
Failed requests: 2042
(Connect: 0, Receive: 133, Length: 1115, Exceptions: 794)
Requests per second: 11253.27 [#/sec] (mean)

なんでこんなにエラーが?

と思って nginx のログ見たら

2016/12/21 16:39:33 [alert] 64771#0: 768 worker_connections are not enough

2016/12/21 16:39:33 [alert] 64771#0: 768 worker_connections are not enough
2016/12/21 16:39:33 [alert] 64771#0: 768 worker_connections are not enough
2016/12/21 16:39:33 [alert] 64771#0: 768 worker_connections are not enough
2016/12/21 16:39:33 [alert] 64771#0: 768 worker_connections are not enough
2016/12/21 16:39:33 [alert] 64771#0: 768 worker_connections are not enough
...

私が悪かった。worker が 768 だかそのくらいだったので 32768 に変更。

軽く流してみてエラーが出なくなったのを確認。


実験3

ab -n 50000 -c 5000 URL

をいろんなタイプのVM、5つからだいたい同時に投げてみる

URL は 423byte の画像

対象サーバがさばける同時アクセス数が10000程度みたいなのもあって、有意な結果は出ず。むしろ、ん?という感じ。まぁ仕方ない。


light.S1 (bd226b3b-6ae7-454d-b53d-c886f7eebe42)

Failed requests:        12849

(Connect: 0, Receive: 4243, Length: 4363, Exceptions: 4243)
Requests per second: 217.24 [#/sec] (mean)

Failed requests:        8685

(Connect: 0, Receive: 2894, Length: 2897, Exceptions: 2894)
Requests per second: 384.97 [#/sec] (mean)

Failed requests:        10541

(Connect: 0, Receive: 3490, Length: 3561, Exceptions: 3490)
Requests per second: 216.05 [#/sec] (mean)

Failed requests:        10038

(Connect: 0, Receive: 3346, Length: 3346, Exceptions: 3346)
Requests per second: 365.49 [#/sec] (mean)

5台目は20分くらい待っても終わらなかったので諦めた。

nginxは元気でした。


light.S2 (6ee2b663-3739-4c41-8e37-2fc3e830729f)

Failed requests:        2355

(Connect: 0, Receive: 545, Length: 1265, Exceptions: 545)
Requests per second: 383.52 [#/sec] (mean)

Failed requests:        6472

(Connect: 0, Receive: 2152, Length: 2168, Exceptions: 2152)
Requests per second: 355.96 [#/sec] (mean)

Failed requests:        4137

(Connect: 0, Receive: 1376, Length: 1385, Exceptions: 1376)
Requests per second: 383.61 [#/sec] (mean)

Failed requests:        7485

(Connect: 0, Receive: 2493, Length: 2499, Exceptions: 2493)
Requests per second: 370.31 [#/sec] (mean)

また5台目は失敗。STDERR 出すようにしてあとで見てみよう・・・


standard.M8 (7ed510fa-f62e-4b4c-9172-4a29b423d674)

Time taken for tests:   138.987 seconds

Failed requests: 5547
(Connect: 0, Receive: 1831, Length: 1884, Exceptions: 1832)
Requests per second: 359.74 [#/sec] (mean)

Time taken for tests:   243.874 seconds

Failed requests: 14276
(Connect: 0, Receive: 4751, Length: 4774, Exceptions: 4751)
Requests per second: 205.02 [#/sec] (mean)

Time taken for tests:   130.334 seconds

Failed requests: 11653
(Connect: 0, Receive: 3872, Length: 3908, Exceptions: 3873)
Requests per second: 383.63 [#/sec] (mean)

4台目失敗

Time taken for tests:   131.817 seconds

Failed requests: 11325
(Connect: 0, Receive: 3772, Length: 3781, Exceptions: 3772)
Requests per second: 379.31 [#/sec] (mean)


highcpu.L8 (b603b7b2-d609-43da-b8fe-e4ec5ec501ab)

Time taken for tests:   127.926 seconds

Failed requests: 6743
(Connect: 0, Receive: 2243, Length: 2257, Exceptions: 2243)
Requests per second: 390.85 [#/sec] (mean)

Time taken for tests:   136.940 seconds

Failed requests: 8372
(Connect: 0, Receive: 2788, Length: 2796, Exceptions: 2788)
Requests per second: 365.12 [#/sec] (mean)

Time taken for tests:   273.500 seconds

Failed requests: 19566
(Connect: 0, Receive: 6482, Length: 6602, Exceptions: 6482)
Requests per second: 182.82 [#/sec] (mean)

Time taken for tests:   153.623 seconds

Failed requests: 11650
(Connect: 0, Receive: 3881, Length: 3888, Exceptions: 3881)
Requests per second: 325.47 [#/sec] (mean)

Time taken for tests:   287.897 seconds

Failed requests: 18816
(Connect: 0, Receive: 6222, Length: 6370, Exceptions: 6224)
Write errors: 0
Requests per second: 173.67 [#/sec] (mean)


実験4

ab -n 100000 -c 10000 URL x 1台ずつ

ただし、-c 10000 は対象サーバが若干さばききれない時がある感じ(他にも色々動いてるので)。


light.S1 (bd226b3b-6ae7-454d-b53d-c886f7eebe42)

2回やってみました

Time taken for tests:   42.288 seconds

Failed requests: 915
(Connect: 0, Receive: 305, Length: 305, Exceptions: 305)
Requests per second: 2364.74 [#/sec] (mean)

Time taken for tests:   48.977 seconds

Failed requests: 3450
(Connect: 0, Receive: 1150, Length: 1150, Exceptions: 1150)
Requests per second: 2041.78 [#/sec] (mean)


light.S2 (6ee2b663-3739-4c41-8e37-2fc3e830729f)

Time taken for tests:   17.714 seconds

Failed requests: 1863
(Connect: 0, Receive: 621, Length: 621, Exceptions: 621)
Requests per second: 5645.24 [#/sec] (mean)

Time taken for tests:   26.811 seconds

Failed requests: 2016
(Connect: 0, Receive: 672, Length: 672, Exceptions: 672)
Requests per second: 3729.85 [#/sec] (mean)

2回で終わりにするつもりだったんですが、差が結構あるのでもうちょっと

Time taken for tests:   9.815 seconds

Failed requests: 0
Requests per second: 10188.09 [#/sec] (mean)

Time taken for tests:   14.282 seconds

Failed requests: 399
(Connect: 0, Receive: 133, Length: 133, Exceptions: 133)
Requests per second: 7001.71 [#/sec] (mean)

よくわからなくなってきたので次いこう。


standard.M8 (7ed510fa-f62e-4b4c-9172-4a29b423d674)

Time taken for tests:   8.139 seconds

Failed requests: 0
Requests per second: 12286.78 [#/sec] (mean)

Time taken for tests:   14.695 seconds

Failed requests: 528
(Connect: 0, Receive: 176, Length: 176, Exceptions: 176)
Requests per second: 6805.24 [#/sec] (mean)

Time taken for tests:   17.448 seconds

Failed requests: 2364
(Connect: 0, Receive: 788, Length: 788, Exceptions: 788)
Requests per second: 5731.28 [#/sec] (mean)

なぜどんどんエラーが増えていくのか・・・

Time taken for tests:   16.555 seconds

Failed requests: 1569
(Connect: 0, Receive: 523, Length: 523, Exceptions: 523)
Requests per second: 6040.30 [#/sec] (mean)


highcpu.L8 (b603b7b2-d609-43da-b8fe-e4ec5ec501ab)

Time taken for tests:   8.707 seconds

Failed requests: 0
Requests per second: 11484.69 [#/sec] (mean)

Time taken for tests:   15.383 seconds

Failed requests: 0
Requests per second: 6500.82 [#/sec] (mean)

Time taken for tests:   14.858 seconds

Failed requests: 723
(Connect: 0, Receive: 241, Length: 241, Exceptions: 241)
Requests per second: 6730.47 [#/sec] (mean)

Time taken for tests:   0.660 seconds

Failed requests: 0
Requests per second: 15149.93 [#/sec] (mean)

テストしすぎて nginx のエラーログが大変なことになってきた tailf 出来ないワロス・・・


実験5

ab -c 10000 -n 100000 2台だいたい同時


highcpu.L8 (b603b7b2-d609-43da-b8fe-e4ec5ec501ab)

Time taken for tests:   34.098 seconds

Failed requests: 4262
(Connect: 0, Receive: 1420, Length: 1420, Exceptions: 1422)
Requests per second: 2932.71 [#/sec] (mean)

Time taken for tests:   33.634 seconds

Failed requests: 7476
(Connect: 0, Receive: 2492, Length: 2492, Exceptions: 2492)
Requests per second: 2973.23 [#/sec] (mean)

もっかい

Time taken for tests:   31.841 seconds

Failed requests: 3282
(Connect: 0, Receive: 1094, Length: 1094, Exceptions: 1094)
Requests per second: 3140.63 [#/sec] (mean)

Time taken for tests:   32.033 seconds

Failed requests: 5698
(Connect: 0, Receive: 1899, Length: 1899, Exceptions: 1900)
Requests per second: 3121.83 [#/sec] (mean)


実験6

ab -c 3000 -n 30000 3台だいたい同時


highcpu.L8 (b603b7b2-d609-43da-b8fe-e4ec5ec501ab)

Time taken for tests:   51.861 seconds

Failed requests: 0
Requests per second: 578.47 [#/sec] (mean)

Time taken for tests:   51.861 seconds

Failed requests: 3
(Connect: 0, Receive: 1, Length: 1, Exceptions: 1)
Requests per second: 578.47 [#/sec] (mean)

Time taken for tests:   13.233 seconds

Complete requests: 30000
Failed requests: 120
(Connect: 0, Receive: 40, Length: 40, Exceptions: 40)
Requests per second: 2267.08 [#/sec] (mean)


実験7

ab -c 500 -n 5000 10台だいたい同時


light.S2

Time taken for tests:   4.475 seconds

Failed requests: 0
Requests per second: 1117.44 [#/sec] (mean)

Time taken for tests:   3.485 seconds

Failed requests: 0
Requests per second: 1434.60 [#/sec] (mean)

Time taken for tests:   4.714 seconds

Failed requests: 0
Requests per second: 1060.69 [#/sec] (mean)

Time taken for tests:   4.086 seconds

Failed requests: 0
Requests per second: 1223.61 [#/sec] (mean)

Time taken for tests:   4.484 seconds

Failed requests: 0
Requests per second: 1115.17 [#/sec] (mean)

Time taken for tests:   4.258 seconds

Failed requests: 0
Requests per second: 1174.36 [#/sec] (mean)

Time taken for tests:   4.682 seconds

Failed requests: 0
Requests per second: 1067.85 [#/sec] (mean)

Time taken for tests:   4.691 seconds

Failed requests: 0
Requests per second: 1065.94 [#/sec] (mean)

Time taken for tests:   3.672 seconds

Failed requests: 0
Requests per second: 1361.64 [#/sec] (mean)

10台目は、ポートフォーワードの設定が終わる前にコマンド投げてしまって取れず・・・


highcpu.L8

Time taken for tests:   3.085 seconds

Failed requests: 0
Requests per second: 1620.68 [#/sec] (mean)

Time taken for tests:   2.993 seconds

Failed requests: 0
Requests per second: 1670.81 [#/sec] (mean)

Time taken for tests:   3.348 seconds

Failed requests: 0
Requests per second: 1493.23 [#/sec] (mean)

Time taken for tests:   2.745 seconds

Failed requests: 0
Requests per second: 1821.66 [#/sec] (mean)

Time taken for tests:   3.347 seconds

Failed requests: 0
Requests per second: 1493.89 [#/sec] (mean)

Time taken for tests:   3.509 seconds

Failed requests: 0
Requests per second: 1424.76 [#/sec] (mean)

Time taken for tests:   2.678 seconds

Failed requests: 0
Requests per second: 1867.35 [#/sec] (mean)

Time taken for tests:   3.560 seconds

Failed requests: 0
Requests per second: 1404.46 [#/sec] (mean)

Time taken for tests:   2.912 seconds

Failed requests: 0
Requests per second: 1717.19 [#/sec] (mean)

Time taken for tests:   0.823 seconds

Failed requests: 0
Requests per second: 6078.38 [#/sec] (mean)

最後のは実行中に他のが終わって空きが出来たから早かったのだと思われます。

この程度だと全然エラー出ませんね。


さいごに

Screen Shot 2016-12-21 at 10.42.10 PM.png

日付変わったら何円使ったかチェックして追記します。


追記

Screen Shot 2016-12-27 at 5.45.04 PM.png

前日に5円使ってるので、280円使ったということになります。

スペック高いの使うと結構5円チョコ56個分もかかるのか・・・