4
1

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

iperf2 で 100Gbps ネットワーク帯域幅を計測してみてみた

Last updated at Posted at 2024-09-09

100Gbps 帯域幅を計測するにはテクニックが必要です。
Fasterdata: Throughput Tool Comparision より〜
Based on our experience, we recommend the following:
・Use iperf2 for parallel streams, bidirectional, or MS Windows-based tests
・Use nuttcp for high-speed UDP testing
・Use iperf3 otherwise.
これらのツールはそれぞれ機能が若干異なり、アーキテクチャも若干異なるため、1 つのツールに必要なものがすべて揃っているとは思わないでください。複数のツールに精通し、特定のユースケースに適したツールを使用するのが最善です。
重要な違いの 1 つは、ツールがシングルスレッドかマルチスレッドかということです。並列ストリームのパフォーマンスをテストする場合は、iperf2などのマルチスレッド ツールを使用する必要があります。

ということで、iperf2 はマルチスレッドなので、iperf3よりも簡単に高速な帯域幅を計測できそうです。前回作成した 100 Gbps NICを搭載した
Oracle Cloud Infrastructure (OCI) Compute BM.Standard.E5を2台用意して、計測してみてみます。

■ 構成図

構成図.jpg

■ iperf2 インストール

iperf の古いバージョンにはバグがあったことに注意してください。
さらに新しいバージョンには多くの優れた機能がありますので、必ず新しいリリースの iperf2 を使用してください。

● iperf2 Download

1) dnf search
リポジトリにパッケージがあるか検索し、無い場合は Downloadします。

[root@bm-e5 ~]# dnf search iperf
This system is receiving updates from OSMS server.
Last metadata expiration check: 0:01:29 ago on Mon 09 Sep 2024 05:55:49 AM GMT.
=================================================================== Name Matched: iperf ===================================================================
iperf3.x86_64 : Measurement tool for TCP/UDP bandwidth performance
iperf3.i686 : Measurement tool for TCP/UDP bandwidth performance

2) iperf2 Download
iperf2 は iperf の名前になっているので、 iperfで検索してDownloadします。
Oracle Linux の場合は、EPLからダウンロードできます。
Oracle Linux 9 (x86_64) EPEL
Oracle Linux 8 (x86_64) EPEL

01_EPEL01.png
01_EPEL02.png

● iperf2 install

1) iperf2 install
dnf install で iperfのURLを指定もしくは、ダウンロードしたパッケージを指定して Install

[root@bm-e5 ~]# dnf install https://yum.oracle.com/repo/OracleLinux/OL9/developer/EPEL/x86_64/getPackage/iperf-2.1.6-2.el9.x86_64.rpm
Last metadata expiration check: 3:15:28 ago on Sun 08 Sep 2024 09:09:02 PM GMT.
iperf-2.1.6-2.el9.x86_64.rpm                                                            3.2 MB/s | 146 kB     00:00
Dependencies resolved.
========================================================================================================================
 Package                 Architecture             Version                          Repository                      Size
========================================================================================================================
Installing:
 iperf                   x86_64                   2.1.6-2.el9                      @commandline                   146 k

Transaction Summary
========================================================================================================================
Install  1 Package

Total size: 146 k
Installed size: 302 k
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                1/1
  Installing       : iperf-2.1.6-2.el9.x86_64                                                                       1/1
  Running scriptlet: iperf-2.1.6-2.el9.x86_64                                                                       1/1
  Verifying        : iperf-2.1.6-2.el9.x86_64                                                                       1/1

Installed:
  iperf-2.1.6-2.el9.x86_64

Complete!

2) iperf2 install確認

[root@bm-e5 ~]# iperf -v
iperf version 2.1.6 (10 December 2021) pthreads

[root@bm-e5 ~]# iperf -h
Usage: iperf [-s|-c host] [options]
       iperf [-h|--help] [-v|--version]

Client/Server:
  -b, --bandwidth #[kmgKMG | pps]  bandwidth to read/send at in bits/sec or packets/sec
  -e, --enhanced    use enhanced reporting giving more tcp/udp and traffic information
  -f, --format    [kmgKMG]   format to report: Kbits, Mbits, KBytes, MBytes
      --hide-ips           hide ip addresses and host names within outputs
  -i, --interval  #        seconds between periodic bandwidth reports
  -l, --len       #[kmKM]    length of buffer in bytes to read or write (Defaults: TCP=128K, v4 UDP=1470, v6 UDP=1450)
  -m, --print_mss          print TCP maximum segment size (MTU - TCP/IP header)
  -o, --output    <filename> output the report or error message to this specified file
  -p, --port      #        client/server port to listen/send on and to connect
      --permit-key         permit key to be used to verify client and server (TCP only)
      --sum-only           output sum only reports
  -u, --udp                use UDP rather than TCP
  -w, --window    #[KM]    TCP window size (socket buffer size)
  -z, --realtime           request realtime scheduler
  -B, --bind <host>[:<port>][%<dev>] bind to <host>, ip addr (including multicast address) and optional port and device
  -C, --compatibility      for use with older versions does not sent extra msgs
  -M, --mss       #        set TCP maximum segment size (MTU - 40 bytes)
  -N, --nodelay            set TCP no delay, disabling Nagle's Algorithm
  -S, --tos       #        set the socket's IP_TOS (byte) field
  -Z, --tcp-congestion <algo>  set TCP congestion control algorithm (Linux only)

Server specific:
  -p, --port      #[-#]    server port(s) to listen on/connect to
  -s, --server             run in server mode
  -1, --singleclient       run one server at a time
      --histograms         enable latency histograms
      --permit-key-timeout set the timeout for a permit key in seconds
      --tcp-rx-window-clamp set the TCP receive window clamp size in bytes
      --tap-dev   #[<dev>] use TAP device to receive at L2 layer
  -t, --time      #        time in seconds to listen for new connections as well as to receive traffic (default not set)
      --udp-histogram #,#  enable UDP latency histogram(s) with bin width and count, e.g. 1,1000=1(ms),1000(bins)
  -B, --bind <ip>[%<dev>]  bind to multicast address and optional device
  -U, --single_udp         run in single threaded UDP mode
      --sum-dstip          sum traffic threads based upon destination ip address (default is src ip)
  -D, --daemon             run the server as a daemon
  -V, --ipv6_domain        Enable IPv6 reception by setting the domain and socket to AF_INET6 (Can receive on both IPv4 and IPv6)

Client specific:
  -c, --client    <host>   run in client mode, connecting to <host>
      --connect-only       run a connect only test
      --connect-retries #  number of times to retry tcp connect
  -d, --dualtest           Do a bidirectional test simultaneously (multiple sockets)
      --fq-rate #[kmgKMG]  bandwidth to socket pacing
      --full-duplex        run full duplex test using same socket
      --ipg                set the the interpacket gap (milliseconds) for packets within an isochronous frame
      --isochronous <frames-per-second>:<mean>,<stddev> send traffic in bursts (frames - emulate video traffic)
      --incr-dstip         Increment the destination ip with parallel (-P) traffic threads
      --incr-dstport       Increment the destination port with parallel (-P) traffic threads
      --incr-srcip         Increment the source ip with parallel (-P) traffic threads
      --incr-srcport       Increment the source port with parallel (-P) traffic threads
      --local-only         Set don't route on socket
      --near-congestion=[w] Use a weighted write delay per the sampled TCP RTT (experimental)
      --no-connect-sync    No sychronization after connect when -P or parallel traffic threads
      --no-udp-fin         No final server to client stats at end of UDP test
  -n, --num       #[kmgKMG]    number of bytes to transmit (instead of -t)
  -r, --tradeoff           Do a fullduplexectional test individually
      --tcp-write-prefetch set the socket's TCP_NOTSENT_LOWAT value in bytes and use event based writes
  -t, --time      #        time in seconds to transmit for (default 10 secs)
      --trip-times         enable end to end measurements (requires client and server clock sync)
      --txdelay-time       time in seconds to hold back after connect and before first write
      --txstart-time       unix epoch time to schedule first write and start traffic
  -B, --bind [<ip> | <ip:port>] bind ip (and optional port) from which to source traffic
  -F, --fileinput <name>   input the data to be transmitted from a file
  -H, --ssm-host <ip>      set the SSM source, use with -B for (S,G)
  -I, --stdin              input the data to be transmitted from stdin
  -L, --listenport #       port to receive fullduplexectional tests back on
  -P, --parallel  #        number of parallel client threads to run
  -R, --reverse            reverse the test (client receives, server sends)
  -S, --tos                IP DSCP or tos settings
  -T, --ttl       #        time-to-live, for multicast (default 1)
  -V, --ipv6_domain        Set the domain to IPv6 (send packets over IPv6)
  -X, --peer-detect        perform server version detection and version exchange

Miscellaneous:
  -x, --reportexclude [CDMSV]   exclude C(connection) D(data) M(multicast) S(settings) V(server) reports
  -y, --reportstyle C      report as a Comma-Separated Values
  -h, --help               print this message and quit
  -v, --version            print version information and quit

[kmgKMG] Indicates options that support a k,m,g,K,M or G suffix
Lowercase format characters are 10^3 based and uppercase are 2^n based
(e.g. 1k = 1000, 1K = 1024, 1m = 1,000,000 and 1M = 1,048,576)

The TCP window size option can be set by the environment variable
TCP_WINDOW_SIZE. Most other options can be set by an environment variable
IPERF_<long option name>, such as IPERF_BANDWIDTH.

Source at <http://sourceforge.net/projects/iperf2/>
Report bugs to <iperf-users@lists.sourceforge.net>

■ iperf2 帯域計測

● Firewall 設定

iperf2 はデフォルト 5001 Portを使用するので Firewallを設定している場合はPort解放します。
ここでは、Firewalldが設定されているので次のコマンドを実行

[root@bm-e5 ~]# firewall-cmd --add-port=5001/tcp

● カーネルのネットワーク設定

Client と Server側両方に 帯域幅でるようにTCP送信/受信バッファ・サイズを
帯域幅遅延積をしようして調整する必要があります。

1) カーネル・パラメータ設定
今回はこの値で実施します。実検証では帯域幅遅延積の求め方から計算して値を決めます。

Client/Server
# sudo sysctl -w net.ipv4.tcp_wmem="4096    16384   16777216"
# sudo sysctl -w net.ipv4.tcp_rmem="4096    131072  16777216"
# sudo sysctl -w net.core.wmem_max=16777216
# sudo sysctl -w net.core.rmem_max=16777216

永続設定する場合は、 /etc/sysctl.conf へ設定し、sysctl -p で設定値を反映させます。

2) 設定確認

Client/Server
# sysctl -a | egrep "wmem|rmem"
	net.core.rmem_default = 212992
	net.core.rmem_max = 16777216
	net.core.wmem_default = 212992
	net.core.wmem_max = 16777216
	net.ipv4.tcp_rmem = 4096	131072	16777216
	net.ipv4.tcp_wmem = 4096	16384	16777216

● iperf2実行

iperf2 は、 ServerとClientでコマンドを実行して通信して計測します。
次の例は、Client側で 送信帯域幅と並列スレッド数を指定して、最大帯域が出るよう調整します。

iperf実効例
Server: iperf -s
Client: iperf -c <Servcer IP Address> -b <Bite Size>  -P <並列度>

1) Server側: iperf2実行

[root@bm-e5-server ~]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------

2) Client側: iperf2実行
-b 帯域幅 と -P 並列数 を調整して 100Gbp出るよう設定します。
今回は、9Gbpsを11並列に設定して実行します。

[root@bm-e5-client ~]$ iperf -c 10.10.0.22  -b 9G -P 11
[ 10] local 10.10.0.23 port 38784 connected with 10.10.0.22 port 5001
[  7] local 10.10.0.23 port 38776 connected with 10.10.0.22 port 5001
[  2] local 10.10.0.23 port 38750 connected with 10.10.0.22 port 5001
------------------------------------------------------------
Client connecting to 10.10.0.22, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[  1] local 10.10.0.23 port 38732 connected with 10.10.0.22 port 5001
[  5] local 10.10.0.23 port 38762 connected with 10.10.0.22 port 5001
[  6] local 10.10.0.23 port 38728 connected with 10.10.0.22 port 5001
[  8] local 10.10.0.23 port 38774 connected with 10.10.0.22 port 5001
[  9] local 10.10.0.23 port 38786 connected with 10.10.0.22 port 5001
[  4] local 10.10.0.23 port 38754 connected with 10.10.0.22 port 5001
[ 11] local 10.10.0.23 port 38798 connected with 10.10.0.22 port 5001
[  3] local 10.10.0.23 port 38748 connected with 10.10.0.22 port 5001
[ ID] Interval       Transfer     Bandwidth
[  2] 0.00-10.01 sec  11.1 GBytes  9.51 Gbits/sec
[  7] 0.00-10.01 sec  11.2 GBytes  9.64 Gbits/sec
[  5] 0.00-10.01 sec  9.20 GBytes  7.89 Gbits/sec
[  8] 0.00-10.01 sec  10.8 GBytes  9.23 Gbits/sec
[  1] 0.00-10.01 sec  10.7 GBytes  9.14 Gbits/sec
[  3] 0.00-10.01 sec  11.2 GBytes  9.59 Gbits/sec
[ 10] 0.00-10.01 sec  10.8 GBytes  9.28 Gbits/sec
[  9] 0.00-10.01 sec  10.4 GBytes  8.88 Gbits/sec
[  4] 0.00-10.01 sec  10.7 GBytes  9.14 Gbits/sec
[ 11] 0.00-10.01 sec  8.66 GBytes  7.43 Gbits/sec
[  6] 0.00-10.01 sec  10.2 GBytes  8.74 Gbits/sec
[SUM] 0.00-10.00 sec   115 GBytes  98.6 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) = 0.219/0.319/0.443/0.067 ms (tot/err) = 11/0

■ 参考

 ・Fasterdata: Throughput Tool Comparision
 ・iperf2 / iperf3
 ・iperf3 FAQ
 ・iPerf - The ultimate speed test tool for TCP, UDP and SCTP
 ・NUTTCP
 ・帯域幅遅延積の求め方

4
1
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
4
1

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?