Go to Qiita Advent Calendar 2024 Top
LoginSignup
11
4

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

@shirok(Shiro Kobayashi)

Oracle Cloud: 東京 - 大阪リージョン間 Remote Peering 接続 で 帯域速度 ５Gbps だしてみてみた

Last updated at Posted at 2021-12-24

Merry Christmas!
Oracle Cloud Infrastructure (OCI) の東京-大阪リージョン間バックボーン帯域幅がすごいとか、ということで 東京-大阪リージョン間 リモートVCNピアリング 構成して iperf3 コマンドでネットワーク帯域幅を測定してみてみます。

■ 構成

構成図.jpg
次の記事を参考に東京-大阪リージョン間 Remote Peering 構成し、コンピュートを配置します。
・参考: Oracle Cloud: リージョン間を Remote VCN Peering してみてみた
OCIのコンピュートは、１OCPUあたり 1Gbpsのネットワーク帯域幅が出るので、今回 25 OCPU (50vCPU) の VM.Standard.E4.Flexシェイプを使用してみてみます。

■ iperf3 事前設定

iperf3 はデフォルト 5201 Portを使用するので、Firewalldへ許可しておきます。

1) 5201/tcp 解放

# firewall-cmd --add-port=5201/tcp
	success

2) 5201/udp 解放

# firewall-cmd --add-port=5201/udp
	success

3) Firewalld 設定確認

# firewall-cmd --list-ports
	5201/tcp 5201/udp

■ UDP (User Datagram Protocol)帯域テスト

Iperf3 による UDP デフォルトは、1Mbit/sec 帯域なので、-b (--bandwidth) オプションで最適な値へ調整して実行します。

[root@tokyo-inst ~]#  iperf3 -c 10.2.0.2 -u -b 5G
	Connecting to host 10.2.0.2, port 5201
	[  4] local 10.1.0.2 port 24319 connected to 10.2.0.2 port 5201
	[ ID] Interval           Transfer     Bandwidth       Total Datagrams
	[  4]   0.00-1.00   sec   561 MBytes  4.70 Gbits/sec  65696
	[  4]   1.00-2.00   sec   596 MBytes  5.00 Gbits/sec  69801
	[  4]   2.00-3.00   sec   595 MBytes  4.99 Gbits/sec  69730
	[  4]   3.00-4.00   sec   600 MBytes  5.03 Gbits/sec  70331
	[  4]   4.00-5.00   sec   594 MBytes  4.99 Gbits/sec  69658
	[  4]   5.00-6.00   sec   592 MBytes  4.97 Gbits/sec  69407
	[  4]   6.00-7.00   sec   596 MBytes  5.00 Gbits/sec  69887
	[  4]   7.00-8.00   sec   596 MBytes  5.00 Gbits/sec  69845
	[  4]   8.00-9.00   sec   596 MBytes  5.00 Gbits/sec  69851
	[  4]   9.00-10.00  sec   596 MBytes  5.00 Gbits/sec  69892
	- - - - - - - - - - - - - - - - - - - - - - - - -
	[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
	[  4]   0.00-10.00  sec  5.78 GBytes  4.97 Gbits/sec  0.007 ms  143514/694098 (21%)
	[  4] Sent 694098 datagrams

	iperf Done.

■ TCP (Transmission Control Protocol)帯域テスト

TCPの場合、Linuxカーネル・パラメーターのTCPチューニングをして帯域速度が出るようにしてみます。

● Linuxカーネル・パラメーター・チューニング

次のパラメーターで最適サイズを調整します。

rmem: receive（read）buffer サイズ(Byte)
wmem: send（write）buffer サイズ(Byte)

net.ipv4.tcp_memカーネルパラメーターでそれぞれの min / pressure / max を設定してみてみます。

● Linuxカーネル・パラメーター設定

1) TCP送信/受信バッファ・サイズ設定
Client と Server側両方に TCP送信/受信バッファ・サイズをより帯域幅でるように OS設定を調整します。

Client/Server
# sudo sysctl -w net.ipv4.tcp_wmem="4096    16384   16777216"
# sudo sysctl -w net.ipv4.tcp_rmem="4096    131072  16777216"
# sudo sysctl -w net.core.wmem_max=16777216
# sudo sysctl -w net.core.rmem_max=16777216

永続設定する場合は、 /etc/sysctl.conf へ設定し、sysctl -p で設定値を反映させます。

2) 設定確認

Client/Server
# sysctl -a | egrep "wmem|rmem"
	net.core.rmem_default = 212992
	net.core.rmem_max = 16777216
	net.core.wmem_default = 212992
	net.core.wmem_max = 16777216
	net.ipv4.tcp_rmem = 4096	131072	16777216
	net.ipv4.tcp_wmem = 4096	16384	16777216

● IPerf3 実行

iperf3 コマンドには、いくつかオプションがありますが、-b (--bandwidth) で MAX値あたりで帯域制御してパケットドロップを回避するようにすると高値が出やすくなります。

[root@tokyo-inst ~]# iperf3 -c 10.2.0.2 -b 4.8G
	Connecting to host 10.2.0.2, port 5201
	[  4] local 10.1.0.2 port 57682 connected to 10.2.0.2 port 5201
	[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
	[  4]   0.00-1.00   sec   552 MBytes  4.63 Gbits/sec    0   8.43 MBytes
	[  4]   1.00-2.00   sec   565 MBytes  4.74 Gbits/sec    7   6.02 MBytes
	[  4]   2.00-3.00   sec   526 MBytes  4.41 Gbits/sec    4   4.43 MBytes
	[  4]   3.00-4.00   sec   509 MBytes  4.27 Gbits/sec    0   4.93 MBytes
	[  4]   4.00-5.00   sec   563 MBytes  4.72 Gbits/sec    0   5.43 MBytes
	[  4]   5.00-6.00   sec   600 MBytes  5.03 Gbits/sec  259   2.21 MBytes
	[  4]   6.00-7.00   sec   480 MBytes  4.03 Gbits/sec  193   4.71 MBytes
	[  4]   7.00-8.00   sec   535 MBytes  4.49 Gbits/sec    0   5.15 MBytes
	[  4]   8.00-9.00   sec   583 MBytes  4.89 Gbits/sec    0   5.59 MBytes
	[  4]   9.00-10.00  sec   631 MBytes  5.29 Gbits/sec    0   6.01 MBytes
	- - - - - - - - - - - - - - - - - - - - - - - - -
	[ ID] Interval           Transfer     Bandwidth       Retr
	[  4]   0.00-10.00  sec  5.41 GBytes  4.65 Gbits/sec  463             sender
	[  4]   0.00-10.00  sec  5.41 GBytes  4.65 Gbits/sec                  receiver

● IPerf3 Parallel 実行

今回の OCI コンピュート・シェイプは VM.Standard.E4.Flex (AMD) 1 OCPU当たり 1Gbps最大ネットワーク帯域幅なので 25 OCPU 用意し、パラレル実行で 25 Gbits/sec 出ることを確認してみてみます。

[root@tokyo-inst ~]# iperf3 -c 10.2.0.2 -P 5
	Connecting to host 10.2.0.2, port 5201
	[  4] local 10.1.0.2 port 10048 connected to 10.2.0.2 port 5201
	[  6] local 10.1.0.2 port 10050 connected to 10.2.0.2 port 5201
	[  8] local 10.1.0.2 port 10052 connected to 10.2.0.2 port 5201
	[ 10] local 10.1.0.2 port 10054 connected to 10.2.0.2 port 5201
	[ 12] local 10.1.0.2 port 10056 connected to 10.2.0.2 port 5201
	[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
	[  4]   0.00-1.00   sec   610 MBytes  5.12 Gbits/sec    0   16.1 MBytes
	[  6]   0.00-1.00   sec   576 MBytes  4.83 Gbits/sec    0   8.08 MBytes
	[  8]   0.00-1.00   sec   564 MBytes  4.73 Gbits/sec    0   8.77 MBytes
	[ 10]   0.00-1.00   sec   617 MBytes  5.17 Gbits/sec    0   15.4 MBytes
	[ 12]   0.00-1.00   sec   573 MBytes  4.80 Gbits/sec    1   11.3 MBytes
	[SUM]   0.00-1.00   sec  2.87 GBytes  24.7 Gbits/sec    1
	- - - - - - - - - - - - - - - - - - - - - - - - -
	[  4]   1.00-2.00   sec   644 MBytes  5.40 Gbits/sec    0   16.1 MBytes
	[  6]   1.00-2.00   sec   609 MBytes  5.11 Gbits/sec    0   8.08 MBytes
	[  8]   1.00-2.00   sec   599 MBytes  5.02 Gbits/sec    0   8.77 MBytes
	[ 10]   1.00-2.00   sec   642 MBytes  5.39 Gbits/sec    0   15.4 MBytes
	[ 12]   1.00-2.00   sec   609 MBytes  5.11 Gbits/sec    0   11.3 MBytes
	[SUM]   1.00-2.00   sec  3.03 GBytes  26.0 Gbits/sec    0
	- - - - - - - - - - - - - - - - - - - - - - - - -
	[  4]   2.00-3.00   sec   642 MBytes  5.39 Gbits/sec    0   16.1 MBytes
	[  6]   2.00-3.00   sec   608 MBytes  5.10 Gbits/sec    0   8.08 MBytes
	[  8]   2.00-3.00   sec   601 MBytes  5.04 Gbits/sec    0   8.77 MBytes
	[ 10]   2.00-3.00   sec   644 MBytes  5.40 Gbits/sec    0   15.4 MBytes
	[ 12]   2.00-3.00   sec   609 MBytes  5.11 Gbits/sec    0   11.3 MBytes
	[SUM]   2.00-3.00   sec  3.03 GBytes  26.0 Gbits/sec    0
	- - - - - - - - - - - - - - - - - - - - - - - - -
	[  4]   3.00-4.00   sec   631 MBytes  5.30 Gbits/sec   25   11.3 MBytes
	[  6]   3.00-4.00   sec   610 MBytes  5.12 Gbits/sec    0   8.08 MBytes
	[  8]   3.00-4.00   sec   601 MBytes  5.04 Gbits/sec    0   8.77 MBytes
	[ 10]   3.00-4.00   sec   645 MBytes  5.41 Gbits/sec    0   15.4 MBytes
	[ 12]   3.00-4.00   sec   612 MBytes  5.14 Gbits/sec    0   11.3 MBytes
	[SUM]   3.00-4.00   sec  3.03 GBytes  26.0 Gbits/sec   25
	- - - - - - - - - - - - - - - - - - - - - - - - -
	[  4]   4.00-5.00   sec   620 MBytes  5.20 Gbits/sec   41   3.89 MBytes
	[  6]   4.00-5.00   sec   616 MBytes  5.17 Gbits/sec    0   8.08 MBytes
	[  8]   4.00-5.00   sec   609 MBytes  5.11 Gbits/sec    0   8.77 MBytes
	[ 10]   4.00-5.00   sec   651 MBytes  5.46 Gbits/sec    0   15.4 MBytes
	[ 12]   4.00-5.00   sec   604 MBytes  5.06 Gbits/sec    7   7.96 MBytes
	[SUM]   4.00-5.00   sec  3.03 GBytes  26.0 Gbits/sec   48
	- - - - - - - - - - - - - - - - - - - - - - - - -
	[  4]   5.00-6.00   sec   366 MBytes  3.07 Gbits/sec    0   4.29 MBytes
	[  6]   5.00-6.00   sec   681 MBytes  5.71 Gbits/sec    0   8.08 MBytes
	[  8]   5.00-6.00   sec   664 MBytes  5.57 Gbits/sec    0   8.77 MBytes
	[ 10]   5.00-6.00   sec   720 MBytes  6.04 Gbits/sec    0   15.4 MBytes
	[ 12]   5.00-6.00   sec   671 MBytes  5.63 Gbits/sec    0   8.00 MBytes
	[SUM]   5.00-6.00   sec  3.03 GBytes  26.0 Gbits/sec    0
	- - - - - - - - - - - - - - - - - - - - - - - - -
	[  4]   6.00-7.00   sec   402 MBytes  3.38 Gbits/sec    0   4.69 MBytes
	[  6]   6.00-7.00   sec   670 MBytes  5.62 Gbits/sec    0   8.08 MBytes
	[  8]   6.00-7.00   sec   658 MBytes  5.52 Gbits/sec    0   8.77 MBytes
	[ 10]   6.00-7.00   sec   709 MBytes  5.95 Gbits/sec    0   15.4 MBytes
	[ 12]   6.00-7.00   sec   664 MBytes  5.57 Gbits/sec    0   8.00 MBytes
	[SUM]   6.00-7.00   sec  3.03 GBytes  26.0 Gbits/sec    0
	- - - - - - - - - - - - - - - - - - - - - - - - -
	[  4]   7.00-8.00   sec   370 MBytes  3.10 Gbits/sec    1   3.70 MBytes
	[  6]   7.00-8.00   sec   675 MBytes  5.66 Gbits/sec    0   8.08 MBytes
	[  8]   7.00-8.00   sec   664 MBytes  5.57 Gbits/sec    0   8.77 MBytes
	[ 10]   7.00-8.00   sec   719 MBytes  6.03 Gbits/sec    0   15.4 MBytes
	[ 12]   7.00-8.00   sec   674 MBytes  5.65 Gbits/sec    0   8.00 MBytes
	[SUM]   7.00-8.00   sec  3.03 GBytes  26.0 Gbits/sec    1
	- - - - - - - - - - - - - - - - - - - - - - - - -
	[  4]   8.00-9.00   sec   356 MBytes  2.99 Gbits/sec    0   4.14 MBytes
	[  6]   8.00-9.00   sec   680 MBytes  5.70 Gbits/sec    0   8.08 MBytes
	[  8]   8.00-9.00   sec   668 MBytes  5.60 Gbits/sec    0   8.77 MBytes
	[ 10]   8.00-9.00   sec   721 MBytes  6.05 Gbits/sec    0   15.4 MBytes
	[ 12]   8.00-9.00   sec   679 MBytes  5.69 Gbits/sec    0   8.00 MBytes
	[SUM]   8.00-9.00   sec  3.03 GBytes  26.0 Gbits/sec    0
	- - - - - - - - - - - - - - - - - - - - - - - - -
	[  4]   9.00-10.00  sec   409 MBytes  3.43 Gbits/sec    0   4.46 MBytes
	[  6]   9.00-10.00  sec   710 MBytes  5.96 Gbits/sec    0   8.08 MBytes
	[  8]   9.00-10.00  sec   699 MBytes  5.86 Gbits/sec    0   8.77 MBytes
	[ 10]   9.00-10.00  sec   755 MBytes  6.33 Gbits/sec    0   15.4 MBytes
	[ 12]   9.00-10.00  sec   529 MBytes  4.44 Gbits/sec    1   6.00 MBytes
	[SUM]   9.00-10.00  sec  3.03 GBytes  26.0 Gbits/sec    1
	- - - - - - - - - - - - - - - - - - - - - - - - -
	[ ID] Interval           Transfer     Bandwidth       Retr
	[  4]   0.00-10.00  sec  4.93 GBytes  4.24 Gbits/sec   67             sender
	[  4]   0.00-10.00  sec  4.93 GBytes  4.23 Gbits/sec                  receiver
	[  6]   0.00-10.00  sec  6.28 GBytes  5.40 Gbits/sec    0             sender
	[  6]   0.00-10.00  sec  6.28 GBytes  5.40 Gbits/sec                  receiver
	[  8]   0.00-10.00  sec  6.18 GBytes  5.31 Gbits/sec    0             sender
	[  8]   0.00-10.00  sec  6.17 GBytes  5.30 Gbits/sec                  receiver
	[ 10]   0.00-10.00  sec  6.66 GBytes  5.72 Gbits/sec    0             sender
	[ 10]   0.00-10.00  sec  6.66 GBytes  5.72 Gbits/sec                  receiver
	[ 12]   0.00-10.00  sec  6.08 GBytes  5.22 Gbits/sec    9             sender
	[ 12]   0.00-10.00  sec  6.07 GBytes  5.22 Gbits/sec                  receiver
	[SUM]   0.00-10.00  sec  30.1 GBytes  25.9 Gbits/sec   76             sender
	[SUM]   0.00-10.00  sec  30.1 GBytes  25.9 Gbits/sec                  receiver

	iperf Done.

Merry Christmas!

■ 参考

・ネットワーク・パフォーマンス
・iPerf 3 user documentation
・OCIコンピュート・シェイプ
・The Oracle Cloud Infrastructure backbone

11
4
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
11
4

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?