BBR : Bottleneck Bandwidth and Round-trip propagation time
iperf x 10 times faster - result
References
- Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yeganeh, Van Jacobson “BBR: Congestion Based Congestion Control,” Communications of the ACM, Vol. 60 No. 2, Pages 58-66 https://queue.acm.org/detail.cfm?id=3022184
![vanjacobson8[1].png](https://qiita-user-contents.imgix.net/https%3A%2F%2Fqiita-image-store.s3.amazonaws.com%2F0%2F114634%2F71da2bbd-ac2d-7322-495d-1ecbfafd1488.png?ixlib=rb-4.0.0&auto=format&gif-q=60&q=75&s=7e2db19a25e0e5934b726b221f113878)
Congestion Control algorithms in UEK5 preview - try out BBR
https://blogs.oracle.com/wim/done-cancel-v6-
パケットロスに基づかない新しい輻輳制御の仕組み ― BBR
-
centOS7
enable Oracle Linux UEKR5
enabled=1
[ol7_developer_UEKR5]
name=Oracle Linux $releasever Unbreakable Enterprise Kernel Release 5 Packages for Development and test ($basearch)
baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/developer_UEKR5/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1
Let's Try BBR
- before BBR
iperf -c iperf.xxx -i 2 -t 30
------------------------------------------------------------
Client connecting to iperf.xxx, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local xxx.xxx.xxx.xxx port 60808 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 2.0 sec 192 KBytes 788 Kbits/sec
[ 3] 2.0- 4.0 sec 177 KBytes 724 Kbits/sec
[ 3] 4.0- 6.0 sec 256 KBytes 1.05 Mbits/sec
[ 3] 6.0- 8.0 sec 768 KBytes 3.15 Mbits/sec
[ 3] 8.0-10.0 sec 1.88 MBytes 7.86 Mbits/sec
[ 3] 10.0-12.0 sec 2.00 MBytes 8.39 Mbits/sec
[ 3] 12.0-14.0 sec 1.46 MBytes 6.14 Mbits/sec
[ 3] 14.0-16.0 sec 1.36 MBytes 5.72 Mbits/sec
[ 3] 16.0-18.0 sec 1.24 MBytes 5.21 Mbits/sec
[ 3] 18.0-20.0 sec 1.24 MBytes 5.21 Mbits/sec
[ 3] 20.0-22.0 sec 1.06 MBytes 4.43 Mbits/sec
[ 3] 22.0-24.0 sec 764 KBytes 3.13 Mbits/sec
[ 3] 24.0-26.0 sec 827 KBytes 3.39 Mbits/sec
[ 3] 26.0-28.0 sec 891 KBytes 3.65 Mbits/sec
[ 3] 28.0-30.0 sec 827 KBytes 3.39 Mbits/sec
[ 3] 0.0-30.1 sec 14.8 MBytes 4.14 Mbits/sec
- set BBR
# sysctl -w net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr
# sysctl -w net.core.default_qdisc=fq
net.core.default_qdisc = fq
cf. with UEKR4, you cannot set BBR
# sysctl -w net.ipv4.tcp_congestion_control=bbr
sysctl: setting key "net.ipv4.tcp_congestion_control": No such file or directory
net.ipv4.tcp_congestion_control = bbr
- (set permanently)
echo 'net.core.default_qdisc=fq' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_congestion_control=bbr' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
- after
iperf -c iperf.xxx -i 2 -t 30
------------------------------------------------------------
Client connecting to iperf.xxx, TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 3] local xxx.xxx.xxx.xxx port 33064 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 2.0 sec 2.38 MBytes 9.96 Mbits/sec
[ 3] 2.0- 4.0 sec 4.00 MBytes 16.8 Mbits/sec
[ 3] 4.0- 6.0 sec 5.25 MBytes 22.0 Mbits/sec
[ 3] 6.0- 8.0 sec 8.00 MBytes 33.6 Mbits/sec
[ 3] 8.0-10.0 sec 8.96 MBytes 37.6 Mbits/sec
[ 3] 10.0-12.0 sec 3.17 MBytes 13.3 Mbits/sec
[ 3] 12.0-14.0 sec 5.00 MBytes 21.0 Mbits/sec
[ 3] 14.0-16.0 sec 7.75 MBytes 32.5 Mbits/sec
[ 3] 16.0-18.0 sec 7.62 MBytes 32.0 Mbits/sec
[ 3] 18.0-20.0 sec 10.2 MBytes 43.0 Mbits/sec
[ 3] 20.0-22.0 sec 10.2 MBytes 43.0 Mbits/sec
[ 3] 22.0-24.0 sec 6.25 MBytes 26.2 Mbits/sec
[ 3] 24.0-26.0 sec 10.9 MBytes 45.6 Mbits/sec
[ 3] 26.0-28.0 sec 10.6 MBytes 44.6 Mbits/sec
[ 3] 28.0-30.0 sec 10.8 MBytes 45.1 Mbits/sec
[ 3] 0.0-30.4 sec 111 MBytes 30.7 Mbits/sec
- follow up
iperf -c www -p 80 -f M -i 1 -t 30 -e
------------------------------------------------------------
Write buffer size: 0.12 MByte
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 0.00-1.00 sec 0.62 MBytes 0.62 MBytes/sec 5/0 1 29K/38437 us
[ 3] 1.00-2.00 sec 0.88 MBytes 0.88 MBytes/sec 7/0 5 33K/39418 us
[ 3] 2.00-3.00 sec 1.00 MBytes 1.00 MBytes/sec 8/0 0 50K/37639 us
[ 3] 3.00-4.00 sec 1.00 MBytes 1.00 MBytes/sec 8/0 1 49K/38945 us
[ 3] 4.00-5.00 sec 1.62 MBytes 1.62 MBytes/sec 13/0 0 69K/37844 us
[ 3] 5.00-6.00 sec 1.75 MBytes 1.75 MBytes/sec 14/0 4 63K/38286 us
[ 3] 6.00-7.00 sec 1.25 MBytes 1.25 MBytes/sec 10/0 3 60K/40307 us
[ 3] 7.00-8.00 sec 1.38 MBytes 1.38 MBytes/sec 11/0 3 57K/38726 us
[ 3] 8.00-9.00 sec 1.75 MBytes 1.75 MBytes/sec 14/0 13 53K/37995 us
[ 3] 9.00-10.00 sec 1.50 MBytes 1.50 MBytes/sec 12/0 1 49K/39413 us
[ 3] 10.00-11.00 sec 1.38 MBytes 1.38 MBytes/sec 11/0 0 65K/40179 us
[ 3] 11.00-12.00 sec 1.75 MBytes 1.75 MBytes/sec 14/0 7 59K/38352 us
[ 3] 12.00-13.00 sec 1.25 MBytes 1.25 MBytes/sec 10/0 6 56K/38859 us
[ 3] 13.00-14.00 sec 1.12 MBytes 1.12 MBytes/sec 9/0 4 53K/37793 us
[ 3] 14.00-15.00 sec 1.12 MBytes 1.12 MBytes/sec 9/0 3 53K/40159 us
[ 3] 15.00-16.00 sec 1.50 MBytes 1.50 MBytes/sec 12/0 0 72K/37953 us
[ 3] 16.00-17.00 sec 2.00 MBytes 2.00 MBytes/sec 16/0 0 90K/37801 us
[ 3] 17.00-18.00 sec 1.75 MBytes 1.75 MBytes/sec 14/0 24 56K/40686 us
[ 3] 18.00-19.00 sec 1.38 MBytes 1.38 MBytes/sec 11/0 11 53K/38617 us
[ 3] 19.00-20.00 sec 1.50 MBytes 1.50 MBytes/sec 12/0 0 70K/37634 us
[ 3] 20.00-21.00 sec 1.50 MBytes 1.50 MBytes/sec 12/0 1 65K/38342 us
[ 3] 21.00-22.00 sec 1.88 MBytes 1.88 MBytes/sec 15/0 0 83K/37827 us
[ 3] 22.00-23.00 sec 1.75 MBytes 1.75 MBytes/sec 14/0 2 74K/38250 us
[ 3] 23.00-24.00 sec 2.00 MBytes 2.00 MBytes/sec 16/0 3 69K/41142 us
[ 3] 24.00-25.00 sec 0.88 MBytes 0.88 MBytes/sec 7/0 19 33K/38047 us
[ 3] 25.00-26.00 sec 1.00 MBytes 1.00 MBytes/sec 8/0 0 53K/40102 us
[ 3] 26.00-27.00 sec 1.50 MBytes 1.50 MBytes/sec 12/0 3 50K/37855 us
[ 3] 27.00-28.00 sec 1.25 MBytes 1.25 MBytes/sec 10/0 0 66K/38692 us
[ 3] 28.00-29.00 sec 2.00 MBytes 2.00 MBytes/sec 16/0 0 86K/38185 us
[ 3] 29.00-30.00 sec 1.62 MBytes 1.62 MBytes/sec 13/0 11 76K/38041 us
[ 3] 0.00-30.03 sec 42.9 MBytes 1.43 MBytes/sec 343/0 125 76K/38041 us
sysctl -w net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr
sysctl -w net.core.default_qdisc=fq
net.core.default_qdisc = fq
iperf -c www -p 80 -f M -i 1 -t 30 -e
------------------------------------------------------------
Write buffer size: 0.12 MByte
TCP window size: 0.12 MByte (default)
------------------------------------------------------------
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 0.00-1.00 sec 23.9 MBytes 23.9 MBytes/sec 191/0 895 2856K/33006 us
[ 3] 1.00-2.00 sec 50.8 MBytes 50.8 MBytes/sec 406/0 273 4595K/36386 us
[ 3] 2.00-3.00 sec 42.5 MBytes 42.5 MBytes/sec 340/0 562 3967K/37662 us
[ 3] 3.00-4.00 sec 41.1 MBytes 41.1 MBytes/sec 329/0 734 4457K/40878 us
[ 3] 4.00-5.00 sec 50.6 MBytes 50.6 MBytes/sec 405/0 606 3044K/32899 us
[ 3] 5.00-6.00 sec 49.0 MBytes 49.0 MBytes/sec 392/0 278 4669K/42670 us
[ 3] 6.00-7.00 sec 50.9 MBytes 50.9 MBytes/sec 407/0 239 4409K/45483 us
[ 3] 7.00-8.00 sec 43.4 MBytes 43.4 MBytes/sec 347/0 770 2638K/33185 us
[ 3] 8.00-9.00 sec 54.1 MBytes 54.1 MBytes/sec 433/0 6 4072K/43616 us
[ 3] 9.00-10.00 sec 43.8 MBytes 43.8 MBytes/sec 350/0 362 2757K/40692 us
[ 3] 10.00-11.00 sec 51.9 MBytes 51.9 MBytes/sec 415/0 177 4929K/34683 us
[ 3] 11.00-12.00 sec 50.6 MBytes 50.6 MBytes/sec 405/0 607 4686K/36481 us
[ 3] 12.00-13.00 sec 54.2 MBytes 54.2 MBytes/sec 434/0 368 5413K/34346 us
[ 3] 13.00-14.00 sec 58.6 MBytes 58.6 MBytes/sec 469/0 716 2918K/34584 us
[ 3] 14.00-15.00 sec 49.5 MBytes 49.5 MBytes/sec 396/0 632 5438K/40706 us
[ 3] 15.00-16.00 sec 47.4 MBytes 47.4 MBytes/sec 379/0 516 4626K/40840 us
[ 3] 16.00-17.00 sec 53.1 MBytes 53.1 MBytes/sec 425/0 477 2743K/49570 us
[ 3] 17.00-18.00 sec 54.9 MBytes 54.9 MBytes/sec 439/0 240 4827K/37155 us
[ 3] 18.00-19.00 sec 45.6 MBytes 45.6 MBytes/sec 365/0 442 4892K/46708 us
[ 3] 19.00-20.00 sec 48.5 MBytes 48.5 MBytes/sec 388/0 333 3085K/33415 us
[ 3] 20.00-21.00 sec 53.9 MBytes 53.9 MBytes/sec 431/0 352 4776K/39637 us
[ 3] 21.00-22.00 sec 48.6 MBytes 48.6 MBytes/sec 389/0 850 5127K/44432 us
[ 3] 22.00-23.00 sec 62.2 MBytes 62.2 MBytes/sec 498/0 400 5483K/41042 us
[ 3] 23.00-24.00 sec 59.8 MBytes 59.8 MBytes/sec 478/0 396 5376K/35238 us
[ 3] 24.00-25.00 sec 57.6 MBytes 57.6 MBytes/sec 461/0 384 4875K/45756 us
[ 3] 25.00-26.00 sec 54.2 MBytes 54.2 MBytes/sec 434/0 221 2927K/33351 us
[ 3] 26.00-27.00 sec 48.4 MBytes 48.4 MBytes/sec 387/0 878 5096K/34626 us
[ 3] 27.00-28.00 sec 47.6 MBytes 47.6 MBytes/sec 381/0 600 5331K/37356 us
[ 3] 28.00-29.00 sec 56.5 MBytes 56.5 MBytes/sec 452/0 606 2508K/33246 us
[ 3] 29.00-30.00 sec 54.6 MBytes 54.6 MBytes/sec 437/0 525 5215K/34458 us
[ 3] 0.00-30.03 sec 1508 MBytes 50.2 MBytes/sec 12063/0 14445 5215K/34458 us