目的
Microsoft AzureとIBMCloudをインターネットを介したIPsecVPNをActive/Activeで接続する。
次に、BGPを用いた経路交換を実施し、AzureとIBMCloud間で仮想マシン間の通信が可能か確認する。
最後に、インタフェース障害や機器障害を想定したテストを実施し、問題なく通信が可能か確認する。
細かすぎる説明は省略する場合があります。
基本的には、Azure x IBMCloud IPsecVPN/BGP接続の記事の発展形となる。
そのため、細かい説明は上記の記事を参照してください。
構成
- ローカル端末：macOS Big Sur 11.6 1台
- プラットフォーム:Microsoft Azure
- 仮想ネットワーク:PublicIPアドレス/仮想NWゲートウェイ/ローカルNWゲートウェイ 2サービス
- Virtual Machines:CentOS 7.9 1台
- プラットフォーム:IBMCloud
- ルータ:Vyatta AT&T vRouter 5600 1912g 2台
- 仮想サーバ:CentOS 7.7(Virtual Server for Classic) 1台
構成図
Azureサービスのデプロイ
Active/ActiveでIPsecVPNをデプロイする際の変更点のみ記載する。
Azure VPN Gateway(仮想ネットワークゲートウェイ)の作成
アクティブ/アクティブの有効化と、２つのアドレスに対してBGP接続するための設定
Azure ローカルネットワークゲートウェイの作成
Azure側の２台に対して、IBMCloud側も2台のルータで接続するため、ローカルネットワークゲートウェイも2台分作成する。
1台目:IBMCloud#1の定義
2台目:IBMCloud#2の定義
Azure接続用IPsecVPNの設定
２台分のIPSecVPNの設定が必要
１台目：Azure#1 ~ IBMCloud#1間
2台目：Azure#2 ~ IBMCloud#2間
IBMCloud側の設定(Vyatta/VRA)
IPsecVPN
1号機側の設定
20.222.88.aaaはAzure側のGW#1
128.168.104.aaaはIBMCloud側のVRA#1
set interfaces vti vti0 address 169.254.21.1/30
set interfaces vti vti0 description Azure
set interfaces vti vti0 mtu 1436
set security vpn ipsec esp-group ESP-Azure compression disable
set security vpn ipsec esp-group ESP-Azure lifetime 3600
set security vpn ipsec esp-group ESP-Azure mode tunnel
set security vpn ipsec esp-group ESP-Azure pfs dh-group20
set security vpn ipsec esp-group ESP-Azure proposal 1 encryption aes256
set security vpn ipsec esp-group ESP-Azure proposal 1 hash sha2_256
set security vpn ipsec ike-group IKE-Azure dead-peer-detection action restart
set security vpn ipsec ike-group IKE-Azure dead-peer-detection interval 10
set security vpn ipsec ike-group IKE-Azure dead-peer-detection timeout 30
set security vpn ipsec ike-group IKE-Azure ike-version 2
set security vpn ipsec ike-group IKE-Azure lifetime 3600
set security vpn ipsec ike-group IKE-Azure proposal 1 dh-group 20
set security vpn ipsec ike-group IKE-Azure proposal 1 encryption aes256
set security vpn ipsec ike-group IKE-Azure proposal 1 hash sha2_256
set security vpn ipsec site-to-site peer 20.222.88.aaa authentication mode pre-shared-secret
set security vpn ipsec site-to-site peer 20.222.88.aaa authentication pre-shared-secret ibmcloudazure
set security vpn ipsec site-to-site peer 20.222.88.aaa connection-type initiate
set security vpn ipsec site-to-site peer 20.222.88.aaa default-esp-group ESP-Azure
set security vpn ipsec site-to-site peer 20.222.88.aaa ike-group IKE-Azure
set security vpn ipsec site-to-site peer 20.222.88.aaa local-address 128.168.104.aaa
set security vpn ipsec site-to-site peer 20.222.88.aaa vti bind vti0
set security vpn ipsec site-to-site peer 20.222.88.aaa vti esp-group ESP-Azure
2号機側の設定
20.222.89.bbbはAzure側のGW#2
128.168.104.bbbはIBMCloud側のVRA#2
set interfaces vti vti0 address 169.254.22.1/30
set interfaces vti vti0 description Azure
set interfaces vti vti0 mtu 1436
set security vpn ipsec esp-group ESP-Azure compression disable
set security vpn ipsec esp-group ESP-Azure lifetime 3600
set security vpn ipsec esp-group ESP-Azure mode tunnel
set security vpn ipsec esp-group ESP-Azure pfs dh-group20
set security vpn ipsec esp-group ESP-Azure proposal 1 encryption aes256
set security vpn ipsec esp-group ESP-Azure proposal 1 hash sha2_256
set security vpn ipsec ike-group IKE-Azure dead-peer-detection action restart
set security vpn ipsec ike-group IKE-Azure dead-peer-detection interval 10
set security vpn ipsec ike-group IKE-Azure dead-peer-detection timeout 30
set security vpn ipsec ike-group IKE-Azure ike-version 2
set security vpn ipsec ike-group IKE-Azure lifetime 3600
set security vpn ipsec ike-group IKE-Azure proposal 1 dh-group 20
set security vpn ipsec ike-group IKE-Azure proposal 1 encryption aes256
set security vpn ipsec ike-group IKE-Azure proposal 1 hash sha2_256
set security vpn ipsec site-to-site peer 20.222.89.bbb authentication mode pre-shared-secret
set security vpn ipsec site-to-site peer 20.222.89.bbb authentication pre-shared-secret ibmcloudazure
set security vpn ipsec site-to-site peer 20.222.89.bbb connection-type initiate
set security vpn ipsec site-to-site peer 20.222.89.bbb default-esp-group ESP-Azure
set security vpn ipsec site-to-site peer 20.222.89.bbb ike-group IKE-Azure
set security vpn ipsec site-to-site peer 20.222.89.bbb local-address 128.168.104.bbb
set security vpn ipsec site-to-site peer 20.222.89.bbb vti bind vti0
set security vpn ipsec site-to-site peer 20.222.89.bbb vti esp-group ESP-Azure
BGPの設定(IBMCloud ~ Azure間)
1号機側の設定
set protocols bgp 65001 address-family ipv4-unicast network 10.192.145.192/26
set protocols bgp 65001 neighbor 169.254.21.2 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp 65001 neighbor 169.254.21.2 remote-as 65515
set protocols bgp 65001 neighbor 169.254.21.2 timers holdtime 30
set protocols bgp 65001 neighbor 169.254.21.2 timers keepalive 10
set protocols bgp 65001 parameters router-id 169.254.21.1
2号機側の設定
set protocols bgp 65001 address-family ipv4-unicast network 10.192.145.192/26
set protocols bgp 65001 neighbor 169.254.22.2 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp 65001 neighbor 169.254.22.2 remote-as 65515
set protocols bgp 65001 neighbor 169.254.22.2 timers holdtime 30
set protocols bgp 65001 neighbor 169.254.22.2 timers keepalive 10
set protocols bgp 65001 parameters router-id 169.254.22.1
OSPFの設定(VRA１号機〜VRA２号機間)
1号機側の設定
10.192.94.146はVRA#1のdp0bond0
10.192.94.175はVRA#2のdp0bond0
set interfaces tunnel tun0 address 172.31.255.1/30
set interfaces tunnel tun0 encapsulation gre
set interfaces tunnel tun0 local-ip 10.192.94.146
set interfaces tunnel tun0 multicast enable
set interfaces tunnel tun0 remote-ip 10.192.94.175
set protocols ospf area 0.0.0.0 network 172.31.255.0/30
set protocols ospf parameters router-id 172.31.255.1
set protocols ospf passive-interface default
set protocols ospf passive-interface-exclude tun0
set protocols ospf redistribute bgp
2号機側の設定
10.192.94.175はVRA#2のdp0bond0
10.192.94.146はVRA#1のdp0bond0
set interfaces tunnel tun0 address 172.31.255.2/30
set interfaces tunnel tun0 encapsulation gre
set interfaces tunnel tun0 local-ip 10.192.94.175
set interfaces tunnel tun0 multicast enable
set interfaces tunnel tun0 remote-ip 10.192.94.146
set protocols ospf area 0.0.0.0 network 172.31.255.0/30
set protocols ospf parameters router-id 172.31.255.2
set protocols ospf passive-interface default
set protocols ospf passive-interface-exclude tun0
set protocols ospf redistribute bgp
Azure側状態確認
IPsecVPNの状態
BGPの状態
IBMCloud側状態確認
VPNの状態
1号機側
vyatta@kou-vra001:~$ show vpn ike sa
Peer ID / IP Local ID / IP
------------ -------------
20.222.88.aaa 128.168.104.aaa
State Encrypt Hash D-H Grp A-Time L-Time IKEv
----- ------------ -------- ------- ------ ------ ----
up aes256 sha2_256 20 1740 3600 2
vyatta@kou-vra001:~$ show vpn ipsec sa
Peer ID / IP Local ID / IP
------------ -------------
20.222.88.aaa 128.168.104.aaa
Tunnel Id State Bytes Out/In Encrypt Hash DH A-Time L-Time
------ ---------- ----- ------------- ------------ -------- -- ------ ------
vti 7 up 22.2K/24.0K aes256 sha2_256 20 19757 3600
2号機側:VPN
vyatta@kou-vra002:~$ show vpn ike sa
Peer ID / IP Local ID / IP
------------ -------------
20.222.89.bbb 128.168.104.bbb
State Encrypt Hash D-H Grp A-Time L-Time IKEv
----- ------------ -------- ------- ------ ------ ----
up aes256 sha2_256 20 660 3600 2
vyatta@kou-vra002:~$ show vpn ipsec sa
Peer ID / IP Local ID / IP
------------ -------------
20.222.89.bbb 128.168.104.bbb
Tunnel Id State Bytes Out/In Encrypt Hash DH A-Time L-Time
------ ---------- ----- ------------- ------------ -------- -- ------ ------
vti 7 up 6.4K/13.2K aes256 sha2_256 20 18676 3600
BGP/OSPF/ルーティングの状態
1号機側
vyatta@kou-vra001:~$ show ip bgp neighbors
BGP neighbor is 169.254.21.2, remote AS 65515, local AS 65001, external link
BGP version 4, remote router ID 192.168.0.5
BGP state = Established, up for 00:28:23
Last read 00:28:23, hold time is 30, keepalive interval is 10 seconds
Configured hold time is 30, keepalive interval is 10 seconds
Neighbor capabilities:
Route refresh: advertised and received (new)
Address family IPv4 Unicast: advertised and received
Address family IPv6 Unicast: received
Received 200 messages, 0 notifications, 0 in queue
Sent 174 messages, 0 notifications, 0 in queue
Route refresh request: received 0, sent 0
Minimum time between advertisement runs is 30 seconds
For address family: IPv4 Unicast
BGP table version 8, neighbor version 8
Index 1, Offset 0, Mask 0x2
Graceful restart: received
Inbound soft reconfiguration allowed
Community attribute sent to this neighbor (both)
1 accepted prefixes
1 announced prefixes
Connections established 7; dropped 6
Local host: 169.254.21.1, Local port: 57254
Foreign host: 169.254.21.2, Foreign port: 179
Nexthop: 169.254.21.1
Nexthop global: ::
Nexthop local: fe80::200:5efe:80a8:68e9
BGP connection: non shared network
Last Reset: 00:29:57, due to BGP Notification sent
Notification Error Message: (Cease/Other Configuration Change.)
vyatta@kou-vra001:~$ show ip ospf neighbor
Total number of full neighbors: 1
OSPF process 0 routing-instance(default):
Neighbor ID Pri State Dead Time Address Interface
172.31.255.2 1 Full/ - 00:00:32 172.31.255.2 tun0
vyatta@kou-vra001:~$ show ip route bgp
IP Route Table for Routing-instance "default"
B *> 192.168.0.0/16 [20/0] via 169.254.21.2, vti0, 00:28:31
vyatta@kou-vra001:~$ show ip route ospf
IP Route Table for Routing-instance "default"
O E2 192.168.0.0/16 [110/1] via 172.31.255.2, tun0, 00:08:17
2号機側
vyatta@kou-vra002:~$ show ip bgp neighbors
BGP neighbor is 169.254.22.2, remote AS 65515, local AS 65001, external link
BGP version 4, remote router ID 192.168.0.4
BGP state = Established, up for 00:09:37
Last read 00:09:37, hold time is 30, keepalive interval is 10 seconds
Configured hold time is 30, keepalive interval is 10 seconds
Neighbor capabilities:
Route refresh: advertised and received (new)
Address family IPv4 Unicast: advertised and received
Address family IPv6 Unicast: received
Received 72 messages, 0 notifications, 0 in queue
Sent 61 messages, 0 notifications, 0 in queue
Route refresh request: received 0, sent 0
Minimum time between advertisement runs is 30 seconds
For address family: IPv4 Unicast
BGP table version 8, neighbor version 8
Index 1, Offset 0, Mask 0x2
Graceful restart: received
Inbound soft reconfiguration allowed
Community attribute sent to this neighbor (both)
1 accepted prefixes
1 announced prefixes
Connections established 5; dropped 4
Local host: 169.254.22.1, Local port: 57766
Foreign host: 169.254.22.2, Foreign port: 179
Nexthop: 169.254.22.1
Nexthop global: ::
Nexthop local: fe80::200:5efe:80a8:68fd
BGP connection: non shared network
Last Reset: 00:11:29, due to BGP Notification sent
Notification Error Message: (Cease/Other Configuration Change.)
vyatta@kou-vra002:~$ show ip ospf neighbor
Total number of full neighbors: 1
OSPF process 0 routing-instance(default):
Neighbor ID Pri State Dead Time Address Interface
172.31.255.1 1 Full/ - 00:00:36 172.31.255.1 tun0
vyatta@kou-vra002:~$ show ip route bgp
IP Route Table for Routing-instance "default"
B *> 192.168.0.0/16 [20/0] via 169.254.22.2, vti0, 00:09:43
vyatta@kou-vra002:~$ show ip route ospf
IP Route Table for Routing-instance "default"
O E2 192.168.0.0/16 [110/1] via 172.31.255.1, tun0, 00:29:58
エンドツーエンド疎通確認
ICMP
AzureサーバからIBMCloudのサーバに対してICMP
[azureuser@SV1 ~]$ ping 10.192.145.230
PING 10.192.145.230 (10.192.145.230) 56(84) bytes of data.
64 bytes from 10.192.145.230: icmp_seq=1 ttl=62 time=4.87 ms
64 bytes from 10.192.145.230: icmp_seq=2 ttl=62 time=5.08 ms
64 bytes from 10.192.145.230: icmp_seq=3 ttl=62 time=4.87 ms
64 bytes from 10.192.145.230: icmp_seq=4 ttl=62 time=4.82 ms
64 bytes from 10.192.145.230: icmp_seq=5 ttl=62 time=4.85 ms
IBMCloudサーバからAzureのサーバに対してポート80での疎通確認
[root@azure-sv ~]# hping3 -S -p 80 192.168.1.4
HPING 192.168.1.4 (eth0 192.168.1.4): S set, 40 headers + 0 data bytes
len=46 ip=192.168.1.4 ttl=62 DF id=0 sport=80 flags=SA seq=0 win=29200 rtt=4.9 ms
len=46 ip=192.168.1.4 ttl=62 DF id=0 sport=80 flags=SA seq=1 win=29200 rtt=5.8 ms
len=46 ip=192.168.1.4 ttl=62 DF id=0 sport=80 flags=SA seq=2 win=29200 rtt=6.7 ms
len=46 ip=192.168.1.4 ttl=62 DF id=0 sport=80 flags=SA seq=3 win=29200 rtt=5.6 ms
len=46 ip=192.168.1.4 ttl=62 DF id=0 sport=80 flags=SA seq=4 win=29200 rtt=5.5 ms
traceroute/tracepath
Azureサーバからの経路確認
Active/Activeで接続されているため、疎通確認の度に経路が変わっていることがわかる
[azureuser@SV1 ~]$ sudo traceroute -I 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
1 169.254.22.1 (169.254.22.1) 5.965 ms 5.972 ms 5.974 ms
2 * * *
3 10.192.145.230 (10.192.145.230) 7.746 ms 7.751 ms 7.750 ms
[azureuser@SV1 ~]$ traceroute 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
1 169.254.21.1 (169.254.21.1) 5.753 ms 5.705 ms 5.674 ms
2 * * *
3 * * 10.192.145.230 (10.192.145.230) 5.526 ms
[azureuser@SV1 ~]$ traceroute 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
1 169.254.22.1 (169.254.22.1) 4.512 ms 169.254.21.1 (169.254.21.1) 4.086 ms 169.254.22.1 (169.254.22.1) 4.435 ms
2 * * *
3 10.192.145.230 (10.192.145.230) 5.076 ms 5.064 ms 5.031 ms
[azureuser@SV1 ~]$ traceroute 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
1 169.254.21.1 (169.254.21.1) 3.886 ms 169.254.22.1 (169.254.22.1) 3.789 ms 3.754 ms
2 * * *
3 10.192.145.230 (10.192.145.230) 5.207 ms 5.185 ms *
IBMCloudサーバからの経路確認
[root@ibmcloud-sv ~]# tracepath 192.168.1.4
1?: [LOCALHOST] pmtu 1500
1: 10.192.145.193 0.644ms
1: 10.192.145.193 0.311ms
2: 10.192.94.146 0.727ms
3: 10.192.94.146 0.860ms pmtu 1422
3: 192.168.1.4 4.790ms reached
Resume: pmtu 1422 hops 3 back 3
[root@ibmcloud-sv ~]# tracepath 192.168.1.4
1?: [LOCALHOST] pmtu 1500
1: 10.192.145.193 0.656ms
1: 10.192.145.193 0.395ms
2: 10.192.94.175 0.872ms
3: 10.192.94.175 0.778ms pmtu 1422
3: 192.168.1.4 5.004ms reached
Resume: pmtu 1422 hops 3 back 3
HTTP
AzureからIBMCloudのHTTP(80)サーバへの疎通確認
[azureuser@SV1 html]$ while true; do curl -w "gettime:%{time_total} " http://10.192.145.230; sleep 1; done
gettime:0.010 ibmcloud sv1
gettime:0.008 ibmcloud sv1
gettime:0.008 ibmcloud sv1
gettime:0.007 ibmcloud sv1
gettime:0.008 ibmcloud sv1
IBMCloudからAzureのHTTP(80)サーバへの疎通確認
[root@ibmcloud-sv ~]# while true; do curl -w "gettime:%{time_total} " http://192.168.1.4; sleep 1; done
gettime:0.010 azure sv1
gettime:0.008 azure sv1
gettime:0.009 azure sv1
gettime:0.008 azure sv1
gettime:0.009 azure sv1
障害時の通信確認
VRA#1側のPublicインタフェース障害
ポータルのページからVRA#1のPublicインタフェースをdisableにする
Azure側の状態確認
VRA#1側の状態確認
dp0bon1(publicインタフェース)とvti0がダウンしていることがわかる
vyatta@kou-vra001:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Speed/Duplex Description
--------- ---------- --- ------------ -----------
dp0bond0 10.192.94.146/26 u/u 2g/full
dp0bond1 128.168.104.aaa/27 u/D -/-
dp0s0 - u/u a-1g/a-full
dp0s1 - u/D auto/auto
dp0s2 - u/u a-1g/a-full
dp0s3 - u/D auto/auto
erspan0 - A/D -/-
tun0 172.31.255.1/30 u/u -/-
vti0 169.254.21.1/30 A/D -/- Azure
ipsecも停止している
vyatta@kou-vra001:~$ show vpn ike sa
Peer ID / IP Local ID / IP
------------ -------------
20.222.88.aaa 128.168.104.aaa
State Encrypt Hash D-H Grp A-Time L-Time IKEv
----- ------------ -------- ------- ------ ------ ----
init n/a n/a n/a n/a 3600 2
vyatta@kou-vra001:~$ show vpn ipsec sa
vyatta@kou-vra001:~$
bgpも停止している
vyatta@kou-vra001:~$ show ip bgp neighbors
BGP neighbor is 169.254.21.2, remote AS 65515, local AS 65001, external link
BGP version 4, remote router ID 0.0.0.0
BGP state = Active
Last read 00:01:06, hold time is 30, keepalive interval is 10 seconds
Configured hold time is 30, keepalive interval is 10 seconds
Received 0 messages, 0 notifications, 0 in queue
Sent 0 messages, 0 notifications, 0 in queue
Route refresh request: received 0, sent 0
Minimum time between advertisement runs is 30 seconds
For address family: IPv4 Unicast
BGP table version 11, neighbor version 0
Index 1, Offset 0, Mask 0x2
Inbound soft reconfiguration allowed
Community attribute sent to this neighbor (both)
0 accepted prefixes
0 announced prefixes
Connections established 10; dropped 10
Next connect timer due in 34 seconds
Last Reset: 00:05:19, due to BGP Notification sent
Notification Error Message: (Hold Timer Expired/Unspecified Error Subcode)
Azure(192.168.0.0/16)へのルートはOSPFでVRA#2からルーティングをもらっている
vyatta@kou-vra001:~$ show ip route ospf
IP Route Table for Routing-instance "default"
O E2 *> 192.168.0.0/16 [110/1] via 172.31.255.2, tun0, 00:23:10
traceroute/tracepath
Azure側からはVRA#2側のみを通って通信していることがわかる
[azureuser@SV1 html]$ traceroute 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
1 169.254.22.1 (169.254.22.1) 4.346 ms 4.285 ms 4.281 ms
2 * * *
3 10.192.145.230 (10.192.145.230) 5.905 ms * *
[azureuser@SV1 html]$ traceroute 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
1 169.254.22.1 (169.254.22.1) 3.819 ms 3.776 ms 3.732 ms
2 * * *
3 10.192.145.230 (10.192.145.230) 6.349 ms * 6.303 ms
IBMCloud側では、VRA#1~VRA#2間のOSPF経路を経由して、VRA#2側から通信している事がわかる
[root@ibmcloud-sv ~]# tracepath 192.168.1.4
1?: [LOCALHOST] pmtu 1500
1: 10.192.145.193 0.947ms
1: 10.192.145.193 0.635ms
2: 10.192.94.175 0.820ms
3: 10.192.94.175 0.463ms pmtu 1422
3: 192.168.1.4 4.576ms reached
Resume: pmtu 1422 hops 3 back 3
[root@ibmcloud-sv ~]# tracepath 192.168.1.4
1?: [LOCALHOST] pmtu 1500
1: 10.192.145.193 0.538ms
1: 10.192.145.193 0.360ms
2: 10.192.94.146 0.768ms
3: 10.192.94.175 0.703ms pmtu 1422
3: 192.168.1.4 4.034ms reached
Resume: pmtu 1422 hops 3 back 3
HTTP
azure側からibmcloud側のサーバに対して、http getに30秒
[azureuser@SV1 html]$ while true; do curl -w "gettime:%{time_total} " http://10.192.145.230; sleep 1; done
gettime:0.009 ibmcloud sv1
gettime:0.010 ibmcloud sv1
gettime:31.080 ibmcloud sv1
gettime:0.009 ibmcloud sv1
gettime:0.009 ibmcloud sv1
ibmcloud側からazure側のサーバに対して、http getに23秒
[root@ibmcloud-sv ~]# while true; do curl -w "gettime:%{time_total} " http://192.168.1.4; sleep 1; done
gettime:0.009 azure sv1
gettime:0.009 azure sv1
gettime:23.617 azure sv1
gettime:0.010 azure sv1
gettime:0.010 azure sv1
VRA#2側のPublicインタフェース障害
ポータルのページからVRA#2のPublicインタフェースをdisableにする
Azure側の状態確認
VRA#2側の状態確認
dp0bon1(publicインタフェース)とvti0がダウンしていることがわかる
vyatta@kou-vra002:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Speed/Duplex Description
--------- ---------- --- ------------ -----------
dp0bond0 10.192.94.175/26 u/u 2g/full
dp0bond1 128.168.104.bbb/27 u/D -/-
dp0s0 - u/u a-1g/a-full
dp0s1 - u/D auto/auto
dp0s2 - u/u a-1g/a-full
dp0s3 - u/D auto/auto
erspan0 - A/D -/-
tun0 172.31.255.2/30 u/u -/-
vti0 169.254.22.1/30 A/D -/- Azure
ipsecも停止している
vyatta@kou-vra002:~$ show vpn ike sa
Peer ID / IP Local ID / IP
------------ -------------
20.222.89.bbb 128.168.104.bbb
State Encrypt Hash D-H Grp A-Time L-Time IKEv
----- ------------ -------- ------- ------ ------ ----
init n/a n/a n/a n/a 3600 2
vyatta@kou-vra002:~$ show vpn ipse sa
vyatta@kou-vra002:~$
bgpも停止している
vyatta@kou-vra002:~$ show ip bgp neighbors
BGP neighbor is 169.254.22.2, remote AS 65515, local AS 65001, external link
BGP version 4, remote router ID 0.0.0.0
BGP state = Active
Last read 00:00:10, hold time is 30, keepalive interval is 10 seconds
Configured hold time is 30, keepalive interval is 10 seconds
Received 0 messages, 0 notifications, 0 in queue
Sent 0 messages, 0 notifications, 0 in queue
Route refresh request: received 0, sent 0
Minimum time between advertisement runs is 30 seconds
For address family: IPv4 Unicast
BGP table version 9, neighbor version 0
Index 1, Offset 0, Mask 0x2
Inbound soft reconfiguration allowed
Community attribute sent to this neighbor (both)
0 accepted prefixes
0 announced prefixes
Connections established 6; dropped 6
Next connect timer due in 87 seconds
Last Reset: 00:04:07, due to BGP Notification sent
Notification Error Message: (Hold Timer Expired/Unspecified Error Subcode)
Azure(192.168.0.0/16)へのルートはOSPFでVRA#2からルーティングをもらっている
vyatta@kou-vra002:~$ show ip route ospf
IP Route Table for Routing-instance "default"
O E2 *> 192.168.0.0/16 [110/1] via 172.31.255.1, tun0, 00:09:22
traceroute/tracepath
Azure側からはVRA#1側のみを通って通信していることがわかる
[azureuser@SV1 html]$ traceroute 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
1 169.254.21.1 (169.254.21.1) 6.663 ms 6.610 ms 6.570 ms
2 * * *
3 10.192.145.230 (10.192.145.230) 7.672 ms 7.663 ms *
[azureuser@SV1 html]$ traceroute 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
1 169.254.21.1 (169.254.21.1) 4.283 ms 4.223 ms 4.191 ms
2 * * *
3 10.192.145.230 (10.192.145.230) 4.533 ms 4.881 ms *
IBMCloud側では、VRA#1~VRA#2間のOSPF経路を経由して、VRA#1側から通信している事がわかる
[root@ibmcloud-sv ~]# tracepath 192.168.1.4
1?: [LOCALHOST] pmtu 1500
1: 10.192.145.193 0.587ms
1: 10.192.145.193 0.376ms
2: 10.192.94.175 0.689ms
3: 10.192.94.146 0.876ms pmtu 1422
3: 192.168.1.4 4.822ms reached
Resume: pmtu 1422 hops 3 back 3
[root@ibmcloud-sv ~]# tracepath 192.168.1.4
1?: [LOCALHOST] pmtu 1500
1: 10.192.145.193 0.938ms
1: 10.192.145.193 0.611ms
2: 10.192.94.146 1.172ms
3: 10.192.94.146 0.552ms pmtu 1422
3: 192.168.1.4 5.104ms reached
Resume: pmtu 1422 hops 3 back 3
HTTP
azure側からibmcloud側のサーバに対して、http getに30秒
[azureuser@SV1 html]$ while true; do curl -w "gettime:%{time_total} " http://10.192.145.230; sleep 1; done
gettime:0.010 ibmcloud sv1
gettime:0.009 ibmcloud sv1
gettime:31.062 ibmcloud sv1
gettime:0.010 ibmcloud sv1
gettime:0.010 ibmcloud sv1
ibmcloud側からazure側のサーバに対して、http getに30秒
[root@ibmcloud-sv ~]# while true; do curl -w "gettime:%{time_total} " http://192.168.1.4; sleep 1; done
gettime:0.010 azure sv1
gettime:0.010 azure sv1
gettime:31.072 azure sv1
gettime:0.010 azure sv1
gettime:0.011 azure sv1
VRA#1/VRA#2機器障害
確認結果はほぼ同じなので割愛する
VRA#1障害
Azure to IBMCloud
IBMCloud to Azure
VRA#2障害
Azure to IBMCloud
IBMCloud to Azure
