#■目的
Interop Tokyo 2019にいってきました。
NECさんのUNIVERGE QX-W2120がShowNetの構成要素に組まれており、安心の冗長構成がされていました。

ということで、UNIVERGE IXルーターとOCIをBGPでIPsec接続して冗長化した２つのIPSec TunnelをAS-Path PrependでTunnel１を優先するよう経路制御を行い、可用性検証してみます。
BGPの設定と確認方法はIXプロフェッショナル @hongmnh さんに協力いただきました。

#■ X2015 構成

#■手順
主な手順は、こちらを参考としてStatic Routesの設定部分をBGP設定に変更して構成します。
Oracle Cloud:NEC UNIVERGE IXルーターでOCIへIPsec VPN接続してみてみた

##●Oracle Cloud IPSec作成
OCI Webコンソールから、[Network] > [IPSec Connection]から[Create IPsec COnnect]をクリックし、以下の様に設定

・IPSec情報

・IPSec Tunnel01情報

・IPSec Tunnel02情報

設定完了したら、[Create IPSec COnnect]をクリックし作成

作成すると、「Oracle VPN IP Address」に各TunnnelにPublic IPが付与されるので、このIPをルーター設定時に使用します。

##●BGP設定

!
ip prefix-list as-path-in 10 permit any
ip prefix-list as-path-out 10 permit any
!
route-map bgp permit 65000
!
!AS-Path制御設定
route-map rmap-in permit 10
  match ip address prefix-list as-path-in
  set as-path prepend 31898 31898 
!
route-map rmap-out permit 10
  match ip address prefix-list as-path-out
  set as-path prepend 65000 65000 
!
! Keepaliveメッセージを10秒、holdtimeを30秒として設定
router bgp 65000
  neighbor 10.254.200.202 remote-as 31898
  neighbor 10.254.200.202 timers 10 30
  neighbor 10.254.200.206 remote-as 31898
  neighbor 10.254.200.206 timers 10 30
  address-family ipv4 unicast
    neighbor 10.254.200.206 route-map rmap-in in
    neighbor 10.254.200.206 route-map rmap-out out
    network 192.168.100.0/24
!
!Tunnel interface にBGP用IPを設定
interface Tunnel0.0
  tunnel mode ipsec
  ip address 10.254.200.201/30
  ip tcp adjust-mss auto
  ipsec policy tunnel ipsec-map1 out
  no shutdown
!
interface Tunnel1.0
  tunnel mode ipsec
  ip address 10.254.200.205/30
  ip tcp adjust-mss auto
  ipsec policy tunnel ipsec-map2 out
  no shutdown

##●config確認

Router(config)# show config
    Using 3714 out of 524288 bytes

    ! NEC Portable Internetwork Core Operating System Software
    ! IX Series IX2105 (magellan-sec) Software, Version 9.6.12A, MAINTENANCE RELEASE SOFTWARE
    ! Compiled Oct 05-Thu-2017 19:27:09 JST #2
    ! Last updated Jun 14-Fri-2019 00:58:40 JST
    !
    !% You must restart the router for this configuration to take effect.
    !
    timezone +09 00
    !
    username user password hash 123ABAD44 administrator
    !
    !
    ip ufs-cache enable
    ip multipath per-flow
    ip route default GigaEthernet0.1
    ip prefix-list as-path-in 10 permit any
    ip prefix-list as-path-out 10 permit any
    ip access-list sec-list permit ip src any dest any
    ip access-list web-http-acl permit ip src any dest 192.168.100.240/32
    ip access-list web_console permit ip src any dest 192.168.100.240/32
    !
    !
    !
    ike proposal ike-prop encryption aes-256 hash sha2-256 group 1536-bit
    !
    ike policy ike-pol1 peer 100.100.100.101 key IPSecIX200Password ike-prop
    !
    ike policy ike-pol2 peer 100.100.100.102 key IPSecIX200Password ike-prop
    !
    ipsec autokey-proposal ipsec-prop esp-aes-256 esp-sha lifetime time 3600
    !
    ipsec autokey-map ipsec-map1 sec-list peer 100.100.100.101 ipsec-prop pfs 1536-bit
    !
    ipsec autokey-map ipsec-map2 sec-list peer 100.100.100.102 ipsec-prop pfs 1536-bit
    !
    !
    !
    !
    !
    !
    !
    proxy-dns ip enable
    proxy-dns interface GigaEthernet0.1 priority 254
    !
    telnet-server ip enable
    !
    ssh-server ip enable
    !
    http-server username user
    http-server ip access-list web-http-acl
    http-server ip enable
    !
    !
    !
    watch-group watch_tunnel0 10
    event 20 ip unreach-host 100.100.100.101 Tunnel0.0 source GigaEthernet1.0
    action 20 ipsec clear-sa Tunnel0.0
    !
    network-monitor watch_tunnel0 enable
    !
    watch-group watch_tunnel1 10
    event 20 ip unreach-host 100.100.100.102 Tunnel1.0 source GigaEthernet1.0
    action 20 ipsec clear-sa Tunnel1.0
    !
    network-monitor watch_tunnel1 enable
    !
    !
    route-map bgp permit 65000
    !
    route-map rmap-in permit 10
    match ip address prefix-list as-path-in
    set as-path prepend 31898 31898
    !
    route-map rmap-out permit 10
    match ip address prefix-list as-path-out
    set as-path prepend 65000 65000
    !
    ppp profile web-ppp-gigaethernet0.1
    authentication myname user@ntt.ne.jp
    authentication password user@ntt.ne.jp a12cpe
    !
    ip dhcp profile web-dhcp-gigaethernet1.0
    dns-server 192.168.100.240
    !
    router bgp 65000
    neighbor 10.254.200.202 remote-as 31898
    neighbor 10.254.200.202 timers 10 30
    neighbor 10.254.200.206 remote-as 31898
    neighbor 10.254.200.206 timers 10 30
    address-family ipv4 unicast
        neighbor 10.254.200.206 route-map rmap-in in
        neighbor 10.254.200.206 route-map rmap-out out
        network 192.168.100.0/24
    !
    device GigaEthernet0
    !
    device GigaEthernet1
    !
    interface GigaEthernet0.0
    no ip address
    shutdown
    !
    interface GigaEthernet1.0
    description LAN1
    ip address 192.168.100.240/24
    ip dhcp binding web_gigaethernet1.0
    no shutdown
    !
    interface GigaEthernet0.1
    description WAN1
    encapsulation pppoe
    auto-connect
    ppp binding web-ppp-gigaethernet0.1
    ip address ipcp
    ip tcp adjust-mss auto
    ip napt enable
    ip napt hairpinning
    no shutdown
    !
    interface Loopback0.0
    no ip address
    !
    interface Null0.0
    no ip address
    !
    interface Tunnel0.0
    tunnel mode ipsec
    ip address 10.254.200.201/30
    ip tcp adjust-mss auto
    ipsec policy tunnel ipsec-map1 out
    no shutdown
    !
    interface Tunnel1.0
    tunnel mode ipsec
    ip address 10.254.200.205/30
    ip tcp adjust-mss auto
    ipsec policy tunnel ipsec-map2 out
    no shutdown
    !
    web-console system information
    o lan1 GigaEthernet1.0
    o wizard easy-inet-only-pppoe
    o wan1 GigaEthernet0.1

#■BGP確認
##●BGP確立確認
すべてのStatusがESTABLISHEDであることを確認

Router(config)# show ip bgp summary
    BGP router ID 192.168.100.240, local AS number 65000
    1 BGP AS-PATH entries

    Neighbor         V    AS    MsgRcvd MsgSent Up/DownTime   State
    10.254.200.202 4    31898 31      23      0:00:36       ESTABLISHED
    10.254.200.206 4    31898 29      23      0:00:35       ESTABLISHED

    Total number of neighbors 2

##●ルート情報伝搬確認

Router(config)# show ip route
    IP Routing Table - 8 entries, 3 hidden, 2037 frees
    Entries: 4 Connected, 1 Static, 0 RIP, 0 OSPF, 3 BGP
    Codes: C - Connected, S - Static, R - RIP, O - OSPF, IA - OSPF inter area
        N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
        E1 - OSPF external type 1, E2 - OSPF external type 2, B - BGP
        * - Candidate default, s - Summary
    Timers: Age
    S*   0.0.0.0/0 [1/1] is directly connected, GigaEthernet0.1
        10.0.0.0/8 is subnetted, 3 subnets
    B      10.0.0.0/24 [20/0] via 10.254.200.202, Tunnel0.0
        10.254.0.0/16 is subnetted, 2 subnets
    C      10.254.200.200/30 [0/1] is directly connected, Tunnel0.0
    C      10.254.200.204/30 [0/1] is directly connected, Tunnel1.0
    C    192.168.100.0/24 [0/1] is directly connected, GigaEthernet1.0
        200.200.200.0/24 is subnetted, 1 subnets
    C      200.200.200.201/32 [0/1] is directly connected, GigaEthernet0.1

・BGP AS Path設定確認
バックアップ・パスに：AS番号が３つ付与されていることを確認
・プライマリ・パス(10.254.200.202)　：31898 i
・バップアツプ・パス(10.254.200.206)：31898 31898 31898 i

Router(config)# show ip bgp
    BGP table version is 25, local router ID is 192.168.100.240
    Local AS number 65000
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *  10.0.0.0/24        10.254.200.206                        31898 31898 31898 i
    *>                    10.254.200.202                        31898 i
    *> 192.168.100.0/24   0.0.0.0                  1             i

    Total number of prefixes 7

Router(config)# show ip bgp paths
    Address    RefCount Next Hop               MED      LocPrf  Path
    0x0302       12 10.254.200.202          0         100  31898 i
    0x0313        3 10.254.200.206          0         100  31898 31898 31898 i
    0x0314        2 0.0.0.0                  1*        100  i
    0x0311        1 10.254.200.201          1*        100  i
    0x0301        1 10.254.200.205          1*        100  65000 65000 i

##●プライマリ・パス/バップアツプ・パス確認
・バップアツプ・パス

Router(config)# show ip bgp neighbors 10.254.200.202 advertised-routes
    BGP table version is 25, local router ID is 192.168.100.240
    Local AS number 65000
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *> 192.168.100.0/24   10.254.200.201          1             i

    Total number of prefixes 1

Router(config)# show ip bgp neighbors 10.254.200.202 received-routes
    BGP table version is 25, local router ID is 192.168.100.240
    Local AS number 65000
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *> 10.0.0.0/24        10.254.200.202                        31898 i

    Total number of prefixes 1

・プライマリ・パス

Router(config)# show ip bgp neighbors 10.254.200.206 received-routes
    BGP table version is 25, local router ID is 192.168.100.240
    Local AS number 65000
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *  10.0.0.0/24        10.254.200.206                        31898 31898 31898 i

    Total number of prefixes 1

Router(config)# show ip bgp neighbors 10.254.200.206 advertised-routes
    BGP table version is 25, local router ID is 192.168.100.240
    Local AS number 65000
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *> 192.168.100.0/24   10.254.200.205          1             65000 65000 i

    Total number of prefixes 1

#■Oracle Cloud IPSec/BGPステータス確認
ルータと接続が確立されると「IPSec Status」と「BGP Status」がUPします

#■プライマリ・パス障害検証
プライマリ・パス Interfaceを停止すると非疎通になるが、holdtime30秒以内に疎通が復旧することを確認し、
切り戻しで停止したプライマリ・パス Tunnnel0.0 Interfaceを起動し疎通に問題が無いことを確認します。

##●事前確認

・ping監視実行
On-Premises側からOCI側のInstanseへpingを行い、障害試験実施によりどのタイミングで疎通できなくなるか確認

onp-inst01:~ user$ ping 10.0.0.2
    PING 10.0.0.2 (10.0.0.2): 56 data bytes
    64 bytes from 10.0.0.2: icmp_seq=0 ttl=62 time=6.210 ms
    64 bytes from 10.0.0.2: icmp_seq=1 ttl=62 time=6.067 ms
    64 bytes from 10.0.0.2: icmp_seq=2 ttl=62 time=14.000 ms

・BGP確立確認
すべてのStatusがESTABLISHEDであることを確認

Router(config)# show ip bgp summary
    BGP router ID 192.168.100.240, local AS number 65000
    4 BGP AS-PATH entries

    Neighbor         V    AS    MsgRcvd MsgSent Up/DownTime   State
    10.254.200.202 4    31898 18268   16511   1d21h48m28s   ESTABLISHED
    10.254.200.206 4    31898 18284   16520   1d21h50m27s   ESTABLISHED

    Total number of neighbors 2

・BGPパス確認
すべてのPathがあることを確認

Router(config)# show ip bgp
    BGP table version is 25, local router ID is 192.168.100.240
    Local AS number 65000
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *  10.0.0.0/24        10.254.200.206                        31898 31898 31898 i
    *>                    10.254.200.202                        31898 i
    *> 192.168.100.0/24   0.0.0.0                  1             i

    Total number of prefixes 5

##●プライマリ・パス Interface停止実行

Router(config)# interface Tunnel0.0
Router(config-Tunnel1.0)# shutdown
Router(config)# show interfaces Tunnel0.0
    Interface Tunnel0.0 is administratively down
    ・・・

・Ping:応答確認
プライマリ・パス Interface停止により疎通しなくなることを確認

onp-inst01:~ user$ ping 10.0.0.2
    ・・・
    64 bytes from 10.0.0.2: icmp_seq=0 ttl=62 time=6.210 ms
    64 bytes from 10.0.0.2: icmp_seq=1 ttl=62 time=6.067 ms
    64 bytes from 10.0.0.2: icmp_seq=2 ttl=62 time=14.000 ms
    64 bytes from 10.0.0.2: icmp_seq=3 ttl=62 time=4.979 ms
    Request timeout for icmp_seq 4
    Request timeout for icmp_seq 5
    Request timeout for icmp_seq 6
    Request timeout for icmp_seq 7
    Request timeout for icmp_seq 8

・プライマリ・パス BGP Status IDLE確認

Router(config)# show ip bgp summary
    BGP router ID 192.168.100.240, local AS number 65000
    3 BGP AS-PATH entries

    Neighbor         V    AS    MsgRcvd MsgSent Up/DownTime   State
    10.254.200.202 4    31898 721     641     0:02:28       ESTABLISHED
    10.254.200.206 4    31898 415     378     0:00:24       IDLE

・BGP Path確認
バックアップ・パスのみがあることを確認

Router(config)# show ip bgp
    BGP table version is 94, local router ID is 192.168.100.240
    Local AS number 65000
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *> 10.0.0.0/24        10.254.200.202                        31898 31898 31898 i
    10.254.200.0/24   0.0.0.0                  0             i
    *> 192.168.100.0/24   0.0.0.0                  1             i

    Total number of prefixes 3
    Router(config)#

・Ping　30秒で復旧確認
holdtime30秒以内に疎通が復旧することを確認

onp-inst01:~ user$ ping 10.0.0.2
    PING 10.0.0.2 (10.0.0.2): 56 data bytes
    64 bytes from 10.0.0.2: icmp_seq=0 ttl=62 time=6.210 ms
    64 bytes from 10.0.0.2: icmp_seq=1 ttl=62 time=6.067 ms
    64 bytes from 10.0.0.2: icmp_seq=2 ttl=62 time=14.000 ms
    64 bytes from 10.0.0.2: icmp_seq=3 ttl=62 time=4.979 ms
    Request timeout for icmp_seq 4
    Request timeout for icmp_seq 5
    Request timeout for icmp_seq 6
    Request timeout for icmp_seq 7
    Request timeout for icmp_seq 8
    Request timeout for icmp_seq 9
    Request timeout for icmp_seq 10
    Request timeout for icmp_seq 11
    Request timeout for icmp_seq 12
    Request timeout for icmp_seq 13
    Request timeout for icmp_seq 14
    Request timeout for icmp_seq 15
    Request timeout for icmp_seq 16
    Request timeout for icmp_seq 17
    Request timeout for icmp_seq 18
    Request timeout for icmp_seq 19
    Request timeout for icmp_seq 20
    Request timeout for icmp_seq 21
    Request timeout for icmp_seq 22
    Request timeout for icmp_seq 23
    Request timeout for icmp_seq 24
    Request timeout for icmp_seq 25
    Request timeout for icmp_seq 26
    Request timeout for icmp_seq 27
    Request timeout for icmp_seq 28
    Request timeout for icmp_seq 29
    64 bytes from 10.0.0.2: icmp_seq=30 ttl=62 time=5.853 ms
    64 bytes from 10.0.0.2: icmp_seq=31 ttl=62 time=5.505 ms
    64 bytes from 10.0.0.2: icmp_seq=32 ttl=62 time=5.388 ms
    64 bytes from 10.0.0.2: icmp_seq=33 ttl=62 time=38.859 ms
    64 bytes from 10.0.0.2: icmp_seq=34 ttl=62 time=29.210 ms
    64 bytes from 10.0.0.2: icmp_seq=35 ttl=62 time=8.350 ms

#■切り戻し
##●プライマリ・パス Interface起動

Router(config)# interface Tunnel0.0
Router(config-Tunnel0.0)# no shutdown

・プライマリ・パス Interface起動確認

Router(config-Tunnel0.0)# show interfaces stats
    Interface Tunnel0.0 is up
    ・・・

・BGP Status ESTABLESHED確認
プライマリ・パスがIDLEからESTABLISHEDに復旧したことを確認

Router(config)# show ip bgp summary
    BGP router ID 192.168.100.240, local AS number 65000
    4 BGP AS-PATH entries

    Neighbor         V    AS    MsgRcvd MsgSent Up/DownTime   State
    10.254.200.202 4    31898 806     719     0:15:20       ESTABLISHED
    10.254.200.206 4    31898 424     386     0:00:57       ESTABLISHED

    Total number of neighbors 2

・BGP Path確認
プライマリ・パスが復旧していることを確認

Router(config)# show ip bgp
    BGP table version is 133, local router ID is 192.168.100.240
    Local AS number 65000
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *> 10.0.0.0/24        10.254.200.206                        31898 i
    *                     10.254.200.202                        31898 31898 31898 i
    10.254.200.0/24   0.0.0.0                  0             i
    *> 192.168.100.0/24   0.0.0.0                  1             i

    Total number of prefixes 6

・ping 異常無し確認
切り戻しによる非疎通が無いことを確認

    ・・・
    Request timeout for icmp_seq 27
    Request timeout for icmp_seq 28
    Request timeout for icmp_seq 29
    64 bytes from 10.0.0.2: icmp_seq=30 ttl=62 time=5.853 ms
    64 bytes from 10.0.0.2: icmp_seq=31 ttl=62 time=5.505 ms
    64 bytes from 10.0.0.2: icmp_seq=32 ttl=62 time=5.388 ms
    64 bytes from 10.0.0.2: icmp_seq=33 ttl=62 time=38.859 ms
    64 bytes from 10.0.0.2: icmp_seq=34 ttl=62 time=29.210 ms
    64 bytes from 10.0.0.2: icmp_seq=35 ttl=62 time=8.350 ms
    ・・・
    64 bytes from 10.0.0.2: icmp_seq=300 ttl=62 time=11.249 ms
    64 bytes from 10.0.0.2: icmp_seq=301 ttl=62 time=43.254 ms
    64 bytes from 10.0.0.2: icmp_seq=302 ttl=62 time=12.917 ms
    64 bytes from 10.0.0.2: icmp_seq=303 ttl=62 time=12.908 ms
    64 bytes from 10.0.0.2: icmp_seq=304 ttl=62 time=25.170 ms
    64 bytes from 10.0.0.2: icmp_seq=305 ttl=62 time=49.422 ms
    64 bytes from 10.0.0.2: icmp_seq=306 ttl=62 time=13.343 ms
    64 bytes from 10.0.0.2: icmp_seq=307 ttl=62 time=38.864 ms
    64 bytes from 10.0.0.2: icmp_seq=308 ttl=62 time=25.237 ms

#■BGP確立連続稼働確認
1日以上連続稼働させてBGPの設定に問題がないことを確認

Router(config)# show ip bgp summary
    BGP router ID 192.168.100.240, local AS number 65000
    4 BGP AS-PATH entries

    Neighbor         V    AS    MsgRcvd MsgSent Up/DownTime   State
    10.254.200.202 4    31898 18268   16511   1d21h48m28s   ESTABLISHED
    10.254.200.206 4    31898 18284   16520   1d21h50m27s   ESTABLISHED

##■参考
・マニュアル：https://jpn.nec.com/univerge/ix/Manual/index.html#crm