はじめに
MXシリーズでMPLS/VPLSを利用したサービスを提供しているのですが、ある日新規に設定したVPLSがUpしない事象に遭遇しました。
原因や対策についての結果をまとめました。
結論
結論からお話しすると、MXシリーズのルータではLSPのnext-hop ECMPがデフォルト16で設定されています。
そのため、同一宛先に設定されたLSPが16を超過すると、超過したLSPはroute tableに乗らないため通信できないという事象が発生します。
検証環境
下記、Juniper vLabs環境で実施しております。
vMX:21.1R3.11
vMX1 - vMX2 : LSP65本
vMX1 - vMX3 : LSP65本
同一宛先16本しか通信できないことの確認
vMX1
show route table inet.3
vMX2(10.100.100.2)向け：65本中16本が確認できる。残り49本は確認できない。
vMX3(10.100.100.3)向け：65本中16本が確認できる。残り49本は確認できない。
jcluser@vMX1> show route table inet.3
inet.3: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.100.100.2/32 *[RSVP/7/1] 01:18:53, metric 1
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-00
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-01
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-02
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-03
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-04
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-05
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-06
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-07
> to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-08
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-09
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-10
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-11
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-12
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-13
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-14
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-15
10.100.100.3/32 *[RSVP/7/1] 00:40:37, metric 1
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-00
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-01
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-02
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-03
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-04
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-05
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-06
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-07
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-08
> to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-09
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-10
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-11
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-12
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-13
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-14
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-15
jcluser@vMX1>
show vpls connections
各LSPに紐づいているVPLSの状態
vMX2(10.100.100.2)向け：65本中16本がUp、残り49本がVC-Dn。
vMX3(10.100.100.3)向け：65本中16本がUp、残り49本がVC-Dn。
jcluser@vMX1> show vpls connections
Layer-2 VPN connections:
Legend for connection status (St)
EI -- encapsulation invalid NC -- interface encapsulation not CCC/TCC/VPLS
EM -- encapsulation mismatch WE -- interface and instance encaps not same
VC-Dn -- Virtual circuit down NP -- interface hardware not present
CM -- control-word mismatch -> -- only outbound connection is up
CN -- circuit not provisioned <- -- only inbound connection is up
OR -- out of range Up -- operational
OL -- no outgoing label Dn -- down
LD -- local site signaled down CF -- call admission control failure
RD -- remote site signaled down SC -- local and remote site ID collision
LN -- local site not designated LM -- local site ID not minimum designated
RN -- remote site not designated RM -- remote site ID not minimum designated
XX -- unknown connection status IL -- no incoming label
MM -- MTU mismatch MI -- Mesh-Group ID not available
BK -- Backup connection ST -- Standby connection
PF -- Profile parse failure PB -- Profile busy
RS -- remote site standby SN -- Static Neighbor
LB -- Local site not best-site RB -- Remote site not best-site
VM -- VLAN ID mismatch HS -- Hot-standby Connection
Legend for interface status
Up -- operational
Dn -- down
Instance: rt00000
Edge protection: Not-Primary
Local site: rt00000 (1)
connection-site Type St Time last up # Up trans
2 rmt Up Dec 25 04:34:39 2023 1
Remote PE: 10.100.100.2, Negotiated control-word: No
Incoming label: 262154, Outgoing label: 262153
Local interface: lsi.1048576, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt00000 local site 1 remote site 2
Flow Label Transmit: No, Flow Label Receive: No
Instance: rt00001
Edge protection: Not-Primary
Local site: rt00001 (1)
connection-site Type St Time last up # Up trans
2 rmt Up Dec 25 05:54:22 2023 1
Remote PE: 10.100.100.2, Negotiated control-word: No
Incoming label: 262162, Outgoing label: 262161
Local interface: lsi.1048642, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt00001 local site 1 remote site 2
Flow Label Transmit: No, Flow Label Receive: No
~
Instance: rt00015
Edge protection: Not-Primary
Local site: rt00015 (1)
connection-site Type St Time last up # Up trans
2 rmt Up Dec 25 05:54:23 2023 1
Remote PE: 10.100.100.2, Negotiated control-word: No
Incoming label: 262274, Outgoing label: 262273
Local interface: lsi.1048692, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt00015 local site 1 remote site 2
Flow Label Transmit: No, Flow Label Receive: No
Instance: rt00016
Edge protection: Not-Primary
Local site: rt00016 (1)
connection-site Type St Time last up # Up trans
2 rmt VC-Dn ----- 0
Remote PE: 10.100.100.2, Negotiated control-word: No
Incoming label: 262282, Outgoing label: 262281
Local interface: lsi.1048652, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt00016 local site 1 remote site 2
Flow Label Transmit: No, Flow Label Receive: No
~
Instance: rt00064
Edge protection: Not-Primary
Local site: rt00064 (1)
connection-site Type St Time last up # Up trans
2 rmt VC-Dn ----- 0
Remote PE: 10.100.100.2, Negotiated control-word: No
Incoming label: 327690, Outgoing label: 327689
Local interface: lsi.1048705, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt00064 local site 1 remote site 2
Flow Label Transmit: No, Flow Label Receive: No
Instance: rt10000
Edge protection: Not-Primary
Local site: rt10000 (1)
connection-site Type St Time last up # Up trans
3 rmt Up Dec 25 05:43:09 2023 1
Remote PE: 10.100.100.3, Negotiated control-word: No
Incoming label: 327699, Outgoing label: 262145
Local interface: lsi.1048577, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt10000 local site 1 remote site 3
Flow Label Transmit: No, Flow Label Receive: No
Instance: rt10001
Edge protection: Not-Primary
Local site: rt10001 (1)
connection-site Type St Time last up # Up trans
3 rmt Up Dec 25 05:43:09 2023 1
Remote PE: 10.100.100.3, Negotiated control-word: No
Incoming label: 327707, Outgoing label: 262153
Local interface: lsi.1048578, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt10001 local site 1 remote site 3
Flow Label Transmit: No, Flow Label Receive: No
~
Instance: rt10015
Edge protection: Not-Primary
Local site: rt10015 (1)
connection-site Type St Time last up # Up trans
3 rmt Up Dec 25 05:43:09 2023 1
Remote PE: 10.100.100.3, Negotiated control-word: No
Incoming label: 327819, Outgoing label: 262265
Local interface: lsi.1048592, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt10015 local site 1 remote site 3
Flow Label Transmit: No, Flow Label Receive: No
Instance: rt10016
Edge protection: Not-Primary
Local site: rt10016 (1)
connection-site Type St Time last up # Up trans
3 rmt VC-Dn ----- 0
Remote PE: 10.100.100.3, Negotiated control-word: No
Incoming label: 327827, Outgoing label: 262273
Local interface: lsi.1048593, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt10016 local site 1 remote site 3
Flow Label Transmit: No, Flow Label Receive: No
~
Instance: rt10064
Edge protection: Not-Primary
Local site: rt10064 (1)
connection-site Type St Time last up # Up trans
3 rmt VC-Dn ----- 0
Remote PE: 10.100.100.3, Negotiated control-word: No
Incoming label: 393243, Outgoing label: 327689
Local interface: lsi.1048641, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt10064 local site 1 remote site 3
Flow Label Transmit: No, Flow Label Receive: No
jcluser@vMX1>
show route forwarding-table table "table-name"
rt00000,rt10000(Up) : lsi及びvMX2向けInterfaceが確認できる
rt00016,rt10016(VC-Dn) : lsiが確認できない
jcluser@vMX1> show route forwarding-table table rt00000
Routing table: rt00000.vpls
VPLS:
Enabled protocols: ACKed by all peers,
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 dscd 580 1
lsi.1048576 intf 0 indr 1048574 4
10.100.12.2 Push 262153 598 2 ge-0/0/0.0
0x30004/51 user 0 comp 602 2
ge-0/0/1.1000 intf 0 ucst 592 3 ge-0/0/1.1000
0x30005/51 user 0 comp 596 2
0x30003/51 user 0 comp 597 2
jcluser@vMX1> show route forwarding-table table rt00016
Routing table: rt00016.vpls
VPLS:
Enabled protocols: ACKed by all peers,
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 dscd 669 1
ge-0/0/1.1016 intf 0 ucst 762 3 ge-0/0/1.1016
0x30065/51 user 0 comp 1074 2
0x30015/51 user 0 comp 1075 2
jcluser@vMX1>
jcluser@vMX1> show route forwarding-table table rt10000
Routing table: rt10000.vpls
VPLS:
Enabled protocols: ACKed by all peers,
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 dscd 1179 1
lsi.1048577 intf 0 indr 1048575 4
10.100.13.2 Push 262145 1764 2 ge-0/0/2.0
0x30107/51 user 0 comp 1787 2
ge-0/0/1.2000 intf 0 ucst 1195 3 ge-0/0/1.2000
0x30108/51 user 0 comp 1634 2
0x300c6/51 user 0 comp 1635 2
jcluser@vMX1> show route forwarding-table table rt10016
Routing table: rt10016.vpls
VPLS:
Enabled protocols: ACKed by all peers,
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 dscd 1250 1
ge-0/0/1.2016 intf 0 ucst 1339 3 ge-0/0/1.2016
0x30128/51 user 0 comp 1666 2
0x300d6/51 user 0 comp 1667 2
jcluser@vMX1>
ge-0/0/1.1002 intf 0
※ vMX2,vMX3も同様に16本で制限されています。長くなるため本記事ではログは省略しています。
制限拡張
ECMPの値による制限のため、ECMPの値を変更すれば拡張することができます。
設定可能なパラメータは下記ヘルプの通りです。
今回は64で設定して、有効であることを確認しました。
jcluser@vMX1# set chassis maximum-ecmp ?
Possible completions:
16 Maximum 16 ECMP routes
32 Maximum 32 ECMP routes
64 Maximum 64 ECMP routes
96 Maximum 96 ECMP routes
128 Maximum 128 ECMP routes
[edit]
jcluser@vMX1# set chassis maximum-ecmp 64
[edit]
jcluser@vMX1# commit
commit complete
[edit]
jcluser@vMX1#
jcluser@vMX1# run show configuration chassis maximum-ecmp | display set
set chassis maximum-ecmp 64
[edit]
jcluser@vMX1#
vMX1
show route table inet.3
vMX2(10.100.100.2)向け：65本中64本が確認できる。1本(65本目)は確認できない。
vMX3(10.100.100.3)向け：65本中64本が確認できる。1本(65本目)は確認できない。
jcluser@vMX1> show route table inet.3 | no-more
inet.3: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.100.100.2/32 *[RSVP/7/1] 00:10:01, metric 1
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-00
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-01
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-02
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-03
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-04
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-05
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-06
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-07
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-08
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-09
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-10
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-11
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-12
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-13
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-14
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-15
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-16
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-17
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-18
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-19
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-20
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-21
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-22
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-23
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-24
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-25
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-26
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-27
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-28
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-29
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-30
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-31
> to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-32
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-33
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-34
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-35
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-36
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-37
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-38
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-39
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-40
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-41
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-42
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-43
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-44
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-45
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-46
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-47
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-48
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-49
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-50
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-51
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-52
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-53
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-54
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-55
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-56
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-57
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-58
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-59
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-60
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-61
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-62
to 10.100.12.2 via ge-0/0/0.0, label-switched-path vMX1-to-vMX2-63
10.100.100.3/32 *[RSVP/7/1] 00:10:01, metric 1
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-00
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-01
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-02
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-03
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-04
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-05
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-06
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-07
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-08
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-09
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-10
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-11
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-12
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-13
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-14
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-15
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-16
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-17
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-18
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-19
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-20
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-21
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-22
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-23
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-24
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-25
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-26
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-27
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-28
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-29
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-30
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-31
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-32
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-33
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-34
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-35
> to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-36
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-37
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-38
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-39
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-40
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-41
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-42
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-43
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-44
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-45
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-46
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-47
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-48
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-49
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-50
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-51
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-52
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-53
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-54
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-55
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-56
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-57
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-58
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-59
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-60
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-61
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-62
to 10.100.13.2 via ge-0/0/2.0, label-switched-path vMX1-to-vMX3-63
jcluser@vMX1>
show vpls connections
各LSPに紐づいているVPLSの状態
vMX2(10.100.100.2)向け：65本中64本がUp,1本がVC-Dn
vMX3(10.100.100.3)向け：65本中64本がUp,1本がVC-Dn
jcluser@vMX1> show vpls connections
Layer-2 VPN connections:
Legend for connection status (St)
EI -- encapsulation invalid NC -- interface encapsulation not CCC/TCC/VPLS
EM -- encapsulation mismatch WE -- interface and instance encaps not same
VC-Dn -- Virtual circuit down NP -- interface hardware not present
CM -- control-word mismatch -> -- only outbound connection is up
CN -- circuit not provisioned <- -- only inbound connection is up
OR -- out of range Up -- operational
OL -- no outgoing label Dn -- down
LD -- local site signaled down CF -- call admission control failure
RD -- remote site signaled down SC -- local and remote site ID collision
LN -- local site not designated LM -- local site ID not minimum designated
RN -- remote site not designated RM -- remote site ID not minimum designated
XX -- unknown connection status IL -- no incoming label
MM -- MTU mismatch MI -- Mesh-Group ID not available
BK -- Backup connection ST -- Standby connection
PF -- Profile parse failure PB -- Profile busy
RS -- remote site standby SN -- Static Neighbor
LB -- Local site not best-site RB -- Remote site not best-site
VM -- VLAN ID mismatch HS -- Hot-standby Connection
Legend for interface status
Up -- operational
Dn -- down
Instance: rt00000
Edge protection: Not-Primary
Local site: rt00000 (1)
connection-site Type St Time last up # Up trans
2 rmt Up Dec 25 04:34:39 2023 1
Remote PE: 10.100.100.2, Negotiated control-word: No
Incoming label: 262154, Outgoing label: 262153
Local interface: lsi.1048576, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt00000 local site 1 remote site 2
Flow Label Transmit: No, Flow Label Receive: No
Instance: rt00001
Edge protection: Not-Primary
Local site: rt00001 (1)
connection-site Type St Time last up # Up trans
2 rmt Up Dec 25 05:54:22 2023 1
Remote PE: 10.100.100.2, Negotiated control-word: No
Incoming label: 262162, Outgoing label: 262161
Local interface: lsi.1048642, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt00001 local site 1 remote site 2
Flow Label Transmit: No, Flow Label Receive: No
~
Instance: rt00063
Edge protection: Not-Primary
Local site: rt00063 (1)
connection-site Type St Time last up # Up trans
2 rmt Up Dec 25 06:55:54 2023 1
Remote PE: 10.100.100.2, Negotiated control-word: No
Incoming label: 327682, Outgoing label: 327681
Local interface: lsi.1048704, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt00063 local site 1 remote site 2
Flow Label Transmit: No, Flow Label Receive: No
Instance: rt00064
Edge protection: Not-Primary
Local site: rt00064 (1)
connection-site Type St Time last up # Up trans
2 rmt VC-Dn ----- 0
Remote PE: 10.100.100.2, Negotiated control-word: No
Incoming label: 327690, Outgoing label: 327689
Local interface: lsi.1048705, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt00064 local site 1 remote site 2
Flow Label Transmit: No, Flow Label Receive: No
Instance: rt10000
Edge protection: Not-Primary
Local site: rt10000 (1)
connection-site Type St Time last up # Up trans
3 rmt Up Dec 25 05:43:09 2023 1
Remote PE: 10.100.100.3, Negotiated control-word: No
Incoming label: 327699, Outgoing label: 262145
Local interface: lsi.1048577, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt10000 local site 1 remote site 3
Flow Label Transmit: No, Flow Label Receive: No
~
Instance: rt10063
Edge protection: Not-Primary
Local site: rt10063 (1)
connection-site Type St Time last up # Up trans
3 rmt Up Dec 25 06:55:54 2023 1
Remote PE: 10.100.100.3, Negotiated control-word: No
Incoming label: 393235, Outgoing label: 327681
Local interface: lsi.1048640, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt10063 local site 1 remote site 3
Flow Label Transmit: No, Flow Label Receive: No
Instance: rt10064
Edge protection: Not-Primary
Local site: rt10064 (1)
connection-site Type St Time last up # Up trans
3 rmt VC-Dn ----- 0
Remote PE: 10.100.100.3, Negotiated control-word: No
Incoming label: 393243, Outgoing label: 327689
Local interface: lsi.1048641, Status: Up, Encapsulation: VPLS
Description: Intf - vpls rt10064 local site 1 remote site 3
Flow Label Transmit: No, Flow Label Receive: No
jcluser@vMX1>
show route forwarding-table table "table-name"
rt00000,rt00063,rt10000,rt10063(Up) : lsi及びvMX2向けInterfaceが確認できる
rt00064,rt10064(VC-Dn) : lsiが確認できない
jcluser@vMX1> show route forwarding-table table rt00000
Routing table: rt00000.vpls
VPLS:
Enabled protocols: ACKed by all peers,
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 dscd 580 1
lsi.1048576 intf 0 indr 1048574 4
10.100.12.2 Push 262153 598 2 ge-0/0/0.0
0x30004/51 user 0 comp 602 2
ge-0/0/1.1000 intf 0 ucst 592 3 ge-0/0/1.1000
0x30005/51 user 0 comp 596 2
0x30003/51 user 0 comp 597 2
jcluser@vMX1> show route forwarding-table table rt00063
Routing table: rt00063.vpls
VPLS:
Enabled protocols: ACKed by all peers,
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 dscd 879 1
lsi.1048704 intf 0 indr 1048606 4
10.100.12.2 Push 327681 2115 2 ge-0/0/0.0
0x300c2/51 user 0 comp 2307 2
ge-0/0/1.1063 intf 0 ucst 921 3 ge-0/0/1.1063
0x300c3/51 user 0 comp 1175 2
0x30044/51 user 0 comp 1176 2
jcluser@vMX1> show route forwarding-table table rt00064
Routing table: rt00064.vpls
VPLS:
Enabled protocols: ACKed by all peers,
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 dscd 884 1
ge-0/0/1.1064 intf 0 ucst 922 3 ge-0/0/1.1064
0x300c5/51 user 0 comp 1177 2
0x30045/51 user 0 comp 1178 2
jcluser@vMX1> show route forwarding-table table rt10000
Routing table: rt10000.vpls
VPLS:
Enabled protocols: ACKed by all peers,
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 dscd 1179 1
lsi.1048577 intf 0 indr 1048575 4
10.100.13.2 Push 262145 1764 2 ge-0/0/2.0
0x30107/51 user 0 comp 1787 2
ge-0/0/1.2000 intf 0 ucst 1195 3 ge-0/0/1.2000
0x30108/51 user 0 comp 1634 2
0x300c6/51 user 0 comp 1635 2
jcluser@vMX1> show route forwarding-table table rt10063
Routing table: rt10063.vpls
VPLS:
Enabled protocols: ACKed by all peers,
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 dscd 1463 1
lsi.1048640 intf 0 indr 1048654 4
10.100.13.2 Push 327681 2163 2 ge-0/0/2.0
0x30185/51 user 0 comp 2355 2
ge-0/0/1.2063 intf 0 ucst 1502 3 ge-0/0/1.2063
0x30186/51 user 0 comp 1760 2
0x30105/51 user 0 comp 1761 2
jcluser@vMX1> show route forwarding-table table rt10064
Routing table: rt10064.vpls
VPLS:
Enabled protocols: ACKed by all peers,
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 dscd 1468 1
ge-0/0/1.2064 intf 0 ucst 1503 3 ge-0/0/1.2064
0x30188/51 user 0 comp 1762 2
0x30106/51 user 0 comp 1763 2
jcluser@vMX1>
まとめ
機器の導入やOSバージョン検証時には、スケール検証も大切であると体感したトラブルでした。
参考記事
- https://www.juniper.net/documentation/us/en/software/junos/cli-reference/topics/ref/statement/maximum-ecmp-edit-chassis.html
- https://jlabs.juniper.net/vlabs/
- https://www.juniper.net/documentation/jp/ja/software/junos/vpn-l2/topics/example/vpls-bgp-configuring-detailed-solutions.html
- https://www.infraexpert.com/infra/junosr01.html
その他
本検証で使ったコンフィグやshowコマンドの結果の完全版は別途下記の記事に記載しております。