Go to Qiita Advent Calendar 2024 Top
LoginSignup
4
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

SRAAdvent Calendar 2024

Day 2
@uskf(Yusuke Fujimaki)in株式会社SRA

vJunos-switchで仮想Closネットワーク作ってみた

Posted at

SRA Advent Calendarの2日目です。
ネットワークシステムサービス第１事業部の ふじまき です。

仕事でClosネットワークに関わる機会があったので、勉強の為にJuniperが公開しているvJunos-switchとFRRoutingで仮想Closネットワークを作ってみました。

なお、基本サーバ屋なのでJuniperやBGPの事は良く分かってません。:smiley:

構成図

clos-network.png

SPINEスイッチ
(vJunos-switch)		 LEAFスイッチ
(vJunos-switch)		 サーバ
(FRRouting)
ssw01,ssw02 lsw11,lsw12 srv21,srv22
  • fxp0 : vJunos-switchの管理ポート
  • ge000～ge003: vJunos-switchのGigabitEthernetポート（実際にはge-0/0/0 ～ ge-0/0/3）
  • enp6s0～enp8s0: LinuxゲストのNIC
  • 双方向の矢印: 両端の機器（のポート）を接続するブリッジ。命名ルールは"br-"＋両端のホスト名の数字部分をつなげたもの。ここの接続はBGP unnumbered
  • 緑の<3001>～<3012>: vJunos-switchのシリアルポートの待ち受けポート。
    $ telnet localhost 3001
  • 172.16.10.10/32をサービスIPとして広報

ブリッジの準備

KVMホスト上にブリッジを作成

# SPINE-LEAF間
$ for i in 01 02;do for j in 11 12;do sudo nmcli c add type bridge con-name br-$i$j stp no ifname br-$i$j ipv4.method disabled ipv6.method disabled;done;done
# LEAF-Server間
$ for i in 11 12;do for j in 21 22;do sudo nmcli c add type bridge con-name br-$i$j stp no ifname br-$i$j ipv4.method disabled ipv6.method disabled;done;done

vJunos-switchの作成

vJunos-switchはJuniperのDownloadページで仮想ディスクイメージファイルとして提供されているので、それをダウンロードします。（登録不要）

その仮想ディスクでLinuxゲストを作って起動すると、中でFreeBSDゲストがネスト化された仮想マシンとして起動するという構成になっています。（実物もLinux KVMホスト＋FreeBSDゲスト）

ディスクイメージをダウンロードしたら後は仮想マシンを作るだけですが通常と異なる点があります。

シリアルコンソールポート

仮想マシン定義のXMLの<devices>の子に以下のような定義を記載すると127.0.0.1:3001でtelnet接続できるコンソールポートが作られます。

    <console type='tcp'>
      <source mode='bind' host='127.0.0.1' service='3001'/>
      <protocol type='telnet'/>
      <target type='serial' port='0'/>
    </console>

謎のおまじない(SMBIOSパラメータ)

公式ドキュメントだと"ns0"という接頭辞がついてますが、今回作った環境では以下で動作しました。

  <qemu:commandline>
    <qemu:arg value='-smbios'/>
    <qemu:arg value='type=1,product=VM-VEX'/>
  </qemu:commandline>

ネットワークインターフェイス

PCIバス上で若い番号からfxp0 -> ge-0/0/0 -> ge-0/0/1 -> ge-0/0/2 ... にアサインされます。NICの構成を変える場合は順番がずれないように気を付ける必要があります。

最終的な仮想マシン定義

NICを増減させた時に悩みたくないので、NIC以外のデバイスを前に詰めてNICは最後に並べました。

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>ssw01</name>
  <uuid>88efd979-42a0-4a7e-a727-21d959cb94ee</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://ubuntu.com/ubuntu/18.04"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>5242880</memory>
  <currentMemory unit='KiB'>5242880</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-9.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <feature policy='require' name='smx'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/data/ssw01_disk1.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:01:00:0f'/>
      <source bridge='br7'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:01:00:00'/>
      <source bridge='br-0111'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:01:00:01'/>
      <source bridge='br-0112'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:01:00:02'/>
      <source bridge='br1'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <serial type='tcp'>
      <source mode='bind' host='127.0.0.1' service='3001'/>
      <protocol type='telnet'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='tcp'>
      <source mode='bind' host='127.0.0.1' service='3001'/>
      <protocol type='telnet'/>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <audio id='1' type='none'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-smbios'/>
    <qemu:arg value='type=1,product=VM-VEX'/>
  </qemu:commandline>
</domain>

MACアドレスは

  • 上位3bytesはデフォルトの52:54:00
  • 下位3bytesはホスト名の数字部分+Junos内のインターフェイス名(fxp0 -> f, GigabitEthernetは末尾の数字）

SPINEスイッチの設定

show | display set | no-more

set version 23.2R1.14
set system host-name ssw01
set system root-authentication encrypted-password "$6$B1.GePJh$gXwG1zv.1RcezLcJq0YMco6Ha.ingUD9jilfSfvYC0robl1RNPO9r4CoptsgNtml3rGJnXE6f4mNuRRtRzUns0"
set system login user admin uid 2000
set system login user admin class super-user
set system login user admin authentication encrypted-password "$6$skF0AlTS$QpmwseowaWgUbq5gp85j07stjs7b42JjNIC2qQyvPxImEk/ciytBN5L3Pul2QLeD3gDgsVLliaDQVJhKy9WCI."
set system arp aging-timer 5
set system syslog file interactive-commands interactive-commands any
set system syslog file messages any notice
set system syslog file messages authorization info
set interfaces interface-range iface-bgpunnum member "ge-0/0/[0-1]"
set interfaces ge-0/0/0 unit 0 family inet
set interfaces ge-0/0/0 unit 0 family inet6
set interfaces ge-0/0/1 unit 0 family inet
set interfaces ge-0/0/1 unit 0 family inet6
set interfaces ge-0/0/2 unit 0 family inet address 192.168.1.71/24
set interfaces fxp0 unit 0 family inet address 192.168.7.1/24
set interfaces lo0 unit 0 family inet address 10.0.0.1/32
set multi-chassis mc-lag consistency-check
set policy-options policy-statement pol-service-lb from route-filter 172.16.10.10/32 exact
set policy-options policy-statement pol-service-lb then load-balance per-flow
set policy-options policy-statement pol-swtlocal from protocol direct
set policy-options policy-statement pol-swtlocal from route-filter 10.0.0.0/24 orlonger
set policy-options policy-statement pol-swtlocal from route-filter 172.16.10.10/32 exact
set policy-options policy-statement pol-swtlocal then accept
set policy-options policy-statement policy-direct-accept from protocol direct
set policy-options policy-statement policy-direct-accept then accept
set policy-options as-list clos-asn-list members 4200100000-4200199999
set routing-options router-id 10.0.0.1
set routing-options autonomous-system 4200100000
set routing-options static route 0.0.0.0/0 next-hop 192.168.1.64
set routing-options forwarding-table export pol-service-lb
set protocols router-advertisement interface fxp0.0 managed-configuration
set protocols router-advertisement interface ge-0/0/1.0
set protocols router-advertisement interface ge-0/0/0.0
set protocols bgp group bgp_unnum hold-time 9
set protocols bgp group bgp_unnum family inet unicast extended-nexthop
set protocols bgp group bgp_unnum family inet6 unicast
set protocols bgp group bgp_unnum export policy-direct-accept
set protocols bgp group bgp_unnum multipath multiple-as
set protocols bgp group bgp_unnum dynamic-neighbor ndisc peer-auto-discovery family inet6 ipv6-nd
set protocols bgp group bgp_unnum dynamic-neighbor ndisc peer-auto-discovery interface iface-bgpunnum
set protocols bgp group bgp_unnum peer-as-list clos-asn-list
set protocols bgp group outer_network type external
set protocols bgp group outer_network family inet unicast
set protocols bgp group outer_network export pol-swtlocal
set protocols bgp group outer_network neighbor 192.168.1.64 peer-as 65000
set protocols lldp interface all
set protocols lldp-med interface all

ssw02はホスト名及び固有のIPアドレス以外同じなので略

LEAFスイッチの設定

show | display set | no-more

set version 23.2R1.14
set system host-name lsw11
set system root-authentication encrypted-password "$6$B1.GePJh$gXwG1zv.1RcezLcJq0YMco6Ha.ingUD9jilfSfvYC0robl1RNPO9r4CoptsgNtml3rGJnXE6f4mNuRRtRzUns0"
set system login user admin uid 2000
set system login user admin class super-user
set system login user admin authentication encrypted-password "$6$skF0AlTS$QpmwseowaWgUbq5gp85j07stjs7b42JjNIC2qQyvPxImEk/ciytBN5L3Pul2QLeD3gDgsVLliaDQVJhKy9WCI."
set system arp aging-timer 5
set system syslog file interactive-commands interactive-commands any
set system syslog file messages any notice
set system syslog file messages authorization info
set system ntp server 192.168.1.77
set interfaces interface-range iface-bgpunnum member "ge-0/0/[0-3]"
set interfaces interface-range iface-to-spine member "ge-0/0/[0-1]"
set interfaces interface-range iface-to-server member "ge-0/0/[2-3]"
set interfaces ge-0/0/0 unit 0 family inet
set interfaces ge-0/0/0 unit 0 family inet6
set interfaces ge-0/0/1 unit 0 family inet
set interfaces ge-0/0/1 unit 0 family inet6
set interfaces ge-0/0/2 unit 0 family inet
set interfaces ge-0/0/2 unit 0 family inet6
set interfaces ge-0/0/3 unit 0 family inet
set interfaces ge-0/0/3 unit 0 family inet6
set interfaces fxp0 unit 0 family inet address 192.168.7.11/24
set interfaces fxp0 unit 0 family inet6
set interfaces lo0 unit 0 family inet address 10.0.0.11/32
set multi-chassis mc-lag consistency-check
set policy-options prefix-list DEFAULTROUTE 0.0.0.0/0
set policy-options policy-statement ADV_DEFROUTE from prefix-list DEFAULTROUTE
set policy-options policy-statement ADV_DEFROUTE then accept
set policy-options policy-statement learn-from-bgp from protocol bgp
set policy-options policy-statement learn-from-bgp then accept
set policy-options policy-statement pol-service-lb from route-filter 172.16.10.10/32 exact
set policy-options policy-statement pol-service-lb then load-balance per-flow
set policy-options policy-statement policy-direct-accept from protocol direct
set policy-options policy-statement policy-direct-accept then accept
set policy-options as-list clos-asn-list members 4200100000-4200199999
set routing-options router-id 10.0.0.11
set routing-options autonomous-system 4200101000
set routing-options bgp-static route 0.0.0.0/0
set routing-options forwarding-table export pol-service-lb
set protocols router-advertisement interface fxp0.0 managed-configuration
set protocols router-advertisement interface ge-0/0/0.0
set protocols router-advertisement interface ge-0/0/1.0
set protocols router-advertisement interface ge-0/0/2.0
set protocols router-advertisement interface ge-0/0/3.0
set protocols bgp group bgp_to_spine hold-time 9
set protocols bgp group bgp_to_spine family inet unicast extended-nexthop
set protocols bgp group bgp_to_spine family inet6 unicast
set protocols bgp group bgp_to_spine export policy-direct-accept
set protocols bgp group bgp_to_spine export learn-from-bgp
set protocols bgp group bgp_to_spine multipath multiple-as
set protocols bgp group bgp_to_spine dynamic-neighbor spine peer-auto-discovery family inet6 ipv6-nd
set protocols bgp group bgp_to_spine dynamic-neighbor spine peer-auto-discovery interface iface-to-spine
set protocols bgp group bgp_to_spine peer-as-list clos-asn-list
set protocols bgp group bgp_to_server hold-time 9
set protocols bgp group bgp_to_server family inet unicast extended-nexthop
set protocols bgp group bgp_to_server family inet6 unicast
set protocols bgp group bgp_to_server export ADV_DEFROUTE
set protocols bgp group bgp_to_server multipath multiple-as
set protocols bgp group bgp_to_server advertise-bgp-static policy ADV_DEFROUTE
set protocols bgp group bgp_to_server dynamic-neighbor server peer-auto-discovery family inet6 ipv6-nd
set protocols bgp group bgp_to_server dynamic-neighbor server peer-auto-discovery interface iface-to-server
set protocols bgp group bgp_to_server peer-as-list clos-asn-list
set protocols bgp advertise-peer-as
set protocols bgp advertise-bgp-static
set protocols lldp interface all
set protocols lldp-med interface all

lsw12はホスト名及び固有のIPアドレス以外同じなので略

サーバの設定

サービス用のIP 172.16.10.10/32と、サーバ個別のIPアドレス10.0.0.2xを付与するためのdummyインターフェイス作成
IPアドレスはFRRoutingに付与させるのでNetworkManagerはインターフェイス作るだけ。

$ sudo nmcli c add type dummy con-name dummy0 ifname dummy0 ipv4.method disable ipv6.method disable
$ sudo nmcli c add type dummy con-name dummy1 ifname dummy1 ipv4.method disable ipv6.method disable

どっちから応答するか不定なのでrp_filter無効化（デフォルトルートはLEAFスイッチから受け取る）

/etc/sysctl.d/50-rpfilter.conf
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.enp6s0.rp_filter = 0
net.ipv4.conf.enp7s0.rp_filter = 0

サーバのFRRoutingの設定

!
frr version 8.5.3
frr defaults datacenter
hostname srv21
log syslog informational
no ip forwarding
no ipv6 forwarding
!
interface dummy0
 ip address 172.16.10.10/32
exit
!
interface dummy1
 ip address 10.0.0.21/32
exit
!
router bgp 4200101001
 no bgp default ipv4-unicast
 bgp bestpath as-path multipath-relax
 neighbor LEAF peer-group
 neighbor LEAF remote-as external
 neighbor LEAF bfd
 neighbor LEAF timers 1 3
 neighbor LEAF timers connect 5
 neighbor LEAF capability extended-nexthop
 neighbor enp6s0 interface peer-group LEAF
 neighbor enp7s0 interface peer-group LEAF
 !
 address-family ipv4 unicast
  network 10.0.0.21/32
  network 172.16.10.10/32
  neighbor LEAF activate
  neighbor LEAF soft-reconfiguration inbound
 exit-address-family
exit
!
end

srv22はホスト名及び固有のIPアドレス以外同じなので略

AS 65000のBGP Router(FRRouting)

実際にはKVMホスト上で動いているFRRouting

frr version 10.1.1
frr defaults datacenter
hostname kvm1
log syslog informational
no ip forwarding
no ipv6 forwarding
service integrated-vtysh-config
!
ip route 192.168.1.71/32 br1
ip route 192.168.1.72/32 br1
!
router bgp 65000
 bgp router-id 192.168.1.64
 bgp bestpath as-path multipath-relax
 timers bgp 60 180
 neighbor SPINE peer-group
 neighbor SPINE remote-as 4200100000
 neighbor SPINE ebgp-multihop
 neighbor SPINE timers 1 3
 neighbor SPINE timers connect 5
 neighbor 192.168.1.71 peer-group SPINE
 neighbor 192.168.1.72 peer-group SPINE
exit
!
end

確認

srv21,22で何もサービス動いていないので、ssh-keyscanで2台のサーバに分散してること確認

$ ssh-keyscan 172.16.10.10 | grep ssh-ed25519
172.16.10.10 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM0CaFq0agdUsma6uiWci1BiYPUDgvdnZacTHZuTYUBX
$ ssh-keyscan 172.16.10.10 | grep ssh-ed25519
172.16.10.10 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEfzeKvEAVnbZ7SLb8VcEWK7Q37c92zJpbjisv3A9e/D

感想

この構成だとまともに運用出来ないのでVRFで管理用の経路を作る必要あり。

4
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
4
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?