LoginSignup
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

@k-maki(Katsumi Maki)

SR-MPLSのトラフィックをNetFlowで見える化してみる

Last updated at Posted at 2024-12-29

NetFlowのコレクタとしてFluentdを利用し、SR-MPLSのトラフィックを可視化を行った際のメモ (ネットワークシュミレータを利用)

アウトプットのイメージと構成

アウトプットのイメージ

GrafanaのSankey Panelのプラグインを利用し、以下のように宛先のNode-SIDとVPNごとのトラフィックの流量の可視化を行います
SR-MPLSでは1枚目のラベルが宛先のNode-SIDのラベル、2枚目が宛先Node内のVPNを指定するラベルとなるので、Nodeごとのトラフィック量とさらに、その内部のVPN単位のトラフィック量を出せるようにしてみました
image.png

構成

  • vJunosEvolvedを利用しSR-MPLS/EVPN環境を構築
  • 真ん中のルータのMPLSトラフィックをNetFlowのデータとして出す設定を行う
  • ネットワークシュミレータ上にUbuntuを設置してFlowコレクタとして利用
    • Fluentd・Prometheus・Grafanaを利用

image.png

Fluentd・Prometheus・Grafanaを利用したサーバの準備は以下を参考にしてください

ネームシュミレータでvJunosEvolvedを利用の方法は以下を参考にしてください

vJunosEvolvedの準備

SR-MPLS/EVPNをつくる

以下のサイト参考に設定

  • SR-MPLSはOSPFをIGPで利用
  • EVPNでL2VPNを3つ作りそれぞれ左右のPCから通信させる
  • EVPNで利用するBGPは、PE-1/PE-2/PE-3をメッシュでネイバーをはる

MPLSのNetFlow送信設定

以下のサイトを参考に設定

  • 今回は、真ん中のルータの左(et-0/0/0)のポートから受信したMPLSトラフィックをNetFlowのデータとして出す設定を行う

vJunosEvolvedのコンフィグ

上記に内容を反映したコンフィグ

vJunosEvo-P-1のコンフィグはコチラ
vJunosEvo-P-1のコンフィグ
set version 23.4R2-S2.1-EVO
set system host-name vJunosEvo-P-1
set system root-authentication encrypted-password "$6$MmGV7bPW$AuqgoIhnFiZQD9eLmmhS.Fwywh63knLvg96p9VLQXP5jSxMk3QXSBFph4Z3W1Hcvf34xdlIh0QRlHrh/j8rr5/"
set system syslog file interactive-commands interactive-commands any
set system syslog file messages any notice
set system syslog file messages authorization info
set chassis fpc 0 sampling-instance SAMPLING_INSTANCE
set services flow-monitoring version9 template TEMPLATE_MPLS flow-active-timeout 120
set services flow-monitoring version9 template TEMPLATE_MPLS flow-inactive-timeout 60
set services flow-monitoring version9 template TEMPLATE_MPLS template-refresh-rate seconds 100
set services flow-monitoring version9 template TEMPLATE_MPLS option-refresh-rate seconds 600
set services flow-monitoring version9 template TEMPLATE_MPLS mpls-template label-position 1
set services flow-monitoring version9 template TEMPLATE_MPLS mpls-template label-position 2
set interfaces et-0/0/1 unit 0 family inet address 10.9.0.1/30
set interfaces et-0/0/1 unit 0 family mpls filter input FILTER_MPLS
set interfaces et-0/0/2 unit 0 family inet address 10.9.0.5/30
set interfaces et-0/0/2 unit 0 family mpls
set interfaces et-0/0/3 unit 0 family inet address 10.9.0.9/30
set interfaces et-0/0/3 unit 0 family mpls
set interfaces et-0/0/9 unit 0 family inet address 10.99.0.254/24
set interfaces lo0 unit 0 family inet address 10.0.0.10/32
set forwarding-options sampling instance SAMPLING_INSTANCE input rate 10
set forwarding-options sampling instance SAMPLING_INSTANCE family mpls output flow-server 10.99.0.1 port 4739
set forwarding-options sampling instance SAMPLING_INSTANCE family mpls output flow-server 10.99.0.1 version9 template TEMPLATE_MPLS
set forwarding-options sampling instance SAMPLING_INSTANCE family mpls output inline-jflow source-address 10.0.0.10
set firewall family mpls filter FILTER_MPLS term 1 then sample
set firewall family mpls filter FILTER_MPLS term 1 then accept
set routing-options router-id 10.0.0.10
set routing-options autonomous-system 65000
set protocols mpls interface et-0/0/1.0
set protocols mpls interface et-0/0/2.0
set protocols mpls interface et-0/0/3.0
set protocols ospf source-packet-routing node-segment ipv4-index 10
set protocols ospf source-packet-routing srgb start-label 16000
set protocols ospf source-packet-routing srgb index-range 8000
set protocols ospf area 0.0.0.0 interface lo0.0 passive
set protocols ospf area 0.0.0.0 interface lo0.0 metric 10
set protocols ospf area 0.0.0.0 interface et-0/0/1.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface et-0/0/1.0 metric 100
set protocols ospf area 0.0.0.0 interface et-0/0/2.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface et-0/0/2.0 metric 100
set protocols ospf area 0.0.0.0 interface et-0/0/3.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface et-0/0/3.0 metric 100
set protocols ospf area 0.0.0.0 interface et-0/0/9.0 passive
set protocols ospf area 0.0.0.0 interface et-0/0/9.0 metric 100
vJunosEvo-PE-1のコンフィグはコチラ
vJunosEvo-PE-1のコンフィグ
set version 23.4R2-S2.1-EVO
set system host-name vJunosEvo-PE-1
set system root-authentication encrypted-password "$6$MmGV7bPW$AuqgoIhnFiZQD9eLmmhS.Fwywh63knLvg96p9VLQXP5jSxMk3QXSBFph4Z3W1Hcvf34xdlIh0QRlHrh/j8rr5/"
set system syslog file interactive-commands interactive-commands any
set system syslog file messages any notice
set system syslog file messages authorization info
set interfaces et-0/0/0 unit 0 family inet address 10.9.0.2/30
set interfaces et-0/0/0 unit 0 family mpls
set interfaces et-0/0/1 encapsulation ethernet-bridge
set interfaces et-0/0/1 unit 0
set interfaces et-0/0/2 encapsulation ethernet-bridge
set interfaces et-0/0/2 unit 0
set interfaces et-0/0/3 encapsulation ethernet-bridge
set interfaces et-0/0/3 unit 0
set interfaces lo0 unit 0 family inet address 10.0.0.1/32
set routing-instances EVPN_L2VPN_100 instance-type mac-vrf
set routing-instances EVPN_L2VPN_100 protocols evpn encapsulation mpls
set routing-instances EVPN_L2VPN_100 protocols evpn normalization
set routing-instances EVPN_L2VPN_100 service-type vlan-based
set routing-instances EVPN_L2VPN_100 interface et-0/0/1.0
set routing-instances EVPN_L2VPN_100 route-distinguisher 10.0.0.1:100
set routing-instances EVPN_L2VPN_100 vrf-target target:65000:100
set routing-instances EVPN_L2VPN_100 vlans BRIDGE_100 vlan-id none
set routing-instances EVPN_L2VPN_100 vlans BRIDGE_100 interface et-0/0/1.0
set routing-instances EVPN_L2VPN_200 instance-type mac-vrf
set routing-instances EVPN_L2VPN_200 protocols evpn encapsulation mpls
set routing-instances EVPN_L2VPN_200 protocols evpn normalization
set routing-instances EVPN_L2VPN_200 service-type vlan-based
set routing-instances EVPN_L2VPN_200 interface et-0/0/2.0
set routing-instances EVPN_L2VPN_200 route-distinguisher 10.0.0.1:200
set routing-instances EVPN_L2VPN_200 vrf-target target:65000:200
set routing-instances EVPN_L2VPN_200 vlans BRIDGE_200 vlan-id none
set routing-instances EVPN_L2VPN_200 vlans BRIDGE_200 interface et-0/0/2.0
set routing-instances EVPN_L2VPN_300 instance-type mac-vrf
set routing-instances EVPN_L2VPN_300 protocols evpn encapsulation mpls
set routing-instances EVPN_L2VPN_300 protocols evpn normalization
set routing-instances EVPN_L2VPN_300 service-type vlan-based
set routing-instances EVPN_L2VPN_300 interface et-0/0/3.0
set routing-instances EVPN_L2VPN_300 route-distinguisher 10.0.0.1:300
set routing-instances EVPN_L2VPN_300 vrf-target target:65000:300
set routing-instances EVPN_L2VPN_300 vlans BRIDGE_300 vlan-id none
set routing-instances EVPN_L2VPN_300 vlans BRIDGE_300 interface et-0/0/3.0
set routing-options router-id 10.0.0.1
set routing-options autonomous-system 65000
set protocols bgp group MP_BGP type internal
set protocols bgp group MP_BGP local-address 10.0.0.1
set protocols bgp group MP_BGP family evpn signaling
set protocols bgp group MP_BGP neighbor 10.0.0.2
set protocols bgp group MP_BGP neighbor 10.0.0.3
set protocols mpls interface et-0/0/0.0
set protocols ospf source-packet-routing node-segment ipv4-index 1
set protocols ospf source-packet-routing srgb start-label 16000
set protocols ospf source-packet-routing srgb index-range 8000
set protocols ospf area 0.0.0.0 interface lo0.0 passive
set protocols ospf area 0.0.0.0 interface lo0.0 metric 10
set protocols ospf area 0.0.0.0 interface et-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface et-0/0/0.0 metric 100
vJunosEvo-PE-2のコンフィグはコチラ
vJunosEvo-PE-2のコンフィグ
set version 23.4R2-S2.1-EVO
set system host-name vJunosEvo-PE-2
set system root-authentication encrypted-password "$6$MmGV7bPW$AuqgoIhnFiZQD9eLmmhS.Fwywh63knLvg96p9VLQXP5jSxMk3QXSBFph4Z3W1Hcvf34xdlIh0QRlHrh/j8rr5/"
set system syslog file interactive-commands interactive-commands any
set system syslog file messages any notice
set system syslog file messages authorization info
set interfaces et-0/0/0 unit 0 family inet address 10.9.0.6/30
set interfaces et-0/0/0 unit 0 family mpls
set interfaces et-0/0/1 encapsulation ethernet-bridge
set interfaces et-0/0/1 unit 0
set interfaces lo0 unit 0 family inet address 10.0.0.2/32
set routing-instances EVPN_L2VPN_100 instance-type mac-vrf
set routing-instances EVPN_L2VPN_100 protocols evpn encapsulation mpls
set routing-instances EVPN_L2VPN_100 protocols evpn normalization
set routing-instances EVPN_L2VPN_100 service-type vlan-based
set routing-instances EVPN_L2VPN_100 interface et-0/0/1.0
set routing-instances EVPN_L2VPN_100 route-distinguisher 10.0.0.2:100
set routing-instances EVPN_L2VPN_100 vrf-target target:65000:100
set routing-instances EVPN_L2VPN_100 vlans BRIDGE_100 vlan-id none
set routing-instances EVPN_L2VPN_100 vlans BRIDGE_100 interface et-0/0/1.0
set routing-options router-id 10.0.0.2
set routing-options autonomous-system 65000
set protocols bgp group MP_BGP type internal
set protocols bgp group MP_BGP local-address 10.0.0.2
set protocols bgp group MP_BGP family evpn signaling
set protocols bgp group MP_BGP neighbor 10.0.0.1
set protocols bgp group MP_BGP neighbor 10.0.0.3
set protocols mpls interface et-0/0/0.0
set protocols ospf source-packet-routing node-segment ipv4-index 2
set protocols ospf source-packet-routing srgb start-label 16000
set protocols ospf source-packet-routing srgb index-range 8000
set protocols ospf area 0.0.0.0 interface lo0.0 passive
set protocols ospf area 0.0.0.0 interface lo0.0 metric 10
set protocols ospf area 0.0.0.0 interface et-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface et-0/0/0.0 metric 100
vJunosEvo-PE-3のコンフィグはコチラ
vJunosEvo-PE-3のコンフィグ
set version 23.4R2-S2.1-EVO
set system host-name vJunosEvo-PE-3
set system root-authentication encrypted-password "$6$MmGV7bPW$AuqgoIhnFiZQD9eLmmhS.Fwywh63knLvg96p9VLQXP5jSxMk3QXSBFph4Z3W1Hcvf34xdlIh0QRlHrh/j8rr5/"
set system syslog file interactive-commands interactive-commands any
set system syslog file messages any notice
set system syslog file messages authorization info
set interfaces et-0/0/0 unit 0 family inet address 10.9.0.10/30
set interfaces et-0/0/0 unit 0 family mpls
set interfaces et-0/0/2 encapsulation ethernet-bridge
set interfaces et-0/0/2 unit 0
set interfaces et-0/0/3 encapsulation ethernet-bridge
set interfaces et-0/0/3 unit 0
set interfaces lo0 unit 0 family inet address 10.0.0.3/32
set routing-instances EVPN_L2VPN_200 instance-type mac-vrf
set routing-instances EVPN_L2VPN_200 protocols evpn encapsulation mpls
set routing-instances EVPN_L2VPN_200 protocols evpn normalization
set routing-instances EVPN_L2VPN_200 service-type vlan-based
set routing-instances EVPN_L2VPN_200 interface et-0/0/2.0
set routing-instances EVPN_L2VPN_200 route-distinguisher 10.0.0.3:200
set routing-instances EVPN_L2VPN_200 vrf-target target:65000:200
set routing-instances EVPN_L2VPN_200 vlans BRIDGE_200 vlan-id none
set routing-instances EVPN_L2VPN_200 vlans BRIDGE_200 interface et-0/0/2.0
set routing-instances EVPN_L2VPN_300 instance-type mac-vrf
set routing-instances EVPN_L2VPN_300 protocols evpn encapsulation mpls
set routing-instances EVPN_L2VPN_300 protocols evpn normalization
set routing-instances EVPN_L2VPN_300 service-type vlan-based
set routing-instances EVPN_L2VPN_300 interface et-0/0/3.0
set routing-instances EVPN_L2VPN_300 route-distinguisher 10.0.0.3:300
set routing-instances EVPN_L2VPN_300 vrf-target target:65000:300
set routing-instances EVPN_L2VPN_300 vlans BRIDGE_300 vlan-id none
set routing-instances EVPN_L2VPN_300 vlans BRIDGE_300 interface et-0/0/3.0
set routing-options router-id 10.0.0.3
set routing-options autonomous-system 65000
set protocols bgp group MP_BGP type internal
set protocols bgp group MP_BGP local-address 10.0.0.3
set protocols bgp group MP_BGP family evpn signaling
set protocols bgp group MP_BGP neighbor 10.0.0.1
set protocols bgp group MP_BGP neighbor 10.0.0.2
set protocols mpls interface et-0/0/0.0
set protocols ospf source-packet-routing node-segment ipv4-index 3
set protocols ospf source-packet-routing srgb start-label 16000
set protocols ospf source-packet-routing srgb index-range 8000
set protocols ospf area 0.0.0.0 interface lo0.0 passive
set protocols ospf area 0.0.0.0 interface lo0.0 metric 10
set protocols ospf area 0.0.0.0 interface et-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface et-0/0/0.0 metric 100

※rootのパスワードは仮で「root123」にしてます

FluentdでMPLSのNetFlow処理設定

受信するNetFlow情報の確認

  • 「/etc/fluent/fluentd.conf」を編集
fluentd.conf 設定内容
# NetFlowデータを取得
<source>
  @type netflow
  tag netflow.event
  port 4739
</source>

# デバッグ用に出力
<match **>
  @type stdout
</match>
  • 設定内容を反映させるため fluentd をリスタート
    • systemctl restart fluentd.service
  • PC間でPINGを打ち続けて Flowデータをだす
  • ログを tail　してFlowデータをFluentdで受信していることを確認する
    • tail -f /var/log/fluent/fluentd.log
    • MPLSのテンプレートのフォーマットで受信を確認できます
fluentd.log でFlowデータ受信した内容
2024-xx-xx xx:xx:xx.000000000 +0900 netflow.event: {"version":"9","flow_seq_num":"1137","flowset_id":"259","input_snmp":539,"output_snmp":0,"in_bytes":454,"in_pkts":1,"first_switched":"2024-xx-xxTxx:xx:57.957Z","last_switched":"2024-xx-xxTxx:xx:xx.957Z","mpls_label_1":"16003","mpls_label_2":"17","mpls_label_3":"0","host":"10.0.0.10"}

NetFlow情報を整形

  • 再度「/etc/fluent/fluentd.conf」を編集する
    • 受信したMPLSフローデータの「in_bytes」をキーにする
    • 事前に取得したログからラベルを設定し、必要な物以外はコメントアウト
    • Prometheusでデータ取得できるように
  • 以下の内容に変更し設定内容を反映させるため fluentd をリスタート
    • systemctl restart fluentd.service
fluentd.conf 設定内容はコチラ
fluentd.conf 設定内容
#-----------------------
# Prometheus用
#-----------------------
<source>
  @type prometheus
  @id in_prometheus
  bind "0.0.0.0"
  port 24231
  metrics_path "/metrics"
</source>

#-----------------------
# NetFlow用
#-----------------------
<source>
  @type netflow
  tag netflow.event
  port 4739

</source>

<match netflow.event>
  @type copy
  <store>
    @type rewrite_tag_filter
    <rule>
      key mpls_label_1
      pattern /(.*)/
      tag netflow.mpls
    </rule>
  </store>
</match>

<filter netflow.mpls>
  @type prometheus
  <metric>
    name netflow_mpls_in_byte
    desc NetFlow MPLS Traffic input byte
    type counter
    key in_bytes
    <labels>
      #version ${version}
      #flow_seq_num ${flow_seq_num}
      #flowset_id ${flowset_id}
      #first_switched ${first_switched}
      mpls_label_1 ${mpls_label_1}
      mpls_label_2 ${mpls_label_2}
      mpls_label_3 ${mpls_label_3}
      input_snmp ${input_snmp}
      #output_snmp ${output_snmp}
      #last_switched ${last_switched}
      host ${host}
    </labels>
  </metric>
</filter>

<match netflow.mpls>
  #@type stdout  # デバッグ用
  @type null
</match>

Prometheusでのデータ確認

Prometheusでのデータ確認は以下を参考にしてください

GrafanaのSankey-panelのプラグインをインストール

トラフィックを可視化させるための以下のプラグインをインストールする

Ubuntuのサーバ上で以下を実施

Sankey Panelインストール
grafana-cli plugins install netsage-sankey-panel

プラグインインストール後、権限がrootのままのため変更の必要あり

Sankey Panelインストール後の権限
# ll /var/lib/grafana/plugins/
total 16
drwxr-x--- 4 grafana grafana 4096 Dec 28 20:10 ./
drwxr-xr-x 6 grafana grafana 4096 Dec 28 20:02 ../
drwxr-x--- 3 grafana grafana 4096 Nov 23 10:45 grafana-lokiexplore-app/
drwxr-xr-x 3 root    root    4096 Dec 28 20:10 netsage-sankey-panel/

権限を他に合わせて以下のコマンドでgrafanaに変更しプラグインのインストール完了

grafanaに変更
chown -R grafana:grafana /var/lib/grafana/plugins/

Grafana上での設定

Prometheusと連携

Grafanaにログインし、「Data sources」を選択し、次の画面でPrometheusを選択
image.png

次の画面で、Connectionの箇所にURLを「 http://localhost:9090 」で入力し、画面下の「Save & test」でPrometheusのデータと連携
image.png

ダッシュボードの作成

画面左のメニューから「Dashboards」を選び。「Create dashboard」を選択
image.png

「Add visualization」を選択し、data sourceで「prometheus」を選択する
image.png

可視化のためのSankey Panelを作る

ダッシュボード作成画面の右上のパネルの種類から「Sankey Panel」を選択する
image.png
「Queries」で以下のように入力

  • 「Data source」は「prometheus」
  • 「Code」を選択
  • 「Metrics browser」に以下を入力
Metrics browser
label_join(
  sum by (host, mpls_label_1, mpls_label_2) (rate(netflow_mpls_in_byte[$__rate_interval])*8*10),
  "target",
  " VPN:",
  "mpls_label_1",
  "mpls_label_2"
)

image.png

「Metrics browser」は、以下の内容で設定しています

  • rateは、元のデータがbyteでbps計算のため、8倍、またJunosEvoで設定しているflowのinput rateの設定が「input rate 10」(10分の1のパケットをサンプリングしてNetFlowレコードとして出力)のため、さらに10倍
  • MPLSラベルの1枚目と2枚目の組合せを表示させたかったため「label_join」で「target」という名前で設定
  • 単純なSankey Panel表示であれば以下で表示できます
    • sum by (host, mpls_label_1)(rate(netflow_mpls_in_byte[$__rate_interval]))

ここまでやると以下のような図が表示される
image.png

不要な情報が多いため「Transformations」で修正
以下の画面の「Add transformation」→「Organize fields by name」を選択する
image.png
不要な項目を表示にする、ここでは「Time」と「mpls_label_2」を非表示にして、図に表示される文字も変更
image.png

完成
image.png

最後に

Grafanaの「Sankey Panel」のプラグインを利用してSR-MPLSのトラフィックの可視化まで確認できました
線の太さでトラフィック量が可視化されるので全体を俯瞰してトラフィック量の把握ができるようになりました
(今回はPC同士のPINGのサイズを変えてトラフィック量に差をつけました)
通常のIPv4やIPv6のFlowでも同じように宛先ごとのトラフィックの可視化も可能です

0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?