本記事では、プライベートLTEを想定して、Open5GS EPCとeBPF/XDP対応UPFのeUPFを連携動作させるための簡単な構成について説明します。 eUPFのインストールと設定については、こちらを参照してください。

Open5GS CU分離EPCのシミュレーション携帯網の概要

今回、Open5GS EPCのC-Plane、eUPF、およびData Network Gatewayの簡単な構成について説明します。この構成はVirtualbox VMを使用して実装されることに注意して下さい。
このシミュレーション環境は以下の条件を満たす最小構成になります。

1つのSGW-U/UPF(PGW-U)とData Network Gateway
1つのUEと1つのAPN

構築したシミュレーション環境は以下の通りです。

EPC、eUPFとUE/RANに使用しているOSSは以下の通りです。

EPC - Open5GS v2.7.0 (2023.12.04) - https://github.com/open5gs/open5gs
eBPF/XDP PGW-U - eUPF v0.6.0 (2023.12.04) - https://github.com/edgecomllc/eupf
UE / RAN - srsRAN 4G (2023.11.23) - https://github.com/srsran/srsRAN_4G

各VMは以下の通りです。

VM	SW & Role	IP address	OS	CPU (Min)	Memory (Min)	HDD (Min)
VM1	Open5GS EPC C-Plane	192.168.0.111/24	Ubuntu 22.04	1	1GB	20GB
VM2	Open5GS EPC U-Plane(SGW-U)	192.168.0.112/24	Ubuntu 22.04	1	1GB	20GB
VM-UP	eUPF U-Plane(PGW-U)	192.168.0.151/24	Ubuntu 22.04	1	2GB	20GB
VM-DN	Data Network Gateway	192.168.0.152/24	Ubuntu 22.04	1	1GB	10GB
VM3	srsRAN 4G ZMQ RAN (eNodeB)	192.168.0.121/24	Ubuntu 22.04	1	2GB	10GB
VM4	srsRAN 4G ZMQ UE	192.168.0.122/24	Ubuntu 22.04	1	2GB	10GB

各VMのネットワークインターフェースは以下の通りです。

VM	Device	Network Adapter	IP address	Interface	XDP
VM1	enp0s3	NAT(default)	10.0.2.15/24	(VM default NW)	--
	enp0s8	Bridged Adapter	192.168.0.111/24	(Mgmt NW)	--
	enp0s9	NAT Network	192.168.14.111/24	Sxb (N4 for 5GC)	--
VM2	enp0s3	NAT(default)	10.0.2.15/24	(VM default NW)	--
	enp0s8	Bridged Adapter	192.168.0.112/24	(Mgmt NW)	--
	enp0s9	NAT Network	192.168.13.112/24	S1-U,S5u (N3 for 5GC)	--
VM-UP	~~enp0s3~~	~~NAT(default)~~	~~10.0.2.15/24~~	~~(VM default NW)~~ down	--
	enp0s8	Bridged Adapter	192.168.0.151/24	(Mgmt NW)	--
	enp0s9	NAT Network	192.168.13.151/24	S5u (N3 for 5GC)	x
	enp0s10	NAT Network	192.168.14.151/24	Sxb (N4 for 5GC)	--
	enp0s16	NAT Network	192.168.16.151/24	SGi (N6 for 5GC)	x
VM-DN	enp0s3	NAT(default)	10.0.2.15/24	(VM default NW)	--
	enp0s8	Bridged Adapter	192.168.0.152/24	(Mgmt NW)	--
	enp0s9	NAT Network	192.168.16.152/24	N6, default GW for VM-UP	--
VM3	enp0s3	NAT(default)	10.0.2.15/24	(VM default NW)	--
	enp0s8	Bridged Adapter	192.168.0.121/24	(Mgmt NW)	--
	enp0s9	NAT Network	192.168.13.121/24	S1-U (N3 for 5GC)	--
VM4	enp0s3	NAT(default)	10.0.2.15/24	(VM default NW)	--
	enp0s8	Bridged Adapter	192.168.0.122/24	(Mgmt NW)	--

EPCのNF間のインターフェースの内、S1-U、S5u、Sxb、SGiについては、VirtualboxのNATネットワークで以下の通りに設定しました。

Network Name	Network CIDR	Note
N3	192.168.13.0/24	S1-U,S5u for EPC
N4	192.168.14.0/24	Sxb for EPC
N6	192.168.16.0/24	SGi for EPC

加入者情報は以下の通りです。(その他の項目はデフォルト値)
srsRAN 4GのUEの設定に合わせて、OPあるいはOPcを選択してください。

UE	IMSI	APN	OP/OPc
UE	001010000000100	internet	OPc

これらの情報の登録には、Open5GSが提供するWebUIを使用しました。なお、3GPP認証とキー生成機能(MILENAGE)の試験データとして、3GPP TS 35.208 "4.3 Test Sets"が公開されています。

PDNは以下の通りです。

PDN	APN	TUNnel interface of UE
10.45.0.0/16	internet	tun_srsue

eNodeBの主な情報は以下の通りです。

MCC	MNC	TAC	eNodeB ID	Cell ID	E-UTRAN Cell ID
001	01	1	0x19b	0x01	0x19b01

Open5GS EPC、eUPFとZMQ対応srsRAN 4G UE/RANの設定ファイルの変更点

Open5GS、eUPFとZMQ対応srsRAN 4G UE/RANのビルド方法は以下を参照して下さい。

Open5GS v2.7.0 (2023.12.04) - https://open5gs.org/open5gs/docs/guide/02-building-open5gs-from-sources/
eUPF v0.6.0 (2023.12.04) - https://qiita.com/s5uishida/items/8890fb67cb8d71da6080
srsRAN 4G (2023.11.23) - https://github.com/s5uishida/build_srsran_4g_zmq_disable_rf_plugins

Open5GS EPC C-Planeの設定ファイルの変更点

Open5GSでは、PFCPの接続先としてSGW-Uを選択するロジックに使用可能なパラメータは以下の通りです。PFCP(Packet Forwarding Control Protocol)とは、5GCでCU分離のために導入されたプロトコルです。

APN
TAC (Tracking Area Code)
e_CellID

なお、TACとe_CellIDはRANの設定項目です。話を簡単にするために、今回はAPNのみ使用しています。Open5GSでSGW-Uを選択するロジックについては、こちらを参照して下さい。

open5gs/install/etc/open5gs/mme.yaml

--- mme.yaml.orig       2023-12-05 03:49:16.000000000 +0900
+++ mme.yaml    2023-12-05 04:32:50.429030682 +0900
@@ -11,7 +11,7 @@
   freeDiameter: /root/open5gs/install/etc/freeDiameter/mme.conf
   s1ap:
     server:
-      - address: 127.0.0.2
+      - address: 192.168.0.111
   gtpc:
     server:
       - address: 127.0.0.2
@@ -26,14 +26,14 @@
         port: 9090
   gummei:
     - plmn_id:
-        mcc: 999
-        mnc: 70
+        mcc: 001
+        mnc: 01
       mme_gid: 2
       mme_code: 1
   tai:
     - plmn_id:
-        mcc: 999
-        mnc: 70
+        mcc: 001
+        mnc: 01
       tac: 1
   security:
     integrity_order : [ EIA2, EIA1, EIA0 ]

open5gs/install/etc/open5gs/sgwc.yaml

--- sgwc.yaml.orig      2023-12-05 03:49:16.000000000 +0900
+++ sgwc.yaml   2023-12-05 04:34:51.785067121 +0900
@@ -13,10 +13,11 @@
       - address: 127.0.0.3
   pfcp:
     server:
-      - address: 127.0.0.3
+      - address: 192.168.0.111
     client:
       sgwu:
-        - address: 127.0.0.6
+        - address: 192.168.0.112
+          apn: internet
 
 ################################################################################
 # GTP-C Server

open5gs/install/etc/open5gs/smf.yaml

--- smf.yaml.orig       2023-12-05 03:49:16.000000000 +0900
+++ smf.yaml    2023-12-05 04:45:38.004295776 +0900
@@ -8,39 +8,29 @@
 #    peer: 64
 
 smf:
-  sbi:
-    server:
-      - address: 127.0.0.4
-        port: 7777
-    client:
-#      nrf:
-#        - uri: http://127.0.0.10:7777
-      scp:
-        - uri: http://127.0.0.200:7777
   pfcp:
     server:
-      - address: 127.0.0.4
+      - address: 192.168.14.111
     client:
       upf:
-        - address: 127.0.0.7
+        - address: 192.168.14.151
+          dnn: internet
   gtpc:
     server:
       - address: 127.0.0.4
   gtpu:
     server:
-      - address: 127.0.0.4
+      - address: 192.168.14.111
   metrics:
     server:
       - address: 127.0.0.4
         port: 9090
   session:
     - subnet: 10.45.0.1/16
-    - subnet: 2001:db8:cafe::1/48
+      dnn: internet
   dns:
     - 8.8.8.8
     - 8.8.4.4
-    - 2001:4860:4860::8888
-    - 2001:4860:4860::8844
   mtu: 1400
 #  p-cscf:
 #    - 127.0.0.1

Open5GS EPC U-Planeの設定ファイルの変更点

open5gs/install/etc/open5gs/sgwu.yaml

--- sgwu.yaml.orig      2023-12-05 03:49:16.000000000 +0900
+++ sgwu.yaml   2023-12-05 04:55:03.425607103 +0900
@@ -10,13 +10,13 @@
 sgwu:
   pfcp:
     server:
-      - address: 127.0.0.6
+      - address: 192.168.0.112
     client:
 #      sgwc:    # SGW-U PFCP Client try to associate SGW-C PFCP Server
 #        - address: 127.0.0.3
   gtpu:
     server:
-      - address: 127.0.0.6
+      - address: 192.168.13.112
 
 ################################################################################
 # PFCP Server

eUPFの設定ファイルの変更点

元の設定ファイルは、こちらを参照して下さい。

eupf/config.yml
元ファイルからの変更はありません。

ZMQ対応srsRAN 4G UE/RANの設定ファイルの変更点

RAN(eNodeB)の設定ファイルの変更点

srsRAN_4G/build/srsenb/enb.conf

--- enb.conf.example    2023-12-02 14:18:34.000000000 +0900
+++ enb.conf    2023-12-02 15:06:53.357631639 +0900
@@ -22,9 +22,9 @@
 enb_id = 0x19B
 mcc = 001
 mnc = 01
-mme_addr = 127.0.1.100
-gtp_bind_addr = 127.0.1.1
-s1c_bind_addr = 127.0.1.1
+mme_addr = 192.168.0.111
+gtp_bind_addr = 192.168.13.121
+s1c_bind_addr = 192.168.0.121
 s1c_bind_port = 0
 n_prb = 50
 #tm = 4
@@ -80,8 +80,8 @@
 #time_adv_nsamples = auto
 
 # Example for ZMQ-based operation with TCP transport for I/Q samples
-#device_name = zmq
-#device_args = fail_on_disconnect=true,tx_port=tcp://*:2000,rx_port=tcp://localhost:2001,id=enb,base_srate=23.04e6
+device_name = zmq
+device_args = fail_on_disconnect=true,tx_port=tcp://192.168.0.121:2000,rx_port=tcp://192.168.0.122:2001,id=enb,base_srate=23.04e6
 
 #####################################################################
 # Packet capture configuration

srsRAN_4G/build/srsenb/rr.conf

--- rr.conf.example     2023-12-02 14:18:34.000000000 +0900
+++ rr.conf     2023-05-02 11:52:54.000000000 +0900
@@ -55,7 +55,7 @@
   {
     // rf_port = 0;
     cell_id = 0x01;
-    tac = 0x0007;
+    tac = 0x0001;
     pci = 1;
     // root_seq_idx = 204;
     dl_earfcn = 3350;

UE(IMSI-001010000000100)の設定ファイルの変更点

srsRAN_4G/build/srsue/ue.conf

--- ue.conf.example     2023-05-02 10:51:20.000000000 +0900
+++ ue.conf     2023-05-02 12:01:28.000000000 +0900
@@ -42,8 +42,8 @@
 #continuous_tx     = auto
 
 # Example for ZMQ-based operation with TCP transport for I/Q samples
-#device_name = zmq
-#device_args = tx_port=tcp://*:2001,rx_port=tcp://localhost:2000,id=ue,base_srate=23.04e6
+device_name = zmq
+device_args = tx_port=tcp://192.168.0.122:2001,rx_port=tcp://192.168.0.121:2000,id=ue,base_srate=23.04e6
 
 #####################################################################
 # EUTRA RAT configuration
@@ -139,9 +139,9 @@
 [usim]
 mode = soft
 algo = milenage
-opc  = 63BFA50EE6523365FF14C1F45F88737D
-k    = 00112233445566778899aabbccddeeff
-imsi = 001010123456780
+opc  = E8ED289DEBA952E4283B54E88E6183CA
+k    = 465B5CE8B199B49FAA5F0A2EE238A6BC
+imsi = 001010000000100
 imei = 353490069873319
 #reader =
 #pin  = 1234
@@ -180,8 +180,8 @@
 #                      Supported: 0 - NULL, 1 - Snow3G, 2 - AES, 3 - ZUC
 #####################################################################
 [nas]
-#apn = internetinternet
-#apn_protocol = ipv4
+apn = internet
+apn_protocol = ipv4
 #user = srsuser
 #pass = srspass
 #force_imsi_attach = false

Open5GS EPC、eUPFとZMQ対応srsRAN 4G UE/RANのネットワーク設定

eUPFとData Network Gatewayのネットワーク設定

こちらとこちらを参照して下さい。

Open5GS、eUPFとUERANSIMのビルド

Open5GS、eUPFとUERANSIMのビルド方法は以下を参考にして下さい。

Open5GS v2.7.0 (2023.12.04) - https://open5gs.org/open5gs/docs/guide/02-building-open5gs-from-sources/
eUPF v0.6.0 (2023.12.04) - https://qiita.com/s5uishida/items/8890fb67cb8d71da6080
srsRAN 4G (2023.11.23) - https://github.com/s5uishida/build_srsran_4g_zmq_disable_rf_plugins

Open5GS 5GC C-PlaneマシンにMongoDBをインストールして下さい。なお、MongoDBのデータベースを簡単に確認するには、MongoDB Compassが便利です。

Open5GS EPC、eUPFとZMQ対応srsRAN 4G UE/RANを実行

最初にeUPFを起動し、次にOpen5GS EPC U-Plane(SGW-U)とC-Plane、ZMQ対応srsRAN 4G RANとUEを起動します。

eUPFを実行

こちらを参考にして下さい。

Open5GS EPC U-Planeを実行

次に、Open5GS EPC U-Planeを起動します。

./install/bin/open5gs-sgwud &

Open5GS EPC C-Planeを実行

次に、Open5GS EPC C-Planeを起動します。

./install/bin/open5gs-mmed &
./install/bin/open5gs-sgwcd &
./install/bin/open5gs-smfd &
./install/bin/open5gs-hssd &
./install/bin/open5gs-pcrfd &

eUPFとOpen5GS SMF間のPFCP Associationログは以下の通りです。

2023/12/05 20:21:41 INF Got Association Setup Request from: 192.168.14.111. 

2023/12/05 20:21:41 INF 
Association Setup Request:
  Node ID: 192.168.14.111
  Recovery Time: 2023-12-05 20:21:40 +0900 JST

2023/12/05 20:21:41 INF Saving new association: &{ID:192.168.14.111 Addr:192.168.14.111 NextSessionID:1 NextSequenceID:1 Sessions:map[] HeartbeatRetries:0 cancelRetries:<nil>}

ZMQ対応srsRAN 4G RAN(eNodeB)を起動

ZMQ対応srsRAN 4G RAN(eNodeB)を起動し、Open5GS EPCに接続します。

# cd srsRAN_4G/build/srsenb
# ./src/srsenb enb.conf
---  Software Radio Systems LTE eNodeB  ---

Reading configuration file enb.conf...

Built in Release mode using commit eea87b1d8 on branch master.

Opening 1 channels in RF device=zmq with args=fail_on_disconnect=true,tx_port=tcp://192.168.0.121:2000,rx_port=tcp://192.168.0.122:2001,id=enb,base_srate=23.04e6
Supported RF device list: zmq file
CHx base_srate=23.04e6
CHx id=enb
Current sample rate is 1.92 MHz with a base rate of 23.04 MHz (x12 decimation)
CH0 rx_port=tcp://192.168.0.122:2001
CH0 tx_port=tcp://192.168.0.121:2000
CH0 fail_on_disconnect=true

==== eNodeB started ===
Type <t> to view trace
Current sample rate is 11.52 MHz with a base rate of 23.04 MHz (x2 decimation)
Current sample rate is 11.52 MHz with a base rate of 23.04 MHz (x2 decimation)
Setting frequency: DL=2680.0 Mhz, UL=2560.0 MHz for cc_idx=0 nof_prb=50

The Open5GS C-Plane log when executed is as follows.

12/05 20:22:10.188: [mme] INFO: eNB-S1 accepted[192.168.0.121]:47130 in s1_path module (../src/mme/s1ap-sctp.c:114)
12/05 20:22:10.188: [mme] INFO: eNB-S1 accepted[192.168.0.121] in master_sm module (../src/mme/mme-sm.c:108)
12/05 20:22:10.188: [mme] INFO: [Added] Number of eNBs is now 1 (../src/mme/mme-context.c:2808)
12/05 20:22:10.188: [mme] INFO: eNB-S1[192.168.0.121] max_num_of_ostreams : 30 (../src/mme/mme-sm.c:150)

Open5GS C-Planeのログは以下の通りです。

12/05 20:22:10.188: [mme] INFO: eNB-S1 accepted[192.168.0.121]:47130 in s1_path module (../src/mme/s1ap-sctp.c:114)
12/05 20:22:10.188: [mme] INFO: eNB-S1 accepted[192.168.0.121] in master_sm module (../src/mme/mme-sm.c:108)
12/05 20:22:10.188: [mme] INFO: [Added] Number of eNBs is now 1 (../src/mme/mme-context.c:2808)
12/05 20:22:10.188: [mme] INFO: eNB-S1[192.168.0.121] max_num_of_ostreams : 30 (../src/mme/mme-sm.c:150)

ZMQ対応srsRAN 4G UEを起動

ZMQ対応srsRAN 4G UEを起動し、eNodeB、Open5GS EPCに接続します。

# cd srsRAN_4G/build/srsue
# ./src/srsue ue.conf
Reading configuration file ue.conf...

Built in Release mode using commit eea87b1d8 on branch master.

Opening 1 channels in RF device=zmq with args=tx_port=tcp://192.168.0.122:2001,rx_port=tcp://192.168.0.121:2000,id=ue,base_srate=23.04e6
Supported RF device list: zmq file
CHx base_srate=23.04e6
CHx id=ue
Current sample rate is 1.92 MHz with a base rate of 23.04 MHz (x12 decimation)
CH0 rx_port=tcp://192.168.0.121:2000
CH0 tx_port=tcp://192.168.0.122:2001
Waiting PHY to initialize ... done!
Attaching UE...
Current sample rate is 1.92 MHz with a base rate of 23.04 MHz (x12 decimation)
Current sample rate is 1.92 MHz with a base rate of 23.04 MHz (x12 decimation)
.
Found Cell:  Mode=FDD, PCI=1, PRB=50, Ports=1, CP=Normal, CFO=-0.2 KHz
Current sample rate is 11.52 MHz with a base rate of 23.04 MHz (x2 decimation)
Current sample rate is 11.52 MHz with a base rate of 23.04 MHz (x2 decimation)
Found PLMN:  Id=00101, TAC=1
Random Access Transmission: seq=38, tti=341, ra-rnti=0x2
RRC Connected
Random Access Complete.     c-rnti=0x46, ta=0
Network attach successful. IP: 10.45.0.2
 nTp) ((t) 5/12/2023 11:23:6 TZ:99

Open5GS C-Planeのログは以下の通りです。

12/05 20:23:05.989: [mme] INFO: InitialUEMessage (../src/mme/s1ap-handler.c:406)
12/05 20:23:05.989: [mme] INFO: [Added] Number of eNB-UEs is now 1 (../src/mme/mme-context.c:4684)
12/05 20:23:05.989: [mme] INFO: Unknown UE by S_TMSI[G:2,C:1,M_TMSI:0xc0000672] (../src/mme/s1ap-handler.c:486)
12/05 20:23:05.989: [mme] INFO:     ENB_UE_S1AP_ID[1] MME_UE_S1AP_ID[1] TAC[1] CellID[0x19b01] (../src/mme/s1ap-handler.c:582)
12/05 20:23:05.989: [mme] INFO: Unknown UE by GUTI[G:2,C:1,M_TMSI:0xc0000672] (../src/mme/mme-context.c:3538)
12/05 20:23:05.989: [mme] INFO: [Added] Number of MME-UEs is now 1 (../src/mme/mme-context.c:3340)
12/05 20:23:05.989: [emm] INFO: [] Attach request (../src/mme/emm-sm.c:412)
12/05 20:23:05.990: [emm] INFO:     GUTI[G:2,C:1,M_TMSI:0xc0000672] IMSI[Unknown IMSI] (../src/mme/emm-handler.c:236)
12/05 20:23:06.028: [emm] INFO: Identity response (../src/mme/emm-sm.c:382)
12/05 20:23:06.028: [emm] INFO:     IMSI[001010000000100] (../src/mme/emm-handler.c:425)
12/05 20:23:06.144: [mme] INFO: [Added] Number of MME-Sessions is now 1 (../src/mme/mme-context.c:4698)
12/05 20:23:06.223: [sgwc] INFO: [Added] Number of SGWC-UEs is now 1 (../src/sgwc/context.c:239)
12/05 20:23:06.223: [sgwc] INFO: [Added] Number of SGWC-Sessions is now 1 (../src/sgwc/context.c:881)
12/05 20:23:06.223: [sgwc] INFO: UE IMSI[001010000000100] APN[internet] (../src/sgwc/s11-handler.c:237)
12/05 20:23:06.224: [gtp] INFO: gtp_connect() [127.0.0.4]:2123 (../lib/gtp/path.c:60)
12/05 20:23:06.225: [smf] INFO: [Added] Number of SMF-UEs is now 1 (../src/smf/context.c:1019)
12/05 20:23:06.225: [smf] INFO: [Added] Number of SMF-Sessions is now 1 (../src/smf/context.c:3068)
12/05 20:23:06.225: [smf] INFO: UE IMSI[001010000000100] APN[internet] IPv4[10.45.0.2] IPv6[] (../src/smf/s5c-handler.c:275)
12/05 20:23:06.230: [gtp] INFO: gtp_connect() [192.168.13.151]:2152 (../lib/gtp/path.c:60)
12/05 20:23:06.540: [emm] INFO: [001010000000100] Attach complete (../src/mme/emm-sm.c:1298)
12/05 20:23:06.540: [emm] INFO:     IMSI[001010000000100] (../src/mme/emm-handler.c:274)
12/05 20:23:06.540: [emm] INFO:     UTC [2023-12-05T11:23:06] Timezone[0]/DST[0] (../src/mme/emm-handler.c:281)
12/05 20:23:06.540: [emm] INFO:     LOCAL [2023-12-05T20:23:06] Timezone[32400]/DST[0] (../src/mme/emm-handler.c:285)

Open5GS U-Planeのログは以下の通りです。

12/05 20:23:06.187: [sgwu] INFO: UE F-SEID[UP:0x970 CP:0x2a7] (../src/sgwu/context.c:171)
12/05 20:23:06.187: [sgwu] INFO: [Added] Number of SGWU-Sessions is now 1 (../src/sgwu/context.c:176)
12/05 20:23:06.194: [gtp] INFO: gtp_connect() [192.168.13.151]:2152 (../lib/gtp/path.c:60)
12/05 20:23:06.504: [gtp] INFO: gtp_connect() [192.168.13.121]:2152 (../lib/gtp/path.c:60)

eUPFとOpen5GS SMF(PGW-C)のPFCP associationのステータスは次のように表示されます。

2023/12/05 20:23:06 INF Got Session Establishment Request from: 192.168.14.111.
2023/12/05 20:23:06 INF 
Session Establishment Request:
  CreatePDR ID: 1 
    FAR ID: 1 
    QER ID: 1 
    Source Interface: 1 
    UE IPv4 Address: 10.45.0.2 
  CreatePDR ID: 2 
    Outer Header Removal: 0 
    FAR ID: 2 
    QER ID: 1 
    Source Interface: 0 
    TEID: 0 
    Ipv4: <nil> 
    Ipv6: <nil> 
    UE IPv4 Address: 10.45.0.2 
  CreatePDR ID: 3 
    Outer Header Removal: 0 
    FAR ID: 1 
    Source Interface: 3 
    TEID: 0 
    Ipv4: <nil> 
    Ipv6: <nil> 
  CreatePDR ID: 4 
    Outer Header Removal: 0 
    FAR ID: 3 
    Source Interface: 0 
    TEID: 0 
    Ipv4: <nil> 
    Ipv6: <nil> 
    SDF Filter: permit out 58 from ff02::2/128 to assigned 
  CreateFAR ID: 1 
    Apply Action: [2 0] 
    Forwarding Parameters:
      Network Instance:internet 
      Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:60524 IPv4Address:192.168.13.112 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0} 
  CreateFAR ID: 2 
    Apply Action: [2 0] 
    Forwarding Parameters:
      Network Instance:internet 
  CreateFAR ID: 3 
    Apply Action: [2 0] 
    Forwarding Parameters:
      Network Instance:internet 
      Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:192.168.14.111 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0} 
  CreateQER ID: 1 
    Gate Status DL: 0 
    Gate Status UL: 0 
    Max Bitrate DL: 1000000 
    Max Bitrate UL: 1000000 
  CreateBAR ID: 1

2023/12/05 20:23:06 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:1 Teid:60524 RemoteIP:1879943360 LocalIP:2534254784 TransportLevelMarking:0}
2023/12/05 20:23:06 INF WARN: No OuterHeaderCreation
2023/12/05 20:23:06 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:2534254784 TransportLevelMarking:0}
2023/12/05 20:23:06 INF Saving FAR info to session: 3, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:1863231680 LocalIP:2534254784 TransportLevelMarking:0}
2023/12/05 20:23:06 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:1000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0}
2023/12/05 20:23:06 Matched groups: [permit out 58 from ff02::2/128 to assigned 58 ff02::2 128  assigned  ]
2023/12/05 20:23:06 INF Session Establishment Request from 192.168.14.111 accepted.

VM4(UE)でのip addr showは以下の通りです。TUNnelインターフェースtun_srsueは次のように作成され、IPアドレス10.45.0.2がUEに割り当てられたことが分かります。

5: tun_srsue: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 10.45.0.2/24 scope global tun_srsue
       valid_lft forever preferred_lft forever

google.comにpingする

UEのTUNインターフェースを指定して、pingしてみます。

DN=10.45.0.0/16を経由する場合

VM-DNでtcpdumpを実行し、パケットがSGiインターフェースenp0s9を通過することを確認します。

VM4(UE)でping google.com

# ping google.com -I tun_srsue -n
PING google.com (142.251.222.14) from 10.45.0.2 tun_srsue: 56(84) bytes of data.
64 bytes from 142.251.222.14: icmp_seq=1 ttl=61 time=101 ms
64 bytes from 142.251.222.14: icmp_seq=2 ttl=61 time=73.8 ms
64 bytes from 142.251.222.14: icmp_seq=3 ttl=61 time=58.0 ms

VM-DNでtcpdumpを実行

# tcpdump -i enp0s9 -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp0s9, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:25:50.944272 IP 10.45.0.2 > 142.251.222.14: ICMP echo request, id 3, seq 1, length 64
20:25:50.973107 IP 142.251.222.14 > 10.45.0.2: ICMP echo reply, id 3, seq 1, length 64
20:25:51.933226 IP 10.45.0.2 > 142.251.222.14: ICMP echo request, id 3, seq 2, length 64
20:25:51.955657 IP 142.251.222.14 > 10.45.0.2: ICMP echo reply, id 3, seq 2, length 64
20:25:52.919980 IP 10.45.0.2 > 142.251.222.14: ICMP echo request, id 3, seq 3, length 64
20:25:52.938727 IP 142.251.222.14 > 10.45.0.2: ICMP echo reply, id 3, seq 3, length 64

VM-UPで/sys/kernel/debug/tracing/trace_pipeを確認

# cat /sys/kernel/debug/tracing/trace_pipe
...
          <idle>-0       [000] d.s31  3184.066978: bpf_trace_printk: upf: gtp-u received
          <idle>-0       [000] d.s31  3184.066982: bpf_trace_printk: SDF: filter protocol: 4
          <idle>-0       [000] d.s31  3184.066985: bpf_trace_printk: SDF: filter source ip: 0.0.0.2, destination ip: 0.0.0.0
          <idle>-0       [000] d.s31  3184.066987: bpf_trace_printk: SDF: filter source ip mask: 255.255.255.255, destination ip mask: 0.0.0.0
          <idle>-0       [000] d.s31  3184.066988: bpf_trace_printk: SDF: filter source port lower bound: 0, source port upper bound: 65535
          <idle>-0       [000] d.s31  3184.066989: bpf_trace_printk: SDF: filter destination port lower bound: 0, destination port upper bound: 65535
          <idle>-0       [000] d.s31  3184.066989: bpf_trace_printk: SDF: packet protocol: 0
          <idle>-0       [000] d.s31  3184.066990: bpf_trace_printk: SDF: packet source ip: 10.45.0.2, destination ip: 142.251.222.14
          <idle>-0       [000] d.s31  3184.066991: bpf_trace_printk: SDF: packet source port: 0, destination port: 0
          <idle>-0       [000] d.s31  3184.066992: bpf_trace_printk: upf: sdf filter doesn't match teid:1
          <idle>-0       [000] d.s31  3184.066993: bpf_trace_printk: upf: far:1 action:2 outer_header_creation:0
          <idle>-0       [000] d.s31  3184.066994: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1000000000
          <idle>-0       [000] d.s31  3184.066995: bpf_trace_printk: upf: session for teid:1 far:1 outer_header_removal:0
          <idle>-0       [000] d.s31  3184.067003: bpf_trace_printk: upf: bpf_fib_lookup 10.45.0.2 -> 142.251.222.14: nexthop: 192.168.16.152
          <idle>-0       [000] d.s31  3184.096132: bpf_trace_printk: upf: downlink session for ip:10.45.0.2  far:0 action:2
          <idle>-0       [000] d.s31  3184.096135: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1000000000
          <idle>-0       [000] d.s31  3184.096136: bpf_trace_printk: upf: use mapping 10.45.0.2 -> TEID:60524
          <idle>-0       [000] d.s31  3184.096138: bpf_trace_printk: upf: send gtp pdu 192.168.13.151 -> 192.168.13.112
          <idle>-0       [000] d.s31  3184.096146: bpf_trace_printk: upf: bpf_fib_lookup 192.168.13.151 -> 192.168.13.112: nexthop: 192.168.13.112
          <idle>-0       [000] d.s31  3185.055908: bpf_trace_printk: upf: gtp-u received
          <idle>-0       [000] d.s31  3185.055912: bpf_trace_printk: SDF: filter protocol: 4
          <idle>-0       [000] d.s31  3185.055915: bpf_trace_printk: SDF: filter source ip: 0.0.0.2, destination ip: 0.0.0.0
          <idle>-0       [000] d.s31  3185.055916: bpf_trace_printk: SDF: filter source ip mask: 255.255.255.255, destination ip mask: 0.0.0.0
          <idle>-0       [000] d.s31  3185.055917: bpf_trace_printk: SDF: filter source port lower bound: 0, source port upper bound: 65535
          <idle>-0       [000] d.s31  3185.055918: bpf_trace_printk: SDF: filter destination port lower bound: 0, destination port upper bound: 65535
          <idle>-0       [000] d.s31  3185.055918: bpf_trace_printk: SDF: packet protocol: 0
          <idle>-0       [000] d.s31  3185.055938: bpf_trace_printk: SDF: packet source ip: 10.45.0.2, destination ip: 142.251.222.14
          <idle>-0       [000] d.s31  3185.055939: bpf_trace_printk: SDF: packet source port: 0, destination port: 0
          <idle>-0       [000] d.s31  3185.055940: bpf_trace_printk: upf: sdf filter doesn't match teid:1
          <idle>-0       [000] d.s31  3185.055941: bpf_trace_printk: upf: far:1 action:2 outer_header_creation:0
          <idle>-0       [000] d.s31  3185.055942: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1000000000
          <idle>-0       [000] d.s31  3185.055943: bpf_trace_printk: upf: session for teid:1 far:1 outer_header_removal:0
          <idle>-0       [000] d.s31  3185.055950: bpf_trace_printk: upf: bpf_fib_lookup 10.45.0.2 -> 142.251.222.14: nexthop: 192.168.16.152
          <idle>-0       [000] d.s31  3185.078676: bpf_trace_printk: upf: downlink session for ip:10.45.0.2  far:0 action:2
          <idle>-0       [000] d.s31  3185.078680: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1000000000
          <idle>-0       [000] d.s31  3185.078681: bpf_trace_printk: upf: use mapping 10.45.0.2 -> TEID:60524
          <idle>-0       [000] d.s31  3185.078683: bpf_trace_printk: upf: send gtp pdu 192.168.13.151 -> 192.168.13.112
          <idle>-0       [000] d.s31  3185.078691: bpf_trace_printk: upf: bpf_fib_lookup 192.168.13.151 -> 192.168.13.112: nexthop: 192.168.13.112
          <idle>-0       [000] d.s31  3186.042681: bpf_trace_printk: upf: gtp-u received
          <idle>-0       [000] d.s31  3186.042685: bpf_trace_printk: SDF: filter protocol: 4
          <idle>-0       [000] d.s31  3186.042688: bpf_trace_printk: SDF: filter source ip: 0.0.0.2, destination ip: 0.0.0.0
          <idle>-0       [000] d.s31  3186.042690: bpf_trace_printk: SDF: filter source ip mask: 255.255.255.255, destination ip mask: 0.0.0.0
          <idle>-0       [000] d.s31  3186.042691: bpf_trace_printk: SDF: filter source port lower bound: 0, source port upper bound: 65535
          <idle>-0       [000] d.s31  3186.042692: bpf_trace_printk: SDF: filter destination port lower bound: 0, destination port upper bound: 65535
          <idle>-0       [000] d.s31  3186.042692: bpf_trace_printk: SDF: packet protocol: 0
          <idle>-0       [000] d.s31  3186.042693: bpf_trace_printk: SDF: packet source ip: 10.45.0.2, destination ip: 142.251.222.14
          <idle>-0       [000] d.s31  3186.042694: bpf_trace_printk: SDF: packet source port: 0, destination port: 0
          <idle>-0       [000] d.s31  3186.042695: bpf_trace_printk: upf: sdf filter doesn't match teid:1
          <idle>-0       [000] d.s31  3186.042696: bpf_trace_printk: upf: far:1 action:2 outer_header_creation:0
          <idle>-0       [000] d.s31  3186.042697: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1000000000
          <idle>-0       [000] d.s31  3186.042698: bpf_trace_printk: upf: session for teid:1 far:1 outer_header_removal:0
          <idle>-0       [000] d.s31  3186.042705: bpf_trace_printk: upf: bpf_fib_lookup 10.45.0.2 -> 142.251.222.14: nexthop: 192.168.16.152
          <idle>-0       [000] d.s31  3186.061671: bpf_trace_printk: upf: downlink session for ip:10.45.0.2  far:0 action:2
          <idle>-0       [000] d.s31  3186.061674: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1000000000
          <idle>-0       [000] d.s31  3186.061676: bpf_trace_printk: upf: use mapping 10.45.0.2 -> TEID:60524
          <idle>-0       [000] d.s31  3186.061677: bpf_trace_printk: upf: send gtp pdu 192.168.13.151 -> 192.168.13.112
          <idle>-0       [000] d.s31  3186.061684: bpf_trace_printk: upf: bpf_fib_lookup 192.168.13.151 -> 192.168.13.112: nexthop: 192.168.13.112
...

ping以外にも、以下のようにcurlでTUNnelインターフェースを指定してWebにアクセスしてみることもできます。

curl google.com on VM4 (UE)

# curl --interface tun_srsue google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

VM-DNでtcpdumpを実行

20:27:46.955340 IP 10.45.0.2.54964 > 172.217.161.78.80: Flags [S], seq 718929327, win 64240, options [mss 1460,sackOK,TS val 1478275073 ecr 0,nop,wscale 7], length 0
20:27:46.971398 IP 172.217.161.78.80 > 10.45.0.2.54964: Flags [S.], seq 192001, ack 718929328, win 65535, options [mss 1460], length 0
20:27:47.095296 IP 10.45.0.2.54964 > 172.217.161.78.80: Flags [.], ack 1, win 64240, length 0
20:27:47.095296 IP 10.45.0.2.54964 > 172.217.161.78.80: Flags [P.], seq 1:75, ack 1, win 64240, length 74: HTTP: GET / HTTP/1.1
20:27:47.095530 IP 172.217.161.78.80 > 10.45.0.2.54964: Flags [.], ack 75, win 65535, length 0
20:27:47.173318 IP 172.217.161.78.80 > 10.45.0.2.54964: Flags [P.], seq 1:774, ack 75, win 65535, length 773: HTTP: HTTP/1.1 301 Moved Permanently
20:27:47.234283 IP 10.45.0.2.54964 > 172.217.161.78.80: Flags [.], ack 774, win 63467, length 0
20:27:47.234283 IP 10.45.0.2.54964 > 172.217.161.78.80: Flags [F.], seq 75, ack 774, win 63467, length 0
20:27:47.234527 IP 172.217.161.78.80 > 10.45.0.2.54964: Flags [.], ack 76, win 65535, length 0
20:27:47.264013 IP 172.217.161.78.80 > 10.45.0.2.54964: Flags [F.], seq 774, ack 76, win 65535, length 0
20:27:47.332814 IP 10.45.0.2.54964 > 172.217.161.78.80: Flags [.], ack 775, win 63467, length 0

以上で、eBPF/XDP対応UPFのeUPFのPDNに接続し、PDNに任意のパケットを流せるようになり、プライベートLTEを想定したEPCのシミュレーション携帯網として、MECを試す環境を構築できました。

最後に

eBPF/XDP対応UPFのeUPFは、U-Planeのデータネットワークを高速化する技術と期待しています。本記事では、Virtualbox VM上の設定例を書いてみましたが、高性能な物理NICを搭載したサーバを設定する際のご参考になれれば幸いです。

最後に、元記事はGithubに書いたものです。

主な変更履歴

[2024.02.22] 初版。