本記事では、ローカル5G SAを想定して、Open5GS 5GCとeBPF/XDP対応UPFのeUPFを連携動作させるための簡単な構成について説明します。 eUPFのインストールと設定については、こちらを参照してください。

Open5GS 5GCのシミュレーション携帯網の概要

今回、Open5GS 5GCのC-Plane、eUPF(eBPF/XDP UPF)、およびData Network Gatewayの簡単な構成について説明します。この構成はVirtualbox VMを使用して実装されることに注意して下さい。
このシミュレーション環境は以下の条件を満たす最小構成になります。

1つのUPFとData Network Gateway
1つのUEと1つのDNN

構築したシミュレーション環境は以下の通りです。

5GC、eUPFとUE/RANに使用しているOSSは以下の通りです。

5GC - Open5GS v2.7.0 (2023.12.04) - https://github.com/open5gs/open5gs
eBPF/XDP UPF - eUPF v0.6.0 (2023.12.04) - https://github.com/edgecomllc/eupf
UE / RAN - UERANSIM v3.2.6 (2023.11.27) - https://github.com/aligungr/UERANSIM

各VMは以下の通りです。

VM	SW & Role	IP address	OS	CPU (Min)	Memory (Min)	HDD (Min)
VM1	Open5GS 5GC C-Plane	192.168.0.111/24	Ubuntu 22.04	1	2GB	20GB
VM-UP	eUPF U-Plane	192.168.0.151/24	Ubuntu 22.04	1	2GB	20GB
VM-DN	Data Network Gateway	192.168.0.152/24	Ubuntu 22.04	1	1GB	10GB
VM2	UERANSIM RAN (gNodeB)	192.168.0.131/24	Ubuntu 22.04	1	1GB	10GB
VM3	UERANSIM UE	192.168.0.132/24	Ubuntu 22.04	1	1GB	10GB

各VMのネットワークインターフェースは以下の通りです。

VM	Device	Network Adapter	IP address	Interface	XDP
VM1	enp0s3	NAT(default)	10.0.2.15/24	(VM default NW)	--
	enp0s8	Bridged Adapter	192.168.0.111/24	(Mgmt NW)	--
	enp0s9	NAT Network	192.168.14.111/24	N4	--
VM-UP	~~enp0s3~~	~~NAT(default)~~	~~10.0.2.15/24~~	~~(VM default NW)~~ down	--
	enp0s8	Bridged Adapter	192.168.0.151/24	(Mgmt NW)	--
	enp0s9	NAT Network	192.168.13.151/24	N3	x
	enp0s10	NAT Network	192.168.14.151/24	N4	--
	enp0s16	NAT Network	192.168.16.151/24	N6	x
VM-DN	enp0s3	NAT(default)	10.0.2.15/24	(VM default NW)	--
	enp0s8	Bridged Adapter	192.168.0.152/24	(Mgmt NW)	--
	enp0s9	NAT Network	192.168.16.152/24	N6, default GW for VM-UP	--
VM2	enp0s3	NAT(default)	10.0.2.15/24	(VM default NW)	--
	enp0s8	Bridged Adapter	192.168.0.131/24	(Mgmt NW)	--
	enp0s9	NAT Network	192.168.13.131/24	N3	--
VM3	enp0s3	NAT(default)	10.0.2.15/24	(VM default NW)	--
	enp0s8	Bridged Adapter	192.168.0.132/24	(Mgmt NW)	--

5GのNF間のインターフェースの内、N3、N4、N6については、VirtualboxのNATネットワークで以下の通りに設定しました。

Network Name	Network CIDR
N3	192.168.13.0/24
N4	192.168.14.0/24
N6	192.168.16.0/24

network instanceをinternetに設定しています。

Network Instance
internet

加入者情報は以下の通りです。(その他の項目はデフォルト値)
UERANSIMのUEの設定に合わせて、OPあるいはOPcを選択してください。

UE	IMSI	DNN	OP/OPc
UE	001010000000000	internet	OPc

これらの情報の登録には、Open5GSが提供するWebUIを使用しました。なお、3GPP認証とキー生成機能(MILENAGE)の試験データとして、3GPP TS 35.208 "4.3 Test Sets"が公開されています。

DNは以下の通りです。

DN	DNN	TUNnel interface of UE
10.45.0.0/16	internet	uesimtun0

Open5GS 5GC、eUPFとUERANSIM UE/RANの設定ファイルの変更点

Open5GS、eUPFとUERANSIMのビルド方法は以下を参照して下さい。

Open5GS v2.7.0 (2023.12.04) - https://open5gs.org/open5gs/docs/guide/02-building-open5gs-from-sources/
eUPF v0.6.0 (2023.12.04) - https://qiita.com/s5uishida/items/8890fb67cb8d71da6080
UERANSIM v3.2.6 (2023.11.27) - https://github.com/aligungr/UERANSIM/wiki/Installation

Open5GS 5GC C-Planeの設定ファイルの変更点

Open5GSでは、PFCPの接続先としてUPFを選択するロジックに使用可能なパラメータは以下の通りです。PFCP(Packet Forwarding Control Protocol)とは、5GCでCU分離のために導入されたプロトコルです。

DNN
TAC (Tracking Area Code)
nr_CellID

なお、TACとnr_CellIDはRANの設定項目です。話を簡単にするために、今回はDNNのみ使用しています。Open5GSでUPFを選択するロジックについては、こちらを参照して下さい。

open5gs/install/etc/open5gs/amf.yaml

--- amf.yaml.orig       2023-12-05 03:49:16.000000000 +0900
+++ amf.yaml    2023-12-05 04:31:14.391992533 +0900
@@ -19,27 +19,27 @@
         - uri: http://127.0.0.200:7777
   ngap:
     server:
-      - address: 127.0.0.5
+      - address: 192.168.0.111
   metrics:
     server:
       - address: 127.0.0.5
         port: 9090
   guami:
     - plmn_id:
-        mcc: 999
-        mnc: 70
+        mcc: 001
+        mnc: 01
       amf_id:
         region: 2
         set: 1
   tai:
     - plmn_id:
-        mcc: 999
-        mnc: 70
+        mcc: 001
+        mnc: 01
       tac: 1
   plmn_support:
     - plmn_id:
-        mcc: 999
-        mnc: 70
+        mcc: 001
+        mnc: 01
       s_nssai:
         - sst: 1
   security:

open5gs/install/etc/open5gs/nrf.yaml

--- nrf.yaml.orig       2023-12-05 03:49:16.000000000 +0900
+++ nrf.yaml    2023-12-05 19:25:32.489805559 +0900
@@ -10,8 +10,8 @@
 nrf:
   serving:  # 5G roaming requires PLMN in NRF
     - plmn_id:
-        mcc: 999
-        mnc: 70
+        mcc: 001
+        mnc: 01
   sbi:
     server:
       - address: 127.0.0.10

open5gs/install/etc/open5gs/smf.yaml

--- smf.yaml.orig       2023-12-05 03:49:16.000000000 +0900
+++ smf.yaml    2023-12-07 05:34:06.510124360 +0900
@@ -19,28 +19,27 @@
         - uri: http://127.0.0.200:7777
   pfcp:
     server:
-      - address: 127.0.0.4
+      - address: 192.168.14.111
     client:
       upf:
-        - address: 127.0.0.7
+        - address: 192.168.14.151
+          dnn: internet
   gtpc:
     server:
       - address: 127.0.0.4
   gtpu:
     server:
-      - address: 127.0.0.4
+      - address: 192.168.14.111
   metrics:
     server:
       - address: 127.0.0.4
         port: 9090
   session:
     - subnet: 10.45.0.1/16
-    - subnet: 2001:db8:cafe::1/48
+      dnn: internet
   dns:
     - 8.8.8.8
     - 8.8.4.4
-    - 2001:4860:4860::8888
-    - 2001:4860:4860::8844
   mtu: 1400
 #  p-cscf:
 #    - 127.0.0.1

eUPFの設定ファイルの変更点

元の設定ファイルは、こちらを参照して下さい。

eupf/config.yml
元ファイルからの変更はありません。

UERANSIM UE/RANの設定ファイルの変更点

RANの設定ファイルの変更点

UERANSIM/config/open5gs-gnb.yaml

--- open5gs-gnb.yaml.orig       2022-07-03 13:06:44.000000000 +0900
+++ open5gs-gnb.yaml    2023-11-03 01:34:38.384618638 +0900
@@ -1,17 +1,17 @@
-mcc: '999'          # Mobile Country Code value
-mnc: '70'           # Mobile Network Code value (2 or 3 digits)
+mcc: '001'          # Mobile Country Code value
+mnc: '01'           # Mobile Network Code value (2 or 3 digits)
 
 nci: '0x000000010'  # NR Cell Identity (36-bit)
 idLength: 32        # NR gNB ID length in bits [22...32]
 tac: 1              # Tracking Area Code
 
-linkIp: 127.0.0.1   # gNB's local IP address for Radio Link Simulation (Usually same with local IP)
-ngapIp: 127.0.0.1   # gNB's local IP address for N2 Interface (Usually same with local IP)
-gtpIp: 127.0.0.1    # gNB's local IP address for N3 Interface (Usually same with local IP)
+linkIp: 192.168.0.131   # gNB's local IP address for Radio Link Simulation (Usually same with local IP)
+ngapIp: 192.168.0.131   # gNB's local IP address for N2 Interface (Usually same with local IP)
+gtpIp: 192.168.13.131    # gNB's local IP address for N3 Interface (Usually same with local IP)
 
 # List of AMF address information
 amfConfigs:
-  - address: 127.0.0.5
+  - address: 192.168.0.111
     port: 38412
 
 # List of supported S-NSSAIs by this gNB

UE(IMSI-001010000000000)の設定ファイルの変更点

UERANSIM/config/open5gs-ue.yaml

--- open5gs-ue.yaml.orig        2023-05-10 19:00:38.000000000 +0900
+++ open5gs-ue.yaml     2023-06-15 21:42:14.363706123 +0900
@@ -1,9 +1,9 @@
 # IMSI number of the UE. IMSI = [MCC|MNC|MSISDN] (In total 15 digits)
-supi: 'imsi-999700000000001'
+supi: 'imsi-001010000000000'
 # Mobile Country Code value of HPLMN
-mcc: '999'
+mcc: '001'
 # Mobile Network Code value of HPLMN (2 or 3 digits)
-mnc: '70'
+mnc: '01'
 # SUCI Protection Scheme : 0 for Null-scheme, 1 for Profile A and 2 for Profile B
 protectionScheme: 0
 # Home Network Public Key for protecting with SUCI Profile A
@@ -28,7 +28,7 @@
 
 # List of gNB IP addresses for Radio Link Simulation
 gnbSearchList:
-  - 127.0.0.1
+  - 192.168.0.131
 
 # UAC Access Identities Configuration
 uacAic:

Open5GS 5GC、eUPFとUERANSIM UE/RANのネットワーク設定

eUPFとData Network Gatewayのネットワーク設定

こちらとこちらを参照して下さい。

Open5GS、eUPFとUERANSIMのビルド

Open5GS、eUPFとUERANSIMのビルド方法は以下を参考にして下さい。

Open5GS v2.7.0 (2023.12.04) - https://open5gs.org/open5gs/docs/guide/02-building-open5gs-from-sources/
eUPF v0.6.0 (2023.12.04) - https://qiita.com/s5uishida/items/8890fb67cb8d71da6080
UERANSIM v3.2.6 (2023.11.27) - https://github.com/aligungr/UERANSIM/wiki/Installation

Open5GS 5GC C-PlaneマシンにMongoDBをインストールして下さい。なお、MongoDBのデータベースを簡単に確認するには、MongoDB Compassが便利です。

Open5GS 5GC、eUPFとUERANSIM UE/RANを実行

最初にeUPFを起動し、Open5GS 5GC、UERANSIM(UEとRANのシミュレータ)を起動します。

eUPFを実行

こちらを参考にして下さい。

Open5GS 5GC C-Planeを実行

./install/bin/open5gs-nrfd &
sleep 2
./install/bin/open5gs-scpd &
sleep 2
./install/bin/open5gs-amfd &
sleep 2
./install/bin/open5gs-smfd &
./install/bin/open5gs-ausfd &
./install/bin/open5gs-udmd &
./install/bin/open5gs-udrd &
./install/bin/open5gs-pcfd &
./install/bin/open5gs-nssfd &
./install/bin/open5gs-bsfd &

eUPFとOpen5GS SMF間のPFCP Associationログは以下の通りです。

2023/12/05 19:49:27 INF Got Association Setup Request from: 192.168.14.111. 

2023/12/05 19:49:27 INF 
Association Setup Request:
  Node ID: 192.168.14.111
  Recovery Time: 2023-12-05 19:49:27 +0900 JST

2023/12/05 19:49:27 INF Saving new association: &{ID:192.168.14.111 Addr:192.168.14.111 NextSessionID:1 NextSequenceID:1 Sessions:map[] HeartbeatRetries:0 cancelRetries:<nil>}

UERANSIMを実行

ここでは、UE(IMSI-001010000000000)とRANを実行する場合を説明します。まず、gNodeBと5GCの間でNG-Setupし、5GCにUEを登録し、PDUセッションを確立します。

UERANSIMの使い方は以下を参照して下さい。

https://github.com/aligungr/UERANSIM/wiki/Usage

gNBを起動

次のようにgNBを起動します。

# ./nr-gnb -c ../config/open5gs-gnb.yaml
UERANSIM v3.2.6
[2023-12-05 19:49:57.942] [sctp] [info] Trying to establish SCTP connection... (192.168.0.111:38412)
[2023-12-05 19:49:57.944] [sctp] [info] SCTP connection established (192.168.0.111:38412)
[2023-12-05 19:49:57.944] [sctp] [debug] SCTP association setup ascId[4]
[2023-12-05 19:49:57.945] [ngap] [debug] Sending NG Setup Request
[2023-12-05 19:49:57.951] [ngap] [debug] NG Setup Response received
[2023-12-05 19:49:57.951] [ngap] [info] NG Setup procedure is successful

Open5GS C-Planeのログは以下の通りです。

12/05 19:49:57.967: [amf] INFO: gNB-N2 accepted[192.168.0.131]:49164 in ng-path module (../src/amf/ngap-sctp.c:113)
12/05 19:49:57.968: [amf] INFO: gNB-N2 accepted[192.168.0.131] in master_sm module (../src/amf/amf-sm.c:741)
12/05 19:49:57.973: [amf] INFO: [Added] Number of gNBs is now 1 (../src/amf/context.c:1231)
12/05 19:49:57.973: [amf] INFO: gNB-N2[192.168.0.131] max_num_of_ostreams : 10 (../src/amf/amf-sm.c:780)

UEを起動

次のようにUEを起動します。これにより、5GCにUEが登録され、PDUセッションが確立します。

# ./nr-ue -c ../config/open5gs-ue.yaml
UERANSIM v3.2.6
[2023-12-05 19:50:28.338] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2023-12-05 19:50:28.338] [rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2023-12-05 19:50:28.339] [nas] [info] Selected plmn[001/01]
[2023-12-05 19:50:28.339] [rrc] [info] Selected cell plmn[001/01] tac[1] category[SUITABLE]
[2023-12-05 19:50:28.339] [nas] [info] UE switches to state [MM-DEREGISTERED/PS]
[2023-12-05 19:50:28.340] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE]
[2023-12-05 19:50:28.340] [nas] [debug] Initial registration required due to [MM-DEREG-NORMAL-SERVICE]
[2023-12-05 19:50:28.343] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2023-12-05 19:50:28.343] [nas] [debug] Sending Initial Registration
[2023-12-05 19:50:28.343] [rrc] [debug] Sending RRC Setup Request
[2023-12-05 19:50:28.344] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]
[2023-12-05 19:50:28.344] [rrc] [info] RRC connection established
[2023-12-05 19:50:28.344] [rrc] [info] UE switches to state [RRC-CONNECTED]
[2023-12-05 19:50:28.344] [nas] [info] UE switches to state [CM-CONNECTED]
[2023-12-05 19:50:28.353] [nas] [debug] Authentication Request received
[2023-12-05 19:50:28.353] [nas] [debug] Received SQN [000000001E21]
[2023-12-05 19:50:28.353] [nas] [debug] SQN-MS [000000000000]
[2023-12-05 19:50:28.358] [nas] [debug] Security Mode Command received
[2023-12-05 19:50:28.359] [nas] [debug] Selected integrity[2] ciphering[0]
[2023-12-05 19:50:28.373] [nas] [debug] Registration accept received
[2023-12-05 19:50:28.373] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2023-12-05 19:50:28.373] [nas] [debug] Sending Registration Complete
[2023-12-05 19:50:28.374] [nas] [info] Initial Registration is successful
[2023-12-05 19:50:28.374] [nas] [debug] Sending PDU Session Establishment Request
[2023-12-05 19:50:28.374] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2023-12-05 19:50:28.578] [nas] [debug] Configuration Update Command received
[2023-12-05 19:50:28.602] [nas] [debug] PDU Session Establishment Accept received
[2023-12-05 19:50:28.606] [nas] [info] PDU Session establishment is successful PSI[1]
[2023-12-05 19:50:28.634] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.45.0.2] is up.

Open5GS C-Planeのログは以下の通りです。

12/05 19:50:28.366: [amf] INFO: InitialUEMessage (../src/amf/ngap-handler.c:401)
12/05 19:50:28.366: [amf] INFO: [Added] Number of gNB-UEs is now 1 (../src/amf/context.c:2550)
12/05 19:50:28.366: [amf] INFO:     RAN_UE_NGAP_ID[1] AMF_UE_NGAP_ID[1] TAC[1] CellID[0x10] (../src/amf/ngap-handler.c:562)
12/05 19:50:28.366: [amf] INFO: [suci-0-001-01-0000-0-0-0000000000] Unknown UE by SUCI (../src/amf/context.c:1835)
12/05 19:50:28.367: [amf] INFO: [Added] Number of AMF-UEs is now 1 (../src/amf/context.c:1616)
12/05 19:50:28.367: [gmm] INFO: Registration request (../src/amf/gmm-sm.c:1165)
12/05 19:50:28.367: [gmm] INFO: [suci-0-001-01-0000-0-0-0000000000]    SUCI (../src/amf/gmm-handler.c:166)
12/05 19:50:28.369: [sbi] WARNING: [UDM] (NRF-discover) NF has already been added [f62f2dc8-935b-41ee-a279-fd82055e6d3f:1] (../lib/sbi/nnrf-handler.c:1057)
12/05 19:50:28.369: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.12:80] (../lib/sbi/context.c:2174)
12/05 19:50:28.369: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.12:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.369: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.12:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.369: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.12:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.370: [sbi] INFO: [UDM] (NF-discover) NF Profile updated [f62f2dc8-935b-41ee-a279-fd82055e6d3f:1] (../lib/sbi/nnrf-handler.c:1095)
12/05 19:50:28.372: [sbi] INFO: [UDM] (SCP-discover) NF registered [f62f2dc8-935b-41ee-a279-fd82055e6d3f:1] (../lib/sbi/path.c:211)
12/05 19:50:28.390: [sbi] WARNING: [UDR] (NRF-discover) NF has already been added [f636378a-935b-41ee-9762-edaaac822635:1] (../lib/sbi/nnrf-handler.c:1057)
12/05 19:50:28.390: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.20:80] (../lib/sbi/context.c:2174)
12/05 19:50:28.390: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.20:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.391: [sbi] INFO: [UDR] (NF-discover) NF Profile updated [f636378a-935b-41ee-9762-edaaac822635:1] (../lib/sbi/nnrf-handler.c:1095)
12/05 19:50:28.392: [sbi] INFO: [UDR] (SCP-discover) NF registered [f636378a-935b-41ee-9762-edaaac822635:1] (../lib/sbi/path.c:211)
12/05 19:50:28.596: [gmm] INFO: [imsi-001010000000000] Registration complete (../src/amf/gmm-sm.c:2146)
12/05 19:50:28.596: [amf] INFO: [imsi-001010000000000] Configuration update command (../src/amf/nas-path.c:612)
12/05 19:50:28.597: [gmm] INFO:     UTC [2023-12-05T10:50:28] Timezone[0]/DST[0] (../src/amf/gmm-build.c:559)
12/05 19:50:28.597: [gmm] INFO:     LOCAL [2023-12-05T19:50:28] Timezone[32400]/DST[0] (../src/amf/gmm-build.c:564)
12/05 19:50:28.598: [amf] INFO: [Added] Number of AMF-Sessions is now 1 (../src/amf/context.c:2571)
12/05 19:50:28.598: [gmm] INFO: UE SUPI[imsi-001010000000000] DNN[internet] S_NSSAI[SST:1 SD:0xffffff] smContextRef [NULL] (../src/amf/gmm-handler.c:1241)
12/05 19:50:28.599: [gmm] INFO: SMF Instance [f643735a-935b-41ee-bd56-85c2d33d8423] (../src/amf/gmm-handler.c:1280)
12/05 19:50:28.601: [smf] INFO: [Added] Number of SMF-UEs is now 1 (../src/smf/context.c:1019)
12/05 19:50:28.602: [smf] INFO: [Added] Number of SMF-Sessions is now 1 (../src/smf/context.c:3068)
12/05 19:50:28.604: [sbi] WARNING: [UDM] (NRF-discover) NF has already been added [f62f2dc8-935b-41ee-a279-fd82055e6d3f:1] (../lib/sbi/nnrf-handler.c:1057)
12/05 19:50:28.604: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.12:80] (../lib/sbi/context.c:2174)
12/05 19:50:28.604: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.12:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.605: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.12:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.605: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.12:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.605: [sbi] INFO: [UDM] (NF-discover) NF Profile updated [f62f2dc8-935b-41ee-a279-fd82055e6d3f:1] (../lib/sbi/nnrf-handler.c:1095)
12/05 19:50:28.608: [sbi] INFO: [UDM] (SCP-discover) NF registered [f62f2dc8-935b-41ee-a279-fd82055e6d3f:1] (../lib/sbi/path.c:211)
12/05 19:50:28.609: [sbi] WARNING: [PCF] (NRF-discover) NF has already been added [f636fbca-935b-41ee-bcac-9dd7f7aab382:1] (../lib/sbi/nnrf-handler.c:1057)
12/05 19:50:28.610: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.13:80] (../lib/sbi/context.c:2174)
12/05 19:50:28.610: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.13:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.610: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.13:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.610: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.13:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.610: [sbi] INFO: [PCF] (NF-discover) NF Profile updated [f636fbca-935b-41ee-bcac-9dd7f7aab382:1] (../lib/sbi/nnrf-handler.c:1095)
12/05 19:50:28.612: [sbi] WARNING: [UDR] (NRF-discover) NF has already been added [f636378a-935b-41ee-9762-edaaac822635:1] (../lib/sbi/nnrf-handler.c:1057)
12/05 19:50:28.612: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.20:80] (../lib/sbi/context.c:2174)
12/05 19:50:28.612: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.20:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.612: [sbi] INFO: [UDR] (NF-discover) NF Profile updated [f636378a-935b-41ee-9762-edaaac822635:1] (../lib/sbi/nnrf-handler.c:1095)
12/05 19:50:28.614: [sbi] WARNING: [UDR] (SCP-discover) NF has already been added [f636378a-935b-41ee-9762-edaaac822635:2] (../lib/sbi/path.c:216)
12/05 19:50:28.615: [sbi] WARNING: [BSF] (NRF-discover) NF has already been added [f62fc472-935b-41ee-bc56-85d1c55fd3cc:1] (../lib/sbi/nnrf-handler.c:1057)
12/05 19:50:28.615: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.15:80] (../lib/sbi/context.c:2174)
12/05 19:50:28.615: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.15:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.615: [sbi] INFO: [BSF] (NF-discover) NF Profile updated [f62fc472-935b-41ee-bc56-85d1c55fd3cc:1] (../lib/sbi/nnrf-handler.c:1095)
12/05 19:50:28.616: [sbi] INFO: [BSF] (SCP-discover) NF registered [f62fc472-935b-41ee-bc56-85d1c55fd3cc:1] (../lib/sbi/path.c:211)
12/05 19:50:28.618: [sbi] INFO: [PCF] (SCP-discover) NF registered [f636fbca-935b-41ee-bcac-9dd7f7aab382:1] (../lib/sbi/path.c:211)
12/05 19:50:28.618: [smf] INFO: UE SUPI[imsi-001010000000000] DNN[internet] IPv4[10.45.0.2] IPv6[] (../src/smf/npcf-handler.c:539)
12/05 19:50:28.619: [gtp] INFO: gtp_connect() [192.168.13.151]:2152 (../lib/gtp/path.c:60)
12/05 19:50:28.628: [core] WARNING: Unknown TLV type [60] (../lib/core/ogs-tlv-msg.c:672)
12/05 19:50:28.629: [sbi] WARNING: [UDM] (NRF-discover) NF has already been added [f62f2dc8-935b-41ee-a279-fd82055e6d3f:1] (../lib/sbi/nnrf-handler.c:1057)
12/05 19:50:28.630: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.12:80] (../lib/sbi/context.c:2174)
12/05 19:50:28.630: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.12:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.630: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.12:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.630: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.12:7777] (../lib/sbi/context.c:1917)
12/05 19:50:28.631: [sbi] INFO: [UDM] (NF-discover) NF Profile updated [f62f2dc8-935b-41ee-a279-fd82055e6d3f:1] (../lib/sbi/nnrf-handler.c:1095)
12/05 19:50:28.633: [sbi] WARNING: [UDM] (SCP-discover) NF has already been added [f62f2dc8-935b-41ee-a279-fd82055e6d3f:2] (../lib/sbi/path.c:216)
12/05 19:50:28.633: [amf] INFO: [imsi-001010000000000:1:11][0:0:NULL] /nsmf-pdusession/v1/sm-contexts/{smContextRef}/modify (../src/amf/nsmf-handler.c:837)

eUPFのPDUセッション確立のログは以下の通りです。

2023/12/05 19:50:28 INF Got Session Establishment Request from: 192.168.14.111.
2023/12/05 19:50:28 INF 
Session Establishment Request:
  CreatePDR ID: 1 
    FAR ID: 1 
    QER ID: 1 
    URR ID: 1 
    Source Interface: 1 
    UE IPv4 Address: 10.45.0.2 
  CreatePDR ID: 2 
    Outer Header Removal: 0 
    FAR ID: 2 
    QER ID: 1 
    Source Interface: 0 
    TEID: 0 
    Ipv4: <nil> 
    Ipv6: <nil> 
    UE IPv4 Address: 10.45.0.2 
  CreatePDR ID: 3 
    Outer Header Removal: 0 
    FAR ID: 1 
    QER ID: 1 
    Source Interface: 3 
    TEID: 0 
    Ipv4: <nil> 
    Ipv6: <nil> 
  CreatePDR ID: 4 
    Outer Header Removal: 0 
    FAR ID: 3 
    Source Interface: 0 
    TEID: 0 
    Ipv4: <nil> 
    Ipv6: <nil> 
    SDF Filter: permit out 58 from ff02::2/128 to assigned 
  CreateFAR ID: 1 
    Apply Action: [12 0] 
    BAR ID: 1 
  CreateFAR ID: 2 
    Apply Action: [2 0] 
    Forwarding Parameters:
      Network Instance:internet 
  CreateFAR ID: 3 
    Apply Action: [2 0] 
    Forwarding Parameters:
      Network Instance:internet 
      Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:192.168.14.111 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0} 
  CreateQER ID: 1 
    Gate Status DL: 0 
    Gate Status UL: 0 
    Max Bitrate DL: 10000000 
    Max Bitrate UL: 10000000 
    QFI: 1 
  CreateURR ID: 1 
    Measurement Method: 2 
    Volume Threshold: &{Flags:1 TotalVolume:104857600 UplinkVolume:0 DownlinkVolume:0} 
  CreateBAR ID: 1

2023/12/05 19:50:28 INF Saving FAR info to session: 1, {Action:12 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:2534254784 TransportLevelMarking:0}
2023/12/05 19:50:28 INF WARN: No OuterHeaderCreation
2023/12/05 19:50:28 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:2534254784 TransportLevelMarking:0}
2023/12/05 19:50:28 INF Saving FAR info to session: 3, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:1863231680 LocalIP:2534254784 TransportLevelMarking:0}
2023/12/05 19:50:28 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:1410065408 MaxBitrateDL:1410065408 StartUL:0 StartDL:0}
2023/12/05 19:50:28 Matched groups: [permit out 58 from ff02::2/128 to assigned 58 ff02::2 128  assigned  ]
2023/12/05 19:50:28 INF Session Establishment Request from 192.168.14.111 accepted.
2023/12/05 19:50:28 INF Got Session Modification Request from: 192.168.14.111. 

2023/12/05 19:50:28 INF Finding association for 192.168.14.111
2023/12/05 19:50:28 INF Finding session 2
2023/12/05 19:50:28 INF 
Session Modification Request:
  UpdateFAR ID: 1 
    Apply Action: [2 0] 
    Update forwarding Parameters:
      Network Instance:internet 
      Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:192.168.13.131 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0} 

2023/12/05 19:50:28 INF Updating FAR info: 1, {FarInfo:{Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:2198710464 LocalIP:2534254784 TransportLevelMarking:0} GlobalId:0}

nr-ueコマンドのコンソールログから、Open5GS 5GCからIPアドレス10.45.0.2がUEに割り当てられたことが分かります。

[2023-12-05 19:50:28.634] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.45.0.2] is up.

念のため、UEのTUNインターフェースに割り当てられたIPアドレスと一致していることを確認して下さい。

# ip addr show
...
5: uesimtun0: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 10.45.0.2/32 scope global uesimtun0
       valid_lft forever preferred_lft forever
    inet6 fe80::5463:1dcc:b5f5:b22f/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
...

google.comにpingする

UEのTUNインターフェースを指定して、pingしてみます。

TUNインターフェースの使い方は以下を参照して下さい。

https://github.com/aligungr/UERANSIM/wiki/Usage

DN=10.45.0.0/16を経由する場合

VM-DNでtcpdumpを実行し、パケットがN6インターフェースenp0s9を通過することを確認します。

VM3(UE)でping google.com

# ping google.com -I uesimtun0 -n
PING google.com (142.251.42.174) from 10.45.0.2 uesimtun0: 56(84) bytes of data.
64 bytes from 142.251.42.174: icmp_seq=1 ttl=61 time=84.2 ms
64 bytes from 142.251.42.174: icmp_seq=2 ttl=61 time=46.9 ms
64 bytes from 142.251.42.174: icmp_seq=3 ttl=61 time=71.3 ms

VM-DNでtcpdumpを実行

# tcpdump -i enp0s9 -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp0s9, link-type EN10MB (Ethernet), snapshot length 262144 bytes
19:53:29.672275 IP 10.45.0.2 > 142.251.42.174: ICMP echo request, id 2, seq 1, length 64
19:53:29.754144 IP 142.251.42.174 > 10.45.0.2: ICMP echo reply, id 2, seq 1, length 64
19:53:30.673990 IP 10.45.0.2 > 142.251.42.174: ICMP echo request, id 2, seq 2, length 64
19:53:30.718800 IP 142.251.42.174 > 10.45.0.2: ICMP echo reply, id 2, seq 2, length 64
19:53:31.675387 IP 10.45.0.2 > 142.251.42.174: ICMP echo request, id 2, seq 3, length 64
19:53:31.744309 IP 142.251.42.174 > 10.45.0.2: ICMP echo reply, id 2, seq 3, length 64

VM-UPで/sys/kernel/debug/tracing/trace_pipeを確認

# cat /sys/kernel/debug/tracing/trace_pipe
...
          <idle>-0       [000] d.s31  1242.819960: bpf_trace_printk: upf: gtp-u received
          <idle>-0       [000] d.s31  1242.819966: bpf_trace_printk: SDF: filter protocol: 4
          <idle>-0       [000] d.s31  1242.819972: bpf_trace_printk: SDF: filter source ip: 0.0.0.2, destination ip: 0.0.0.0
          <idle>-0       [000] d.s31  1242.819974: bpf_trace_printk: SDF: filter source ip mask: 255.255.255.255, destination ip mask: 0.0.0.0
          <idle>-0       [000] d.s31  1242.819977: bpf_trace_printk: SDF: filter source port lower bound: 0, source port upper bound: 65535
          <idle>-0       [000] d.s31  1242.819979: bpf_trace_printk: SDF: filter destination port lower bound: 0, destination port upper bound: 65535
          <idle>-0       [000] d.s31  1242.819980: bpf_trace_printk: SDF: packet protocol: 0
          <idle>-0       [000] d.s31  1242.819982: bpf_trace_printk: SDF: packet source ip: 10.45.0.2, destination ip: 142.251.42.174
          <idle>-0       [000] d.s31  1242.819984: bpf_trace_printk: SDF: packet source port: 0, destination port: 0
          <idle>-0       [000] d.s31  1242.819985: bpf_trace_printk: upf: sdf filter doesn't match teid:1
          <idle>-0       [000] d.s31  1242.819988: bpf_trace_printk: upf: far:1 action:2 outer_header_creation:0
          <idle>-0       [000] d.s31  1242.819990: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1410065408
          <idle>-0       [000] d.s31  1242.819992: bpf_trace_printk: upf: session for teid:1 far:1 outer_header_removal:0
          <idle>-0       [000] d.s31  1242.820004: bpf_trace_printk: upf: bpf_fib_lookup 10.45.0.2 -> 142.251.42.174: nexthop: 192.168.16.152
          <idle>-0       [000] d.s31  1242.902348: bpf_trace_printk: upf: downlink session for ip:10.45.0.2  far:0 action:2
          <idle>-0       [000] d.s31  1242.902353: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1410065408
          <idle>-0       [000] d.s31  1242.902355: bpf_trace_printk: upf: use mapping 10.45.0.2 -> TEID:1
          <idle>-0       [000] d.s31  1242.902359: bpf_trace_printk: upf: send gtp pdu 192.168.13.151 -> 192.168.13.131
          <idle>-0       [000] d.s31  1242.902369: bpf_trace_printk: upf: bpf_fib_lookup 192.168.13.151 -> 192.168.13.131: nexthop: 192.168.13.131
          <idle>-0       [000] d.s31  1243.821561: bpf_trace_printk: upf: gtp-u received
          <idle>-0       [000] d.s31  1243.821567: bpf_trace_printk: SDF: filter protocol: 4
          <idle>-0       [000] d.s31  1243.821572: bpf_trace_printk: SDF: filter source ip: 0.0.0.2, destination ip: 0.0.0.0
          <idle>-0       [000] d.s31  1243.821574: bpf_trace_printk: SDF: filter source ip mask: 255.255.255.255, destination ip mask: 0.0.0.0
          <idle>-0       [000] d.s31  1243.821576: bpf_trace_printk: SDF: filter source port lower bound: 0, source port upper bound: 65535
          <idle>-0       [000] d.s31  1243.821578: bpf_trace_printk: SDF: filter destination port lower bound: 0, destination port upper bound: 65535
          <idle>-0       [000] d.s31  1243.821579: bpf_trace_printk: SDF: packet protocol: 0
          <idle>-0       [000] d.s31  1243.821582: bpf_trace_printk: SDF: packet source ip: 10.45.0.2, destination ip: 142.251.42.174
          <idle>-0       [000] d.s31  1243.821584: bpf_trace_printk: SDF: packet source port: 0, destination port: 0
          <idle>-0       [000] d.s31  1243.821585: bpf_trace_printk: upf: sdf filter doesn't match teid:1
          <idle>-0       [000] d.s31  1243.821635: bpf_trace_printk: upf: far:1 action:2 outer_header_creation:0
          <idle>-0       [000] d.s31  1243.821637: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1410065408
          <idle>-0       [000] d.s31  1243.821640: bpf_trace_printk: upf: session for teid:1 far:1 outer_header_removal:0
          <idle>-0       [000] d.s31  1243.821650: bpf_trace_printk: upf: bpf_fib_lookup 10.45.0.2 -> 142.251.42.174: nexthop: 192.168.16.152
          <idle>-0       [000] d.s31  1243.866875: bpf_trace_printk: upf: downlink session for ip:10.45.0.2  far:0 action:2
          <idle>-0       [000] d.s31  1243.866879: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1410065408
          <idle>-0       [000] d.s31  1243.866882: bpf_trace_printk: upf: use mapping 10.45.0.2 -> TEID:1
          <idle>-0       [000] d.s31  1243.866885: bpf_trace_printk: upf: send gtp pdu 192.168.13.151 -> 192.168.13.131
          <idle>-0       [000] d.s31  1243.866896: bpf_trace_printk: upf: bpf_fib_lookup 192.168.13.151 -> 192.168.13.131: nexthop: 192.168.13.131
          <idle>-0       [000] d.s31  1244.822860: bpf_trace_printk: upf: gtp-u received
          <idle>-0       [000] d.s31  1244.822867: bpf_trace_printk: SDF: filter protocol: 4
          <idle>-0       [000] d.s31  1244.822873: bpf_trace_printk: SDF: filter source ip: 0.0.0.2, destination ip: 0.0.0.0
          <idle>-0       [000] d.s31  1244.822876: bpf_trace_printk: SDF: filter source ip mask: 255.255.255.255, destination ip mask: 0.0.0.0
          <idle>-0       [000] d.s31  1244.822880: bpf_trace_printk: SDF: filter source port lower bound: 0, source port upper bound: 65535
          <idle>-0       [000] d.s31  1244.822882: bpf_trace_printk: SDF: filter destination port lower bound: 0, destination port upper bound: 65535
          <idle>-0       [000] d.s31  1244.822883: bpf_trace_printk: SDF: packet protocol: 0
          <idle>-0       [000] d.s31  1244.822886: bpf_trace_printk: SDF: packet source ip: 10.45.0.2, destination ip: 142.251.42.174
          <idle>-0       [000] d.s31  1244.822888: bpf_trace_printk: SDF: packet source port: 0, destination port: 0
          <idle>-0       [000] d.s31  1244.822890: bpf_trace_printk: upf: sdf filter doesn't match teid:1
          <idle>-0       [000] d.s31  1244.822892: bpf_trace_printk: upf: far:1 action:2 outer_header_creation:0
          <idle>-0       [000] d.s31  1244.822894: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1410065408
          <idle>-0       [000] d.s31  1244.822897: bpf_trace_printk: upf: session for teid:1 far:1 outer_header_removal:0
          <idle>-0       [000] d.s31  1244.822942: bpf_trace_printk: upf: bpf_fib_lookup 10.45.0.2 -> 142.251.42.174: nexthop: 192.168.16.152
          <idle>-0       [000] d.s31  1244.892284: bpf_trace_printk: upf: downlink session for ip:10.45.0.2  far:0 action:2
          <idle>-0       [000] d.s31  1244.892288: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1410065408
          <idle>-0       [000] d.s31  1244.892290: bpf_trace_printk: upf: use mapping 10.45.0.2 -> TEID:1
          <idle>-0       [000] d.s31  1244.892294: bpf_trace_printk: upf: send gtp pdu 192.168.13.151 -> 192.168.13.131
          <idle>-0       [000] d.s31  1244.892305: bpf_trace_printk: upf: bpf_fib_lookup 192.168.13.151 -> 192.168.13.131: nexthop: 192.168.13.131
...

また、以下のようにnr-binderツールにおいて、TUNインターフェースに割り当てられたIPアドレスを指定して、大半のアプリケーションを使用することができます。

VM3(UE)でcurl google.comを実行

# sh nr-binder 10.45.0.2 curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

VM-DNでtcpdumpを実行

19:55:27.220841 IP 10.45.0.2.60687 > 142.251.42.174.80: Flags [S], seq 395724995, win 65280, options [mss 1360,sackOK,TS val 2475647894 ecr 0,nop,wscale 7], length 0
19:55:27.260582 IP 142.251.42.174.80 > 10.45.0.2.60687: Flags [S.], seq 64001, ack 395724996, win 65535, options [mss 1460], length 0
19:55:27.262145 IP 10.45.0.2.60687 > 142.251.42.174.80: Flags [.], ack 1, win 65280, length 0
19:55:27.262396 IP 10.45.0.2.60687 > 142.251.42.174.80: Flags [P.], seq 1:75, ack 1, win 65280, length 74: HTTP: GET / HTTP/1.1
19:55:27.262621 IP 142.251.42.174.80 > 10.45.0.2.60687: Flags [.], ack 75, win 65535, length 0
19:55:27.357432 IP 142.251.42.174.80 > 10.45.0.2.60687: Flags [P.], seq 1:774, ack 75, win 65535, length 773: HTTP: HTTP/1.1 301 Moved Permanently
19:55:27.359426 IP 10.45.0.2.60687 > 142.251.42.174.80: Flags [.], ack 774, win 64507, length 0
19:55:27.362364 IP 10.45.0.2.60687 > 142.251.42.174.80: Flags [F.], seq 75, ack 774, win 64507, length 0
19:55:27.362623 IP 142.251.42.174.80 > 10.45.0.2.60687: Flags [.], ack 76, win 65535, length 0
19:55:27.432592 IP 142.251.42.174.80 > 10.45.0.2.60687: Flags [F.], seq 774, ack 76, win 65535, length 0
19:55:27.434545 IP 10.45.0.2.60687 > 142.251.42.174.80: Flags [.], ack 775, win 64507, length 0

なお、nr-binderはpingでは機能しません。理由はこちらを参照して下さい。

以上で、eBPF/XDP対応UPFのeUPFのDNに接続し、DNに任意のパケットを流せるようになり、SA構成のローカル5Gを想定した5GCのシミュレーション携帯網として、MECを試す環境を構築できました。

最後に

eBPF/XDP対応UPFのeUPFは、U-Planeのデータネットワークを高速化する技術と期待しています。本記事では、Virtualbox VM上の設定例を書いてみましたが、XDP対応のdriverやNICを搭載したサーバを設定する際のご参考になれれば幸いです。

最後に、元記事はGithubに書いたものです。

主な変更履歴

[2024.02.15] 初版。