LoginSignup
10
7

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

@m10i(MINETA Hiroki)

Linux + SR-IOV 事始め

Last updated at Posted at 2017-04-30

ついカッとなって

Intel X540-T2 をぽりちまりした(160k)
このNICではSR-IOVが利用できるので、VMで高速な通信をしたい場合には有効と思われます
早速ですが下記参考元を見つつ、試してみました

参考元

https://access.redhat.com/documentation/ja-JP/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/sect-Virtualization_Host_Configuration_and_Guest_Installation_Guide-SR_IOV-How_SR_IOV_Libvirt_Works.html
http://www.iij.ad.jp/company/development/tech/techweek/pdf/151113_1.pdf

SR-IOV 利用まで

##1. BIOS設定
SR-IOVを利用する為、BIOSより「VT-d(またはAMD-Vi)」「SR-IOV」を有効化します

##2. SR-IOV設定
Intel X540の標準ドライバは「ixgbe」です
ここではSR-IOVを有効化し、「Virtual Function(VF)」の数を定義します
注意 X540を利用してSSH接続されている場合、切断されるためコンソールもしくは
別InterfaceにてSSH接続を行ってください

  • ドライバモジュールを削除
    一緒にixgbevfも削除します
    これがあるとVFもInterfaceとして認識されてしまい、大量のethが発生する為です
shell
# modprobe -r ixgbe
# modprobe -r ixgbevf
  • SR-IOVを有効化してドライバモジュールをロード
    ここではX540-T2を利用し、2ポート有る為16ポート+16ポートにしています
shell
# modprobe ixgbe max_vfs=16,16

##3. 確認

  • ip link showにてVFが追加されているか確認します
shell
# ip link show ens6f0
40: ens6f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether a0:36:9f:50:bf:24 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC f6:9c:5d:de:6a:6b, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC d6:c4:ff:f3:8e:38, spoof checking on, link-state auto, trust off, query_rss off
    vf 2 MAC ea:01:a8:4d:4a:5f, spoof checking on, link-state auto, trust off, query_rss off
    vf 3 MAC c2:75:f5:39:8f:b5, spoof checking on, link-state auto, trust off, query_rss off
    vf 4 MAC 32:07:1a:bb:04:bf, spoof checking on, link-state auto, trust off, query_rss off
    vf 5 MAC 72:ae:de:26:14:70, spoof checking on, link-state auto, trust off, query_rss off
    vf 6 MAC fe:01:58:37:83:12, spoof checking on, link-state auto, trust off, query_rss off
    vf 7 MAC 0e:c1:77:f0:3d:c0, spoof checking on, link-state auto, trust off, query_rss off
    vf 8 MAC 26:16:a0:8e:04:90, spoof checking on, link-state auto, trust off, query_rss off
    vf 9 MAC 06:41:55:a8:5c:ae, spoof checking on, link-state auto, trust off, query_rss off
    vf 10 MAC 5a:a1:c6:ab:03:21, spoof checking on, link-state auto, trust off, query_rss off
    vf 11 MAC 52:59:97:0c:86:b7, spoof checking on, link-state auto, trust off, query_rss off
    vf 12 MAC 8a:df:2e:ae:45:65, spoof checking on, link-state auto, trust off, query_rss off
    vf 13 MAC 72:d6:47:cf:59:00, spoof checking on, link-state auto, trust off, query_rss off
    vf 14 MAC 02:5f:d7:7f:b9:5d, spoof checking on, link-state auto, trust off, query_rss off
    vf 15 MAC 9a:e1:b7:10:ec:4a, spoof checking on, link-state auto, trust off, query_rss off

# ip link show ens6f1
57: ens6f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether a0:36:9f:50:bf:26 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 56:d5:33:d1:12:c4, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC 4e:04:e2:83:47:92, spoof checking on, link-state auto, trust off, query_rss off
    vf 2 MAC a6:ad:91:0c:37:53, spoof checking on, link-state auto, trust off, query_rss off
    vf 3 MAC 42:ff:b4:62:0e:83, spoof checking on, link-state auto, trust off, query_rss off
    vf 4 MAC b6:5d:04:90:f4:b5, spoof checking on, link-state auto, trust off, query_rss off
    vf 5 MAC 22:83:05:69:f9:76, spoof checking on, link-state auto, trust off, query_rss off
    vf 6 MAC 02:17:44:ab:85:36, spoof checking on, link-state auto, trust off, query_rss off
    vf 7 MAC 52:67:f3:be:55:37, spoof checking on, link-state auto, trust off, query_rss off
    vf 8 MAC 72:24:3f:d4:43:9a, spoof checking on, link-state auto, trust off, query_rss off
    vf 9 MAC ee:8c:52:16:90:fc, spoof checking on, link-state auto, trust off, query_rss off
    vf 10 MAC 06:57:e5:db:d9:7e, spoof checking on, link-state auto, trust off, query_rss off
    vf 11 MAC 62:6a:a2:de:22:ea, spoof checking on, link-state auto, trust off, query_rss off
    vf 12 MAC fa:5f:77:54:ec:7b, spoof checking on, link-state auto, trust off, query_rss off
    vf 13 MAC 42:7c:87:b0:b1:4d, spoof checking on, link-state auto, trust off, query_rss off
    vf 14 MAC e6:10:97:50:7d:bc, spoof checking on, link-state auto, trust off, query_rss off
    vf 15 MAC de:7e:4d:07:46:23, spoof checking on, link-state auto, trust off, query_rss off

  • 表示されない場合
    BIOS等でSR-IOVが有効化されていない可能性が高いので確認してください

  • 大量のInterfaceについて
    前述のixgbevfを削除しない場合、VFの数だけInterfaceが認識されます
    OS上で利用するのなら良いのですが、仮想マシンで利用する場合は邪魔でしかないので無効がおすすめです

shell
27: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether c2:75:f5:39:8f:b5 brd ff:ff:ff:ff:ff:ff
28: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 32:07:1a:bb:04:bf brd ff:ff:ff:ff:ff:ff
29: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 72:ae:de:26:14:70 brd ff:ff:ff:ff:ff:ff
30: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether fe:01:58:37:83:12 brd ff:ff:ff:ff:ff:ff
31: eth4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 0e:c1:77:f0:3d:c0 brd ff:ff:ff:ff:ff:ff
32: eth5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 26:16:a0:8e:04:90 brd ff:ff:ff:ff:ff:ff
33: eth6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 06:41:55:a8:5c:ae brd ff:ff:ff:ff:ff:ff
34: eth7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 5a:a1:c6:ab:03:21 brd ff:ff:ff:ff:ff:ff
35: eth8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:59:97:0c:86:b7 brd ff:ff:ff:ff:ff:ff
36: eth9: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 8a:df:2e:ae:45:65 brd ff:ff:ff:ff:ff:ff
37: eth10: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 72:d6:47:cf:59:00 brd ff:ff:ff:ff:ff:ff
38: eth11: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:5f:d7:7f:b9:5d brd ff:ff:ff:ff:ff:ff
39: eth12: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 9a:e1:b7:10:ec:4a brd ff:ff:ff:ff:ff:ff

41: eth14: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 56:d5:33:d1:12:c4 brd ff:ff:ff:ff:ff:ff
42: eth15: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 4e:04:e2:83:47:92 brd ff:ff:ff:ff:ff:ff
43: eth16: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether a6:ad:91:0c:37:53 brd ff:ff:ff:ff:ff:ff
44: eth17: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 42:ff:b4:62:0e:83 brd ff:ff:ff:ff:ff:ff
45: eth18: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether b6:5d:04:90:f4:b5 brd ff:ff:ff:ff:ff:ff
46: eth19: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 22:83:05:69:f9:76 brd ff:ff:ff:ff:ff:ff
47: eth20: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:17:44:ab:85:36 brd ff:ff:ff:ff:ff:ff
48: eth21: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:67:f3:be:55:37 brd ff:ff:ff:ff:ff:ff
49: eth22: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 72:24:3f:d4:43:9a brd ff:ff:ff:ff:ff:ff
50: eth23: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether ee:8c:52:16:90:fc brd ff:ff:ff:ff:ff:ff
51: eth24: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 06:57:e5:db:d9:7e brd ff:ff:ff:ff:ff:ff
52: eth25: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 62:6a:a2:de:22:ea brd ff:ff:ff:ff:ff:ff
53: eth26: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether fa:5f:77:54:ec:7b brd ff:ff:ff:ff:ff:ff
54: eth27: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 42:7c:87:b0:b1:4d brd ff:ff:ff:ff:ff:ff
55: eth28: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether e6:10:97:50:7d:bc brd ff:ff:ff:ff:ff:ff
56: eth29: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether de:7e:4d:07:46:23 brd ff:ff:ff:ff:ff:ff
  • nmcli
    同様に大量の「Wired connection」が認識されます

##4. 永続化
modprobe.dに設定を保存し、再起動後も有効化させます
Intel X540-T2 の為、2port(16+16)分設定
併せてblacklistにixgbevfを追加します
これによりVFをInterfaceとして認識されるのを防ぎます

shell
echo "options ixgbe max_vfs=16,16" > /etc/modprobe.d/ixgbe.conf
echo "blacklist ixgbevf" >> /etc/modprobe.d/ixgbe.conf

##5. virshでの利用
VFデバイスの実装位置を確認します
ここでは03:[11-13]:[0-7]が対象でした
前半16が1ポート目、後半16が2ポート目に該当すると思われます

shell
# lspci | grep X540
03:00.0 Ethernet controller: Intel(R) Ethernet Controller X540-AT2 (rev 01) /* 物理 */
03:00.1 Ethernet controller: Intel(R) Ethernet Controller X540-AT2 (rev 01) /* 物理 */
03:10.0 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:10.1 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:10.2 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:10.3 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:10.4 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:10.5 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:10.6 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:10.7 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:11.0 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:11.1 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:11.2 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:11.3 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:11.4 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:11.5 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:11.6 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:11.7 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:12.0 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:12.1 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:12.2 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:12.3 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:12.4 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:12.5 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:12.6 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:12.7 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:13.0 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:13.1 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:13.2 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:13.3 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:13.4 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:13.5 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:13.6 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
03:13.7 Ethernet controller: Intel(R) X540 Ethernet Controller Virtual Function (rev 01)
  • virsh 確認
    virshでデバイスが確認する場合は「nodedev-list」を利用します
    ここではX540以外のデバイスを選択しないよう、grepで絞りました
shell
# virsh nodedev-list | grep -E "pci_0+_03_1[0-3]_"
pci_0000_03_10_0
pci_0000_03_10_1
pci_0000_03_10_2
pci_0000_03_10_3
pci_0000_03_10_4
pci_0000_03_10_5
pci_0000_03_10_6
pci_0000_03_10_7
pci_0000_03_11_0
pci_0000_03_11_1
pci_0000_03_11_2
pci_0000_03_11_3
pci_0000_03_11_4
pci_0000_03_11_5
pci_0000_03_11_6
pci_0000_03_11_7
pci_0000_03_12_0
pci_0000_03_12_1
pci_0000_03_12_2
pci_0000_03_12_3
pci_0000_03_12_4
pci_0000_03_12_5
pci_0000_03_12_6
pci_0000_03_12_7
pci_0000_03_13_0
pci_0000_03_13_1
pci_0000_03_13_2
pci_0000_03_13_3
pci_0000_03_13_4
pci_0000_03_13_5
pci_0000_03_13_6
pci_0000_03_13_7

#まずはここまで
ここからVMを作るのですが、まだ作成中

10
7
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
10
7

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?