OpenVNetとは
OpenVNetはネットワークの仮想化を実現するためのオープンソースのソフトウェアです。
準備
環境 CentOS6.7
とりあえずインストールマニュアルをなぞっていきます。
http://openvnet.org/installation/
最終的に下の図のようなものができます。
上から
curl -o /etc/yum.repos.d/openvnet.repo -R https://raw.githubusercontent.com/axsh/openvnet/master/deployment/yum_repositories/stable/openvnet.repo
curl -o /etc/yum.repos.d/openvnet-third-party.repo -R https://raw.githubusercontent.com/axsh/openvnet/master/deployment/yum_repositories/stable/openvnet-third-party.repo
yum install -y epel-release
までは順調にいくと思います。
すると
yum install -y openvnet
で
Error: Package: kmod-openvswitch-2.3.1-1.el6.x86_64 (openvnet-third-party)
Requires: kernel(openvswitch_handle_frame_hook) = 0xbe03eb4d
とでて失敗します。
ここはCentOS6.6ではマニュアルのままで成功する環境もありました。
そこで、こちら
https://github.com/mao172/ovs-installer
のovs-installerをお借りしてみます。
curl -L https://raw.githubusercontent.com/mao172/ovs-installer/master/install.sh | bash -s -- -v 2.3.1
そこでもう一回
yum install -y openvnet
をするとComplete!します。
素晴らしいですね。
続いて、
yum install -y mysql-server redis
を行った後、マニュアルには
/etc/openvnet/vnmgr.conf
/etc/openvnet/vna.conf
/etc/openvnet/webapi.conf
の設定に関する記述がありますが
今回はこれは変更しません。
次にbr0を作ります。
vi /etc/sysconfig/network-scripts/ifcfg-br0
で中を
DEVICE=br0
DEVICETYPE=ovs
TYPE=OVSBridge
ONBOOT=yes
BOOTPROTO=static
HOTPLUG=no
OVS_EXTRA="
set bridge ${DEVICE} protocols=OpenFlow10,OpenFlow12,OpenFlow13 --
set bridge ${DEVICE} other_config:disable-in-band=true --
set bridge ${DEVICE} other-config:datapath-id=0000020100000001 --
set bridge ${DEVICE} other-config:hwaddr=02:01:00:00:00:01 --
set-fail-mode ${DEVICE} standalone --
set-controller ${DEVICE} tcp:127.0.0.1:6633
"
とします。
datapath-idはマニュアルと変えてあります。
また、hwaddrはそのままで問題ないと思いますが
同じMACアドレスを持っているものがないようにします。
次にいろいろとスタートさせます。
service openvswitch start
ifup br0
service redis start
/etc/openvnet/common.confに関しても今回は設定を変更する必要はありません。
続いてデータベースを設定していきます。
service mysqld start
chkconfig mysqld on
PATH=/opt/axsh/openvnet/ruby/bin:${PATH}
cd /opt/axsh/openvnet/vnet
bundle exec rake db:create
bundle exec rake db:init
ここまでで準備終了です。
OpenVNet
OpenVNetのvnmgrとwebapiをスタートさせます。
initctl start vnet-vnmgr
initctl start vnet-webapi
datapathを追加します。
dpidはbr0の設定でdatapath-idに設定したものと同じで
今回はマニュアルと違うものを設定しています。
vnctl datapaths add --uuid dp-test1 --display-name test1 --dpid 0x0000020100000001 --node-id vna
次はOpenVNetで使用するネットワークの設定です。
vnctl networks add --uuid nw-test1 --display-name testnet1 --ipv4-network 10.100.0.0 --ipv4-prefix 24 --network-mode virtual
次はこの後入れるLXCをOpenVNetで接続するための
interfaceの設定をします。
vnctl interfaces add --uuid if-inst1 --mode vif --owner-datapath-uuid dp-test1 --mac-address 10:54:ff:00:00:01 --network-uuid nw-test1 --ipv4-address 10.100.0.10 --port-name inst1
vnctl interfaces add --uuid if-inst2 --mode vif --owner-datapath-uuid dp-test1 --mac-address 10:54:ff:00:00:02 --network-uuid nw-test1 --ipv4-address 10.100.0.11 --port-name inst2
そしてOpenVNetのvnaを起動します。
もし、vnmgrとwebapiが起動していなければこちらも起動します。
initctl start vnet-vna
LXC
マニュアル通りLXCでOpenVNetを使ってみます。
まずは必要なものの準備をします。
yum -y install lxc lxc-templates
mkdir /cgroup
echo "cgroup /cgroup cgroup defaults 0 0" >> /etc/fstab
mount /cgroup
yum install -y rsync (既に入っていることもあります)
lxc-create -t centos -n inst1
lxc-create -t centos -n inst2
inst1、inst2の設定ファイルを編集します。
既にいろいろと書いてありますがすべて消して以下の設定を書きます。
vi /var/lib/lxc/inst1/config
lxc.network.type = veth
lxc.network.flags = up
lxc.network.veth.pair = inst1
lxc.network.ipv4 = 10.100.0.10
lxc.network.hwaddr = 10:54:FF:00:00:01
lxc.rootfs = /var/lib/lxc/inst1/rootfs
lxc.include = /usr/share/lxc/config/centos.common.conf
lxc.arch = x86_64
lxc.utsname = inst1
lxc.autodev = 0
同じようにinst2も
vi /var/lib/lxc/inst2/config
lxc.network.type = veth
lxc.network.flags = up
lxc.network.veth.pair = inst2
lxc.network.ipv4 = 10.100.0.11
lxc.network.hwaddr = 10:54:FF:00:00:02
lxc.rootfs = /var/lib/lxc/inst2/rootfs
lxc.include = /usr/share/lxc/config/centos.common.conf
lxc.arch = x86_64
lxc.utsname = inst2
lxc.autodev = 0
それぞれをスタートさせdatapathにさします。
lxc-start -d -n inst1
lxc-start -d -n inst2
ovs-vsctl add-port br0 inst1
ovs-vsctl add-port br0 inst2
それぞれのpasswordを確認します。
cat /var/lib/lxc/inst1/tmp_root_pass
cat /var/lib/lxc/inst2/tmp_root_pass
inst1に入ります。
lxc-console -n inst1
これで10.100.0.11にpingが通れば成功です。
Flowの確認
最後にフローを見てみます。
ovs-ofctl -O OpenFlow13 dump-flows br0
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x900000000000000, duration=325.525s, table=0, n_packets=1, n_bytes=90, priority=1,tun_id=0 actions=drop
cookie=0x900000000000000, duration=325.525s, table=0, n_packets=0, n_bytes=0, priority=0 actions=write_metadata:0x80000000000/0xc0000000000,goto_table:3
cookie=0x900000000000000, duration=325.525s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=CONTROLLER actions=write_metadata:0x4040000000000/0x40c0000000000,goto_table:7
cookie=0x500000000000001, duration=130.748s, table=0, n_packets=17, n_bytes=3158, priority=2,in_port=1 actions=write_metadata:0x700040000000001/0xff000c007fffffff,goto_table:15
cookie=0x5000000fffffffe, duration=325.467s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=LOCAL actions=write_metadata:0x40000000000/0xc0000000000,goto_table:6
cookie=0x500000000000002, duration=130.024s, table=0, n_packets=17, n_bytes=3170, priority=2,in_port=2 actions=write_metadata:0x700040000000002/0xff000c007fffffff,goto_table:15
cookie=0x900000000000000, duration=325.526s, table=3, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=4, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x400000000000001, duration=130.240s, table=4, n_packets=0, n_bytes=0, priority=30,tun_id=0x80000001 actions=write_metadata:0x300000000000001/0xff0000007fffffff,goto_table:20
cookie=0x900000000000000, duration=325.526s, table=6, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=7, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=8, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=9, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=10, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=11, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0xc00001300000001, duration=130.314s, table=11, n_packets=0, n_bytes=0, priority=50,dl_src=10:54:ff:00:00:01 actions=drop
cookie=0xc00001300000002, duration=129.963s, table=11, n_packets=0, n_bytes=0, priority=50,dl_src=10:54:ff:00:00:02 actions=drop
cookie=0xc00001200000002, duration=129.963s, table=11, n_packets=0, n_bytes=0, priority=30,arp,dl_dst=10:54:ff:00:00:02,arp_tpa=10.100.0.11 actions=write_metadata:0x8000000180000000/0xffffffff80000000,goto_table:12
cookie=0xc00001200000002, duration=129.963s, table=11, n_packets=0, n_bytes=0, priority=30,ip,dl_dst=10:54:ff:00:00:02,nw_dst=10.100.0.11 actions=write_metadata:0x8000000180000000/0xffffffff80000000,goto_table:12
cookie=0xc00001200000001, duration=130.166s, table=11, n_packets=0, n_bytes=0, priority=30,ip,dl_dst=10:54:ff:00:00:01,nw_dst=10.100.0.10 actions=write_metadata:0x8000000180000000/0xffffffff80000000,goto_table:12
cookie=0xc00001200000001, duration=130.166s, table=11, n_packets=0, n_bytes=0, priority=30,arp,dl_dst=10:54:ff:00:00:01,arp_tpa=10.100.0.10 actions=write_metadata:0x8000000180000000/0xffffffff80000000,goto_table:12
cookie=0xc00001300000001, duration=130.314s, table=11, n_packets=0, n_bytes=0, priority=50,arp,arp_sha=10:54:ff:00:00:01 actions=drop
cookie=0xc00001300000002, duration=129.963s, table=11, n_packets=0, n_bytes=0, priority=50,arp,arp_sha=10:54:ff:00:00:02 actions=drop
cookie=0x900000000000000, duration=325.526s, table=12, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0xc00001200000002, duration=129.963s, table=12, n_packets=0, n_bytes=0, priority=90,arp,metadata=0x8000000100000000/0xffffffff00000000,arp_spa=10.100.0.11 actions=drop
cookie=0xc00001200000001, duration=130.166s, table=12, n_packets=0, n_bytes=0, priority=90,ip,metadata=0x8000000100000000/0xffffffff00000000,nw_src=10.100.0.10 actions=drop
cookie=0xc00001200000001, duration=130.166s, table=12, n_packets=0, n_bytes=0, priority=90,arp,metadata=0x8000000100000000/0xffffffff00000000,arp_spa=10.100.0.10 actions=drop
cookie=0xc00001200000002, duration=129.963s, table=12, n_packets=0, n_bytes=0, priority=90,ip,metadata=0x8000000100000000/0xffffffff00000000,nw_src=10.100.0.11 actions=drop
cookie=0x900000000000000, duration=325.526s, table=13, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=15, n_packets=8, n_bytes=588, priority=0 actions=drop
cookie=0xc00001200000001, duration=130.166s, table=15, n_packets=2, n_bytes=84, priority=30,arp,metadata=0x700000000000001/0xff0000007fffffff,dl_src=10:54:ff:00:00:01,arp_spa=10.100.0.10,arp_sha=10:54:ff:00:00:01 actions=write_metadata:0x300000000000001/0xff0000007fffffff,goto_table:18
cookie=0xc00001200000002, duration=129.963s, table=15, n_packets=2, n_bytes=84, priority=30,arp,metadata=0x700000000000002/0xff0000007fffffff,dl_src=10:54:ff:00:00:02,arp_spa=10.100.0.11,arp_sha=10:54:ff:00:00:02 actions=write_metadata:0x300000000000001/0xff0000007fffffff,goto_table:18
cookie=0xc00001200000001, duration=130.166s, table=15, n_packets=4, n_bytes=392, priority=30,ip,metadata=0x700000000000001/0xff0000007fffffff,dl_src=10:54:ff:00:00:01,nw_src=10.100.0.10 actions=write_metadata:0x300000000000001/0xff0000007fffffff,goto_table:18
cookie=0xc00001200000002, duration=129.963s, table=15, n_packets=4, n_bytes=392, priority=30,ip,metadata=0x700000000000002/0xff0000007fffffff,dl_src=10:54:ff:00:00:02,nw_src=10.100.0.11 actions=write_metadata:0x300000000000001/0xff0000007fffffff,goto_table:18
cookie=0xc00001200000002, duration=129.963s, table=15, n_packets=7, n_bytes=2394, priority=30,ip,metadata=0x700000000000002/0xff0000007fffffff,dl_src=10:54:ff:00:00:02,nw_src=0.0.0.0 actions=write_metadata:0x300000000000001/0xff0000007fffffff,goto_table:18
cookie=0xc00001200000001, duration=130.166s, table=15, n_packets=7, n_bytes=2394, priority=30,ip,metadata=0x700000000000001/0xff0000007fffffff,dl_src=10:54:ff:00:00:01,nw_src=0.0.0.0 actions=write_metadata:0x300000000000001/0xff0000007fffffff,goto_table:18
cookie=0x900000000000000, duration=325.526s, table=16, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=17, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.525s, table=18, n_packets=26, n_bytes=5740, priority=0 actions=goto_table:20
cookie=0x400000000000001, duration=130.240s, table=20, n_packets=0, n_bytes=0, priority=40,arp,metadata=0x300080000000001/0xff000c007fffffff actions=goto_table:21
cookie=0x900000000000000, duration=325.526s, table=20, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x400000000000001, duration=130.240s, table=20, n_packets=26, n_bytes=5740, priority=30,metadata=0x300000000000001/0xff0000007fffffff actions=goto_table:30
cookie=0x400000000000001, duration=130.226s, table=21, n_packets=0, n_bytes=0, priority=1,arp,metadata=0x300080000000001/0xff000c007fffffff actions=learn(table=43,idle_timeout=36000,priority=35,cookie=0x400000000000001,metadata=0x300040000000001,NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),goto_table:42
cookie=0x900000000000000, duration=325.525s, table=21, n_packets=0, n_bytes=0, priority=0 actions=goto_table:42
cookie=0x400000000000001, duration=130.233s, table=21, n_packets=0, n_bytes=0, priority=3,arp,tun_id=0,metadata=0x300080000000001/0xff000c007fffffff actions=learn(table=43,idle_timeout=36000,priority=35,cookie=0x400000000000001,metadata=0x300040000000001,NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],output:NXM_OF_IN_PORT[]),goto_table:42
cookie=0x900000000000000, duration=325.525s, table=21, n_packets=0, n_bytes=0, priority=2,arp,tun_id=0 actions=goto_table:42
cookie=0x900000000000000, duration=325.525s, table=30, n_packets=26, n_bytes=5740, priority=0 actions=goto_table:42
cookie=0x900000000000000, duration=325.526s, table=31, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=32, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=33, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=34, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=35, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=36, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=37, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.525s, table=40, n_packets=0, n_bytes=0, priority=0 actions=goto_table:41
cookie=0xc00001200000001, duration=130.166s, table=40, n_packets=0, n_bytes=0, priority=40,ip,metadata=0x300000000000001/0xff0000007fffffff,nw_dst=10.100.0.10 actions=set_field:10:54:ff:00:00:01->eth_dst,goto_table:42
cookie=0xc00001200000002, duration=129.963s, table=40, n_packets=0, n_bytes=0, priority=40,ip,metadata=0x300000000000001/0xff0000007fffffff,nw_dst=10.100.0.11 actions=set_field:10:54:ff:00:00:02->eth_dst,goto_table:42
cookie=0x900000000000000, duration=325.526s, table=41, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=42, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x400000000000001, duration=130.240s, table=42, n_packets=26, n_bytes=5740, priority=30,metadata=0x300000000000001/0xff0000007fffffff actions=goto_table:43
cookie=0xc00001200000001, duration=130.166s, table=43, n_packets=6, n_bytes=476, priority=60,metadata=0x300000000000001/0xff0000007fffffff,dl_dst=10:54:ff:00:00:01 actions=write_metadata:0x700000000000001/0xff0000007fffffff,goto_table:45
cookie=0xc00001200000002, duration=129.963s, table=43, n_packets=5, n_bytes=434, priority=60,metadata=0x300000000000001/0xff0000007fffffff,dl_dst=10:54:ff:00:00:02 actions=write_metadata:0x700000000000002/0xff0000007fffffff,goto_table:45
cookie=0x900000000000000, duration=325.526s, table=43, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.525s, table=43, n_packets=15, n_bytes=4830, priority=30,dl_dst=ff:ff:ff:ff:ff:ff actions=goto_table:50
cookie=0x900000000000000, duration=325.525s, table=45, n_packets=0, n_bytes=0, priority=0 actions=goto_table:46
cookie=0xc00007100000002, duration=130.024s, table=45, n_packets=4, n_bytes=392, priority=90,metadata=0x700000000000002/0xff0000007fffffff actions=goto_table:90
cookie=0xc00007100000001, duration=130.586s, table=45, n_packets=4, n_bytes=392, priority=90,metadata=0x700000000000001/0xff0000007fffffff actions=goto_table:90
cookie=0xe00001100000000, duration=325.733s, table=45, n_packets=3, n_bytes=126, priority=90,arp actions=goto_table:90
cookie=0x900000000000000, duration=325.526s, table=46, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.525s, table=50, n_packets=15, n_bytes=4830, priority=0 actions=goto_table:51
cookie=0x900000000000000, duration=325.526s, table=51, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x400000000000001, duration=129.963s, table=51, n_packets=15, n_bytes=4830, priority=1,metadata=0x300000000000001/0xff0000007fffffff actions=output:1,output:2,goto_table:52
cookie=0x900000000000000, duration=325.525s, table=52, n_packets=15, n_bytes=4830, priority=0 actions=goto_table:53
cookie=0x900000000000000, duration=325.525s, table=52, n_packets=0, n_bytes=0, priority=10,metadata=0x80000000000/0xc0000000000 actions=drop
cookie=0x900000000000000, duration=325.526s, table=53, n_packets=15, n_bytes=4830, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=70, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=71, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=72, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=73, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=74, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=80, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.525s, table=80, n_packets=0, n_bytes=0, priority=2,dl_dst=ff:ff:ff:ff:ff:ff actions=drop
cookie=0x900000000000000, duration=325.526s, table=81, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=82, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=84, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=85, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.525s, table=85, n_packets=0, n_bytes=0, priority=1,tun_id=0x10000001 actions=goto_table:86
cookie=0x900000000000000, duration=325.525s, table=85, n_packets=0, n_bytes=0, priority=1,tun_id=0x80000000/0x80000000 actions=goto_table:86
cookie=0x900000000000000, duration=325.526s, table=86, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=90, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x500000000000002, duration=130.024s, table=90, n_packets=5, n_bytes=434, priority=10,metadata=0x700000000000002/0xff0000007fffffff actions=output:2
cookie=0x500000000000001, duration=130.748s, table=90, n_packets=6, n_bytes=476, priority=10,metadata=0x700000000000001/0xff0000007fffffff actions=output:1
cookie=0x900000000000000, duration=325.526s, table=91, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x900000000000000, duration=325.526s, table=92, n_packets=0, n_bytes=0, priority=0 actions=drop
これが下のようだと失敗です。
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x0, duration=3351.584s, table=0, n_packets=46, n_bytes=7068, priority=0 actions=NORMAL
私の場合このようになった原因はwebapiを起動してからデータベースを作成したことでした。
この場合、
initctl restart vnet-webapi
でwebapiを再起動した後、
データベースの登録からやり直すと正しく動くようになりました。