2012年10月17日水曜日

Quantum + OVS + GRE with Devstack


このエントリーをはてなブックマークに追加


OpenStack Folsom で正式コンポーネントになった Quantum(+ OVS) を使った環境構築の解説。

環境構築は面倒なので devstack を利用。CinderについてはNovaと一緒なので省略。

quantumのみの解説で、その他のコンポーネントについては devstack の実行後に得られる nova.conf 等を参照。

*10/19 絵が間違ってたので修正しました。


Quantum の構成

Quantum はこれまでの nova-networkの機能を置き換える。複数のコンポーネントから構成されている。

■quantum-server
APIを受け付ける窓口。クラスタ内で1つ(2重化は未確認)

■quantum-agent
serverからの指示を受け、実際のネットワーク制御を行う。nova-computeと同じノードで動かす。制御対象のネットワークはPlugin形式で選択可能。今回はOpen vSwitchを使う。

■quantum-l3-agent
内部ネットワークのルーティングと外部ネットワーク接続を行う。floating ipの管理も行う。

■quantum-dhcp-agent
内部セグメントで起動したVMにアドレスを払い出す。

これらの配置は公式ドキュメントによると以下の様に配置できる。

ただこの環境はDevstackでは作れない(?)ので、今回の例ではもう少しシンプルな構成で試す(環境の解説は次項)
*devstackで、q-l3,q-dhcp,q-agtだけで./stack.shするとエラー。詳細は面倒なので調べてない。


OVSの基礎

複数ノードで構成する場合、Open vSwitch(OVS) 間を接続させる必要がある。

OVS間の通信については以下の解説がわかりやすい。

Trema 日記
Openvswitch を OpenFlow スイッチとして使う
EtherIP を使って OpenFlow ネットワークを作る
GRE を使って OpenFlow ネットワークを作る

Quantum から OVSを使う場合、OVS間の通信にはVLAN、GREのどちらかを選択できる。今回はVLANスイッチ等を使わずに疎通できるGREを使う。


テスト環境

OSはUbuntu12.04で、OpenSSHのみをインストールした環境。

admin network
      -------+----------------------------------------+-----
             |                                        |
             |                                        |
        eth0 | 192.168.128.100                   eth0 | 192.168.128.101
      +------+---------+                       +------+-------+
      |quantum-agent   |                       |quantum-agent |
      |quantum-l3      |                       |nova-compute  |
      |quantum-dhcp    |                       |              |
      |quantum-server  |                       |              |
      |nova-compute    |                       |              |
      |nova-api        |                       |              |
      |nova-scheduler  |                       |              |
      |nova-cert       |172.26.0.100           |              |
      |nova-console    |eth1                   |              |
      |nova-consoleauth+-----------------------+              |
      |glance-api      |                   eth1|              |
      |glance-registry |           172.26.0.101|              |
      |mysql           |                       |              |
      |rabbit          |                       |              |
      +-----+----------+                       +------+-------+
       eth2 |                                    eth2 |
            |                                         |
            |                                         |
     -------+--------------------+--------------------+-----
     public network              |
                                 |10.0.0.254
                             +---+---+
                             |   R   |
                             +-------+

eth0 は管理ネットワークとしてOpenStackの内部通信に利用
eth1 仮想マシン間の通信はこのネットワーク上のGREのトンネル内で行われる。
eth2 仮想マシンの外側向けの出口、FLOATING IPが割り当てられるネットワーク。


2台のネットワーク構成は以下、
# admin network
auto eth0
iface eth0 inet static
address 192.168.128.100
netmask 255.255.255.0
gateway 192.168.128.1
dns-nameservers 192.168.128.1

# VMs internal
auto eth1
iface eth1 inet static
address 172.26.0.100
netmask 255.255.255.0

# public
auto eth2
iface eth2 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down

# admin network
auto eth0
iface eth0 inet static
address 192.168.128.101
netmask 255.255.255.0
gateway 192.168.128.1
dns-nameservers 192.168.128.1

# VMs internal
auto eth1
iface eth1 inet static
address 172.26.0.101
netmask 255.255.255.0

# public
auto eth2
iface eth2 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down


準備

■devstackの取得まで
$ sudo apt-get update
$ sudo apt-get install -qqy git
$ git clone http://github.com/openstack-dev/devstack.git

■何度もDevstackを実行する際に、パッケージを毎回取得するのが面倒なので事前にインストールを済ませておく。
$ cd devstack/files/apt
$ for i in `cat * |sed -e "s/ #.*$//g"|grep -v ^# |sort |uniq |grep -v mongo |grep -v mysql |grep -v rabbit |grep -v apache |grep -v qpid`; do sudo apt-get install -qqy $i; done

■余分なネットワークを削除
$ sudo virsh net-destroy default
$ sudo virsh net-undefine default

$ sudo reboot

コントローノードの構築

■localrcの作成
# 自ホストIP(ADMIN)
HOST_IP=192.168.128.100

ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=admintoken

disable_service n-net
disable_service n-obj
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3

# GREのトンネリングを有効にする
ENABLE_TENANT_TUNNELS=True

FIXED_RANGE=172.24.17.0/24
NETWORK_GATEWAY=172.24.17.254
FLOATING_RANGE=10.0.0.0/24

# GREのトンネルを張るために使うIP
Q_LOCAL_IP=172.26.0.100
Q_LOCAL_IP は勝手に定義している。


上記の変数を使うように stack.sh を少し編集
--- stack.a.sh  2012-10-15 22:45:10.877090405 +0900
--- stack.sh    2012-10-16 20:20:09.865657008 +0900
+++ stack.a.sh  2012-10-16 20:19:29.349657324 +0900
@@ -1336,7 +1336,7 @@
                 exit 1
             fi
             iniset /$Q_PLUGIN_CONF_FILE OVS enable_tunneling True
-            iniset /$Q_PLUGIN_CONF_FILE OVS local_ip $Q_LOCAL_IP
+            iniset /$Q_PLUGIN_CONF_FILE OVS local_ip $HOST_IP
         fi

         # Setup physical network bridge mappings.  Override
Q_EXT_GW_IP は quantum の /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini で指定される local_ip に設定するIP。これがGREのエンドポイントとして使われるが、標準は $HOST_IPが入ってしまう。


■インストールの実施
$ ./stack.sh

■テストしやすい用にping/sshを許可しておく(demo テナントに対して行われる)
$ source openrc
$ nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
$ nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
$ nova secgroup-list-rules default
HOST_IP=192.168.128.101
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
+-------------+-----------+---------+-----------+--------------+


2台目のノードを追加

■localrcの作成
HOST_IP=192.168.128.101

ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=admintoken

ENABLED_SERVICES=n-cpu,rabbit,g-api,quantum,q-agt
SERVICE_HOST=192.168.128.100
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
Q_HOST=$SERVICE_HOST

ENABLE_TENANT_TUNNELS=True

Q_LOCAL_IP=172.26.0.101
rabbit,g-api は使ってはいないが、指定しないと動かない(?) → こちらを参考にしている

Q_LOCAL_IP は コントローラーと同じ理由で指定。

同じく、 stack.sh を編集。
--- stack.a.sh  2012-10-15 22:45:10.877090405 +0900
+++ stack.sh    2012-10-15 22:45:26.045092578 +0900
@@ -1336,7 +1336,7 @@
                 exit 1
             fi
             iniset /$Q_PLUGIN_CONF_FILE OVS enable_tunneling True
-            iniset /$Q_PLUGIN_CONF_FILE OVS local_ip $HOST_IP
+            iniset /$Q_PLUGIN_CONF_FILE OVS local_ip $Q_LOCAL_IP
         fi
 
         # Setup physical network bridge mappings.  Override

■インストールの実施
$ ./stack.sh


■状態
$ nova-manage service list
Binary           Host          Zone  Status    State Updated_At
nova-cert        devstack-cc   nova  enabled   :-)   2012-10-15 16:49:06
nova-compute     devstack-cc   nova  enabled   :-)   2012-10-15 16:49:14
nova-scheduler   devstack-cc   nova  enabled   :-)   2012-10-15 16:49:14
nova-consoleauth devstack-cc   nova  enabled   :-)   2012-10-15 16:49:10
nova-compute     devstack-node nova  enabled   :-)   2012-10-15 16:49:08


構築直後のネットワークの状態

■OVS
devstack-cc:~$ sudo ovs-vsctl show
70d88f15-7f24-4fac-a509-3f3c30533cdb
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-2"
            Interface "gre-2"
                type: gre
                options: {in_key=flow, out_key=flow, remote_ip="172.26.0.101"}
    Bridge br-int
        Port "tapa9806132-f2"
            tag: 1
            Interface "tapa9806132-f2"
                type: internal
        Port "qr-01b9a7bf-ce"
            tag: 1
            Interface "qr-01b9a7bf-ce"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-f616aed6-66"
            Interface "qg-f616aed6-66"
                type: internal
    ovs_version: "1.4.0+build0"

devstack-node:~$ sudo ovs-vsctl show
52a9d7d5-117d-49ed-8f79-ea4f43092994
    Bridge br-int
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port "gre-1"
            Interface "gre-1"
                type: gre
                options: {in_key=flow, out_key=flow, remote_ip="172.26.0.100"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    ovs_version: "1.4.0+build0"

■コントローラーのIPアドレス
devsack-cc$ sudo ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:06:4e:63 brd ff:ff:ff:ff:ff:ff
    inet 192.168.128.100/24 brd 192.168.128.255 scope global eth0
    inet6 fe80::5054:ff:fe06:4e63/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:c5:1c:e5 brd ff:ff:ff:ff:ff:ff
    inet 172.26.0.100/24 brd 172.26.0.255 scope global eth1
    inet6 fe80::5054:ff:fec5:1ce5/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:ba:e9:55 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:feba:e955/64 scope link 
       valid_lft forever preferred_lft forever
7: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 7a:03:9b:7a:a1:4e brd ff:ff:ff:ff:ff:ff
8: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 52:89:c7:7d:ed:4f brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 scope global br-ex
    inet6 fe80::5089:c7ff:fe7d:ed4f/64 scope link 
       valid_lft forever preferred_lft forever
10: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 9a:7e:07:5d:f9:44 brd ff:ff:ff:ff:ff:ff

■ネットワークネームスペース(netns)内のアドレス
L3/DHCPエージェントが稼働するノードでは、netnsで区切られた空間にもアドレスが割り当てられている。
devsack-cc$ for i in `sudo ip netns`; do echo; echo; echo ----- $i -----; sudo ip netns exec $i ip addr list; done
----- qrouter-34d4f254-4bfe-4f28-9ad5-e762e7014e6f -----
13: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
14: qr-01b9a7bf-ce: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:54:79:55 brd ff:ff:ff:ff:ff:ff
    inet 172.24.17.254/24 brd 172.24.17.255 scope global qr-01b9a7bf-ce
    inet6 fe80::f816:3eff:fe54:7955/64 scope link 
       valid_lft forever preferred_lft forever
15: qg-f616aed6-66: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:ed:a6:2d brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 brd 10.0.0.255 scope global qg-f616aed6-66
    inet6 fe80::f816:3eff:feed:a62d/64 scope link 
       valid_lft forever preferred_lft forever


----- qdhcp-cb2f35e8-2ac7-493d-8089-a302e3be5cee -----
11: tapa9806132-f2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:56:12:af brd ff:ff:ff:ff:ff:ff
    inet 172.24.17.1/24 brd 172.24.17.255 scope global tapa9806132-f2
    inet6 fe80::f816:3eff:fe56:12af/64 scope link 
       valid_lft forever preferred_lft forever
12: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever


■追加ノードのIPアドレス
devstack-node1:~$ sudo ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:5b:b1:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.128.101/24 brd 192.168.128.255 scope global eth0
    inet6 fe80::5054:ff:fe5b:b103/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:6f:e7:54 brd ff:ff:ff:ff:ff:ff
    inet 172.26.0.101/24 brd 172.26.0.255 scope global eth1
    inet6 fe80::5054:ff:fe6f:e754/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:97:3d:67 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe97:3d67/64 scope link 
       valid_lft forever preferred_lft forever
7: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 3a:b3:dd:6d:30:49 brd ff:ff:ff:ff:ff:ff
9: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 12:05:5c:cc:b8:4c brd ff:ff:ff:ff:ff:ff


■ネットワーク構成の可視化
上記のネットワークを図解すると以下のようになる。


quantumの設定

上記の図を見ながら、Quantumの設定を見ていく。


トンネル構成について

トンネルの管理はquantumによって自動で行われている。

$ sudo mysql -uroot -e "use ovs_quantum; show tables;"
+------------------------+
| Tables_in_ovs_quantum  |
+------------------------+
| dnsnameservers         |
| externalnetworks       |
| floatingips            |
| ipallocationpools      |
| ipallocations          |
| ipavailabilityranges   |
| networks               |
| ovs_network_bindings   |
| ovs_tunnel_allocations |
| ovs_tunnel_endpoints   |
| ovs_tunnel_ips         |
| ovs_vlan_allocations   |
| ports                  |
| routers                |
| routes                 |
| subnets                |
+------------------------+

quantum-agent起動時に、/etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini に記載された、
[OVS]
local_ip = 172.26.0.100

このアドレスを、ovs_tunnel_endpoints へ登録する。

$ sudo mysql -uroot -e "use ovs_quantum; select * from ovs_tunnel_endpoints;"
+--------------+----+
| ip_address   | id |
+--------------+----+
| 172.26.0.100 |  1 |
| 172.26.0.101 |  2 |
+--------------+----+

*このアドレスは編集する方法が今の所DBを直接変更するしかなさそう?

そして複数のエンドポイントを持つ場合、起動時にノード間のトンネルが構築される。
(トンネルは作成はされると永続化され、アドレスを変えても古いインターフェースが残るので、local_ipを編集した場合は手動で削除する)

トンネルの数には上限があり、以下の設定で決まっている(GRE自体のトンネルに上限があるらしい??・・・未調査)
[OVS]
enable_tunneling = True
tunnel_id_ranges = 1:1000

使用済みのトンネルはマークされる。
$ sudo mysql -uroot -e "use ovs_quantum; select * from ovs_tunnel_allocations;"
+-----------+-----------+
| tunnel_id | allocated |
+-----------+-----------+
|         1 |         1 |
|         2 |         1 |
|         3 |         0 |
|         4 |         0 |
|         5 |         0 |
|         6 |         0 |
|         7 |         0 |
|         8 |         0 |
|         9 |         0 |
|        10 |         0 |
~~~~~~~~~~~~~~
|       993 |         0 |
|       994 |         0 |
|       995 |         0 |
|       996 |         0 |
|       997 |         0 |
|       998 |         0 |
|       999 |         0 |
|      1000 |         0 |
+-----------+-----------+


定義されているネットワーク

adminユーザに切り替える(全ネットワークを参照できるので)
$ export OS_TENANT_NAME=admin
$ export OS_USERNAME=admin

$ quantum net-list
+--------------------------------------+---------+--------------------------------------+
| id                                   | name    | subnets                              |
+--------------------------------------+---------+--------------------------------------+
| 2888da00-4060-4d2e-979b-d0f86390c76b | ext_net | 9975adc4-8e2f-4152-a68c-b5b3137c0fab |
| cb2f35e8-2ac7-493d-8089-a302e3be5cee | net1    | 7c10a3de-8e1f-4604-a243-ebedd51bf0ae |
+--------------------------------------+---------+--------------------------------------+

$ quantum net-show cb2f35e8-2ac7-493d-8089-a302e3be5cee
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | cb2f35e8-2ac7-493d-8089-a302e3be5cee |
| name                      | net1                                 |
| provider:network_type     | gre                                  |
| provider:physical_network |                                      |
| provider:segmentation_id  | 1                                    |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 7c10a3de-8e1f-4604-a243-ebedd51bf0ae |
| tenant_id                 | 215c241925f543a1a69b3013b474fdd9     |
+---------------------------+--------------------------------------+

$ quantum net-show 2888da00-4060-4d2e-979b-d0f86390c76b
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 2888da00-4060-4d2e-979b-d0f86390c76b |
| name                      | ext_net                              |
| provider:network_type     | gre                                  |
| provider:physical_network |                                      |
| provider:segmentation_id  | 2                                    |
| router:external           | True                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 9975adc4-8e2f-4152-a68c-b5b3137c0fab |
| tenant_id                 | 30927bc975614cdc929ee8ec645d0a21     |
+---------------------------+--------------------------------------+


定義されているサブネット

定義されたネットワークに対して、サブネットが割り当てられる。
サブネットはDHCPが割当てるレンジと、GATEWAYのアドレスが定義されている。

$ quantum subnet-list
+--------------------------------------+------+----------------+--------------------------------------------------+
| id                                   | name | cidr           | allocation_pools                                 |
+--------------------------------------+------+----------------+--------------------------------------------------+
| 7c10a3de-8e1f-4604-a243-ebedd51bf0ae |      | 172.24.17.0/24 | {"start": "172.24.17.1", "end": "172.24.17.253"} |
| 9975adc4-8e2f-4152-a68c-b5b3137c0fab |      | 10.0.0.0/24    | {"start": "10.0.0.2", "end": "10.0.0.254"}       |
+--------------------------------------+------+----------------+--------------------------------------------------+

$ quantum subnet-show 7c10a3de-8e1f-4604-a243-ebedd51bf0ae
+------------------+--------------------------------------------------+
| Field            | Value                                            |
+------------------+--------------------------------------------------+
| allocation_pools | {"start": "172.24.17.1", "end": "172.24.17.253"} |
| cidr             | 172.24.17.0/24                                   |
| dns_nameservers  |                                                  |
| enable_dhcp      | True                                             |
| gateway_ip       | 172.24.17.254                                    |
| host_routes      |                                                  |
| id               | 7c10a3de-8e1f-4604-a243-ebedd51bf0ae             |
| ip_version       | 4                                                |
| name             |                                                  |
| network_id       | cb2f35e8-2ac7-493d-8089-a302e3be5cee             |
| tenant_id        | 215c241925f543a1a69b3013b474fdd9                 |
+------------------+--------------------------------------------------+

$ quantum subnet-show 9975adc4-8e2f-4152-a68c-b5b3137c0fab
+------------------+--------------------------------------------+
| Field            | Value                                      |
+------------------+--------------------------------------------+
| allocation_pools | {"start": "10.0.0.2", "end": "10.0.0.254"} |
| cidr             | 10.0.0.0/24                                |
| dns_nameservers  |                                            |
| enable_dhcp      | False                                      |
| gateway_ip       | 10.0.0.1                                   |
| host_routes      |                                            |
| id               | 9975adc4-8e2f-4152-a68c-b5b3137c0fab       |
| ip_version       | 4                                          |
| name             |                                            |
| network_id       | 2888da00-4060-4d2e-979b-d0f86390c76b       |
| tenant_id        | 30927bc975614cdc929ee8ec645d0a21           |
+------------------+--------------------------------------------+


定義されているポート

サブネットに紐付く形で、OVSにアタッチされるポートが定義されている。ここで定義されたポートはquantumによって作成され、アドレスが割り当てられている。
$ quantum port-list
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                            |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| 01b9a7bf-ce0b-4146-b142-f2c99886de0b |      | fa:16:3e:54:79:55 | {"subnet_id": "7c10a3de-8e1f-4604-a243-ebedd51bf0ae", "ip_address": "172.24.17.254"} |
| a9806132-f2a7-4156-aec2-ed33264896c3 |      | fa:16:3e:56:12:af | {"subnet_id": "7c10a3de-8e1f-4604-a243-ebedd51bf0ae", "ip_address": "172.24.17.1"}   |
| f616aed6-668b-4344-8401-a5a93f3d0d04 |      | fa:16:3e:ed:a6:2d | {"subnet_id": "9975adc4-8e2f-4152-a68c-b5b3137c0fab", "ip_address": "10.0.0.2"}      |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+

$ quantum port-show 01b9a7bf-ce0b-4146-b142-f2c99886de0b
+----------------+--------------------------------------------------------------------------------------+
| Field          | Value                                                                                |
+----------------+--------------------------------------------------------------------------------------+
| admin_state_up | True                                                                                 |
| device_id      | 34d4f254-4bfe-4f28-9ad5-e762e7014e6f                                                 |
| device_owner   | network:router_interface                                                             |
| fixed_ips      | {"subnet_id": "7c10a3de-8e1f-4604-a243-ebedd51bf0ae", "ip_address": "172.24.17.254"} |
| id             | 01b9a7bf-ce0b-4146-b142-f2c99886de0b                                                 |
| mac_address    | fa:16:3e:54:79:55                                                                    |
| name           |                                                                                      |
| network_id     | cb2f35e8-2ac7-493d-8089-a302e3be5cee                                                 |
| status         | ACTIVE                                                                               |
| tenant_id      | 215c241925f543a1a69b3013b474fdd9                                                     |
+----------------+--------------------------------------------------------------------------------------+

$ quantum port-show a9806132-f2a7-4156-aec2-ed33264896c3
+----------------+------------------------------------------------------------------------------------+
| Field          | Value                                                                              |
+----------------+------------------------------------------------------------------------------------+
| admin_state_up | True                                                                               |
| device_id      | dhcp7d55f09b-84d7-5822-983f-c67cc401290b-cb2f35e8-2ac7-493d-8089-a302e3be5cee      |
| device_owner   | network:dhcp                                                                       |
| fixed_ips      | {"subnet_id": "7c10a3de-8e1f-4604-a243-ebedd51bf0ae", "ip_address": "172.24.17.1"} |
| id             | a9806132-f2a7-4156-aec2-ed33264896c3                                               |
| mac_address    | fa:16:3e:56:12:af                                                                  |
| name           |                                                                                    |
| network_id     | cb2f35e8-2ac7-493d-8089-a302e3be5cee                                               |
| status         | ACTIVE                                                                             |
| tenant_id      | 215c241925f543a1a69b3013b474fdd9                                                   |
+----------------+------------------------------------------------------------------------------------+

$ quantum port-show f616aed6-668b-4344-8401-a5a93f3d0d04
+----------------+---------------------------------------------------------------------------------+
| Field          | Value                                                                           |
+----------------+---------------------------------------------------------------------------------+
| admin_state_up | True                                                                            |
| device_id      | 34d4f254-4bfe-4f28-9ad5-e762e7014e6f                                            |
| device_owner   | network:router_gateway                                                          |
| fixed_ips      | {"subnet_id": "9975adc4-8e2f-4152-a68c-b5b3137c0fab", "ip_address": "10.0.0.2"} |
| id             | f616aed6-668b-4344-8401-a5a93f3d0d04                                            |
| mac_address    | fa:16:3e:ed:a6:2d                                                               |
| name           |                                                                                 |
| network_id     | 2888da00-4060-4d2e-979b-d0f86390c76b                                            |
| status         | ACTIVE                                                                          |
| tenant_id      |                                                                                 |
+----------------+---------------------------------------------------------------------------------+

割り当てられたポートとアドレスは以下。

10.0.0.1 はdevstackのスクリプト stack.sh の中で割り当てられている。

stack.sh ログ
+ sudo ip addr add 10.0.0.1/24 dev br-ex
+ sudo ip link set br-ex up
++ awk -F '"' '{ print $8; }'
++ grep router_gateway
++ quantum port-list -c fixed_ips -c device_owner
+ ROUTER_GW_IP=10.0.0.2
+ sudo route add -net 172.24.17.0/24 gw 10.0.0.2

この理由は後で説明する。


仮想マシンを作成した時の変化

demoテナントで実施
$ export OS_USERNAME=demo
$ export OS_TENANT_NAME=demo

■仮想マシンをそれぞれのホストへ1台づつ追加する。

$ nova boot --flavor m1.tiny --image 69989707-d9ca-4905-aff7-39e3b025d704 --security-groups default --nic net-id=cb2f35e8-2ac7-493d-8089-a302e3be5cee testvm01
+------------------------+--------------------------------------+
| Property               | Value                                |
+------------------------+--------------------------------------+
| OS-DCF:diskConfig      | MANUAL                               |
| OS-EXT-STS:power_state | 0                                    |
| OS-EXT-STS:task_state  | scheduling                           |
| OS-EXT-STS:vm_state    | building                             |
| accessIPv4             |                                      |
| accessIPv6             |                                      |
| adminPass              | 5fUEKnrwSAej                         |
| config_drive           |                                      |
| created                | 2012-10-16T14:25:47Z                 |
| flavor                 | m1.tiny                              |
| hostId                 |                                      |
| id                     | 0d001509-b6ee-4657-a9a5-f090a6e37b3f |
| image                  | cirros-0.3.0-x86_64-uec              |
| key_name               | None                                 |
| metadata               | {}                                   |
| name                   | testvm01                             |
| progress               | 0                                    |
| security_groups        | [{u'name': u'default'}]              |
| status                 | BUILD                                |
| tenant_id              | 215c241925f543a1a69b3013b474fdd9     |
| updated                | 2012-10-16T14:25:48Z                 |
| user_id                | 9123dc08c8404ecd9cc5b5359bde48cc     |
+------------------------+--------------------------------------+

$ nova boot --flavor m1.tiny --image 69989707-d9ca-4905-aff7-39e3b025d704 --security-groups default --nic net-id=cb2f35e8-2ac7-493d-8089-a302e3be5cee testvm02
+------------------------+--------------------------------------+
| Property               | Value                                |
+------------------------+--------------------------------------+
| OS-DCF:diskConfig      | MANUAL                               |
| OS-EXT-STS:power_state | 0                                    |
| OS-EXT-STS:task_state  | scheduling                           |
| OS-EXT-STS:vm_state    | building                             |
| accessIPv4             |                                      |
| accessIPv6             |                                      |
| adminPass              | QiYVd6vETTAd                         |
| config_drive           |                                      |
| created                | 2012-10-16T14:27:05Z                 |
| flavor                 | m1.tiny                              |
| hostId                 |                                      |
| id                     | b18485e5-247d-4a47-b76f-56634303eaf3 |
| image                  | cirros-0.3.0-x86_64-uec              |
| key_name               | None                                 |
| metadata               | {}                                   |
| name                   | testvm02                             |
| progress               | 0                                    |
| security_groups        | [{u'name': u'default'}]              |
| status                 | BUILD                                |
| tenant_id              | 215c241925f543a1a69b3013b474fdd9     |
| updated                | 2012-10-16T14:27:05Z                 |
| user_id                | 9123dc08c8404ecd9cc5b5359bde48cc     |
+------------------------+--------------------------------------+

$ nova list
+--------------------------------------+----------+--------+------------------+
| ID                                   | Name     | Status | Networks         |
+--------------------------------------+----------+--------+------------------+
| 0d001509-b6ee-4657-a9a5-f090a6e37b3f | testvm01 | ACTIVE | net1=172.24.17.2 |
| b18485e5-247d-4a47-b76f-56634303eaf3 | testvm02 | ACTIVE | net1=172.24.17.3 |
+--------------------------------------+----------+--------+------------------+

起動ログ。metadataサーバへもアクセスできていることが確認できる。

$ nova console-log --length 21 0d001509-b6ee-4657-a9a5-f090a6e37b3f
cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id
cloud-setup: successful after 1/30 tries: up 54.21. iid=i-00000001
wget: server returned error: HTTP/1.1 404 Not Found
failed to get http://169.254.169.254/latest/meta-data/public-keys
Starting dropbear sshd: generating rsa key... generating dsa key... OK
===== cloud-final: system completely up in 72.77 seconds ====
  instance-id: i-00000001
  public-ipv4: 
  local-ipv4 : 172.24.17.2
wget: server returned error: HTTP/1.1 404 Not Found
cloud-userdata: failed to read user data url: http://169.254.169.254/2009-04-04/user-data
WARN: /etc/rc3.d/S99-cloud-userdata failed
  ____               ____  ____
 / __/ __ ____ ____ / __ \/ __/
/ /__ / // __// __// /_/ /\ \ 
\___//_//_/  /_/   \____/___/ 
 http://launchpad.net/cirros


login as 'cirros' user. default password: 'cubswin:)'. use 'sudo' for root.
cirros login: 

$ nova console-log --length 21 b18485e5-247d-4a47-b76f-56634303eaf3
cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id
cloud-setup: successful after 1/30 tries: up 26.06. iid=i-00000002
wget: server returned error: HTTP/1.1 404 Not Found
failed to get http://169.254.169.254/latest/meta-data/public-keys
Starting dropbear sshd: generating rsa key... generating dsa key... OK
===== cloud-final: system completely up in 32.61 seconds ====
  instance-id: i-00000002
  public-ipv4: 
  local-ipv4 : 172.24.17.3
wget: server returned error: HTTP/1.1 404 Not Found
cloud-userdata: failed to read user data url: http://169.254.169.254/2009-04-04/user-data
WARN: /etc/rc3.d/S99-cloud-userdata failed
  ____               ____  ____
 / __/ __ ____ ____ / __ \/ __/
/ /__ / // __// __// /_/ /\ \ 
\___//_//_/  /_/   \____/___/ 
 http://launchpad.net/cirros


login as 'cirros' user. default password: 'cubswin:)'. use 'sudo' for root.
cirros login: 


■この状態のOVS

devstack-cc$ sudo ovs-vsctl show
70d88f15-7f24-4fac-a509-3f3c30533cdb
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-2"
            Interface "gre-2"
                type: gre
                options: {in_key=flow, out_key=flow, remote_ip="172.26.0.101"}
    Bridge br-int
        Port "tapa9806132-f2"
            tag: 1
            Interface "tapa9806132-f2"
                type: internal
        Port "qr-01b9a7bf-ce"
            tag: 1
            Interface "qr-01b9a7bf-ce"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
        Port "qvo00a6d081-61"
            tag: 1
            Interface "qvo00a6d081-61"
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-f616aed6-66"
            Interface "qg-f616aed6-66"
                type: internal
    ovs_version: "1.4.0+build0"

devstack-node$ sudo ovs-vsctl show
52a9d7d5-117d-49ed-8f79-ea4f43092994
    Bridge br-int
        Port "qvodd945b9a-4a"
            tag: 1
            Interface "qvodd945b9a-4a"
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port "gre-1"
            Interface "gre-1"
                type: gre
                options: {in_key=flow, out_key=flow, remote_ip="172.26.0.100"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    ovs_version: "1.4.0+build0"

いくつかのポートが追加されているのが確認できる。この接続を図解すると以下になる。



新しく br-int に追加されたポートから、TAPインターフェースを経由して、仮想マシンと接続されている。
(確か理由があったけど、忘れた・・・)


■仮想マシンの疎通

コントローラーのルーティング情報

$ sudo route -n
カーネルIP経路テーブル
受信先サイト    ゲートウェイ    ネットマスク   フラグ Metric Ref 使用数 インタフェース
0.0.0.0         192.168.128.1   0.0.0.0         UG    100    0        0 eth0
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 br-ex
172.24.17.0     10.0.0.2        255.255.255.0   UG    0      0        0 br-ex
172.26.0.0      0.0.0.0         255.255.255.0   U     0      0        0 eth1
192.168.128.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0

$ for i in `sudo ip netns`; do echo;echo; echo ----- $i -----;sudo ip netns exec $i route -n; done
----- qrouter-34d4f254-4bfe-4f28-9ad5-e762e7014e6f -----
カーネルIP経路テーブル
受信先サイト    ゲートウェイ    ネットマスク   フラグ Metric Ref 使用数 インタフェース
0.0.0.0         10.0.0.1        0.0.0.0         UG    0      0        0 qg-f616aed6-66
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 qg-f616aed6-66
172.24.17.0     0.0.0.0         255.255.255.0   U     0      0        0 qr-01b9a7bf-ce

----- qdhcp-cb2f35e8-2ac7-493d-8089-a302e3be5cee -----
カーネルIP経路テーブル
受信先サイト    ゲートウェイ    ネットマスク   フラグ Metric Ref 使用数 インタフェース
172.24.17.0     0.0.0.0         255.255.255.0   U     0      0        0 tapa9806132-f2

$ ping 172.24.17.2
PING 172.24.17.2 (172.24.17.2) 56(84) bytes of data.
64 bytes from 172.24.17.2: icmp_req=1 ttl=63 time=141 ms
64 bytes from 172.24.17.2: icmp_req=2 ttl=63 time=0.501 ms
64 bytes from 172.24.17.2: icmp_req=3 ttl=63 time=0.533 ms
64 bytes from 172.24.17.2: icmp_req=4 ttl=63 time=0.515 ms
^C
--- 172.24.17.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.501/35.789/141.609/61.095 ms

この通信はやや複雑だが、以下の経路で通信できている。
仮想マシンのデフォルトGWは172.24.17.254なので、仮想マシンからの通信は逆の経路で戻ってくる。



devstack が br-ex にアドレスを割り当てて、ルーティングを追加しているので、仮想マシンがこの経路を使ってping 応答の返信や、Metadataサーバにアクセスできる。
(これはアクセス制限という意味では望ましくない)

$ for i in `sudo ip netns`; do echo;echo; echo ----- $i -----;sudo ip netns exec $i iptables -nvL -t nat; done
----- qrouter-34d4f254-4bfe-4f28-9ad5-e762e7014e6f -----
Chain PREROUTING (policy ACCEPT 71 packets, 20248 bytes)
 pkts bytes target     prot opt in     out     source               destination
   85 21088 quantum-l3-agent-PREROUTING
                       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain INPUT (policy ACCEPT 65 packets, 19744 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 quantum-l3-agent-OUTPUT
                       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   20  1344 quantum-l3-agent-POSTROUTING
                       all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 quantum-postrouting-bottom
                       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain quantum-l3-agent-OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain quantum-l3-agent-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination
    6   504 ACCEPT     all  --  !qg-f616aed6-66 !qg-f616aed6-66
                                       0.0.0.0/0
                                               0.0.0.0/0            ! ctstate DNAT
   14   840 ACCEPT     all  --  *      *       172.24.17.0/24       192.168.128.100

Chain quantum-l3-agent-PREROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination
   14   840 DNAT       tcp  --  *      *       0.0.0.0/0            169.254.169.254      tcp dpt:80 to:192.168.128.100:8775

Chain quantum-l3-agent-float-snat (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain quantum-l3-agent-snat (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 quantum-l3-agent-float-snat
                       all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 SNAT       all  --  *      *       172.24.17.0/24       0.0.0.0/0            to:10.0.0.2

Chain quantum-postrouting-bottom (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 quantum-l3-agent-snat
                       all  --  *      *       0.0.0.0/0            0.0.0.0/0


----- qdhcp-cb2f35e8-2ac7-493d-8089-a302e3be5cee -----
Chain PREROUTING (policy ACCEPT 65 packets, 19744 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 65 packets, 19744 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 121 packets, 42229 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 121 packets, 42229 bytes)
 pkts bytes target     prot opt in     out     source               destination

■仮想マシン間の通信

普通にホスト側へも、トンネルを介した通信も可能。

$ ssh cirros@172.24.17.2
The authenticity of host '172.24.17.2 (172.24.17.2)' can't be established.
RSA key fingerprint is 4b:c8:13:82:c2:f7:4b:da:89:7d:b5:f9:d4:66:92:48.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.24.17.2' (RSA) to the list of known hosts.
cirros@172.24.17.2's password:
$

$ hostname
cirros

$ ifconfig
eth0      Link encap:Ethernet  HWaddr FA:16:3E:11:74:6E
          inet addr:172.24.17.2  Bcast:172.24.17.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe11:746e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:507 errors:0 dropped:0 overruns:0 frame:0
          TX packets:377 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:92682 (90.5 KiB)  TX bytes:70682 (69.0 KiB)
          Interrupt:11
  
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.24.17.254   0.0.0.0         UG    0      0        0 eth0
172.24.17.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0

$ ping 172.24.17.3
PING 172.24.17.3 (172.24.17.3): 56 data bytes
64 bytes from 172.24.17.3: seq=0 ttl=64 time=15.355 ms
64 bytes from 172.24.17.3: seq=1 ttl=64 time=1.901 ms
64 bytes from 172.24.17.3: seq=2 ttl=64 time=2.966 ms
64 bytes from 172.24.17.3: seq=3 ttl=64 time=1.776 ms

$ ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: seq=0 ttl=63 time=6.603 ms
64 bytes from 10.0.0.1: seq=1 ttl=63 time=1.215 ms

$ ping 172.26.0.100
PING 172.26.0.100 (172.26.0.100): 56 data bytes
64 bytes from 172.26.0.100: seq=0 ttl=63 time=14.689 ms
64 bytes from 172.26.0.100: seq=1 ttl=63 time=1.210 ms

$ ping 192.168.128.100
PING 192.168.128.100 (192.168.128.100): 56 data bytes
64 bytes from 192.168.128.100: seq=0 ttl=63 time=9.672 ms
64 bytes from 192.168.128.100: seq=1 ttl=63 time=1.171 ms
64 bytes from 192.168.128.100: seq=2 ttl=63 time=1.141 ms

仮想ネットワークの追加

仮想マシンを削除しておく。
$ nova list
+--------------------------------------+----------+--------+------------------+
| ID                                   | Name     | Status | Networks         |
+--------------------------------------+----------+--------+------------------+
| 0d001509-b6ee-4657-a9a5-f090a6e37b3f | testvm01 | ACTIVE | net1=172.24.17.2 |
| b18485e5-247d-4a47-b76f-56634303eaf3 | testvm02 | ACTIVE | net1=172.24.17.3 |
+--------------------------------------+----------+--------+------------------+
$ nova delete 0d001509-b6ee-4657-a9a5-f090a6e37b3f
$ nova delete b18485e5-247d-4a47-b76f-56634303eaf3

■1つ目のネットワークを追加
$ quantum net-create add_net1
Created a new network:
+-----------------+--------------------------------------+
| Field           | Value                                |
+-----------------+--------------------------------------+
| admin_state_up  | True                                 |
| id              | b91768ec-84e2-4741-b254-a41f5bc43919 |
| name            | add_net1                             |
| router:external | False                                |
| shared          | False                                |
| status          | ACTIVE                               |
| subnets         |                                      |
| tenant_id       | 215c241925f543a1a69b3013b474fdd9     |
+-----------------+--------------------------------------+
これだけでは特にネットワークに変化無し。
$ sudo ovs-vsctl show
70d88f15-7f24-4fac-a509-3f3c30533cdb
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-2"
            Interface "gre-2"
                type: gre
                options: {in_key=flow, out_key=flow, remote_ip="172.26.0.101"}
    Bridge br-int
        Port "tapa9806132-f2"
            tag: 1
            Interface "tapa9806132-f2"
                type: internal
        Port "qr-01b9a7bf-ce"
            tag: 1
            Interface "qr-01b9a7bf-ce"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-f616aed6-66"
            Interface "qg-f616aed6-66"
                type: internal
    ovs_version: "1.4.0+build0"

■サブネットを追加
$ quantum subnet-create --ip-version 4 --gateway 172.50.0.254 b91768ec-84e2-4741-b254-a41f5bc43919 172.50.0.0/24
Created a new subnet:
+------------------+------------------------------------------------+
| Field            | Value                                          |
+------------------+------------------------------------------------+
| allocation_pools | {"start": "172.50.0.1", "end": "172.50.0.253"} |
| cidr             | 172.50.0.0/24                                  |
| dns_nameservers  |                                                |
| enable_dhcp      | True                                           |
| gateway_ip       | 172.50.0.254                                   |
| host_routes      |                                                |
| id               | 278814e8-0433-456c-ad8c-e62db9b0ebd0           |
| ip_version       | 4                                              |
| name             |                                                |
| network_id       | b91768ec-84e2-4741-b254-a41f5bc43919           |
| tenant_id        | 215c241925f543a1a69b3013b474fdd9               |
+------------------+------------------------------------------------+

これでOVSのポートが追加される。
$ sudo ovs-vsctl show
$ sudo ovs-vsctl show
70d88f15-7f24-4fac-a509-3f3c30533cdb
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-2"
            Interface "gre-2"
                type: gre
                options: {in_key=flow, out_key=flow, remote_ip="172.26.0.101"}
    Bridge br-int
        Port "tapa9806132-f2"
            tag: 1
            Interface "tapa9806132-f2"
                type: internal
        Port "qr-01b9a7bf-ce"
            tag: 1
            Interface "qr-01b9a7bf-ce"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapd0eab10b-57"
            tag: 3
            Interface "tapd0eab10b-57"
                type: internal
        Port br-int
            Interface br-int
                type: internal
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-f616aed6-66"
            Interface "qg-f616aed6-66"
                type: internal
    ovs_version: "1.4.0+build0"

ネットワークネームスペースが追加されている。
$ sudo ip netns |sort
qdhcp-b91768ec-84e2-4741-b254-a41f5bc43919   ← これが追加された
qdhcp-cb2f35e8-2ac7-493d-8089-a302e3be5cee
qrouter-34d4f254-4bfe-4f28-9ad5-e762e7014e6f

追加されたネットワークネームスペースの状態
$ sudo ip netns exec qdhcp-b91768ec-84e2-4741-b254-a41f5bc43919 ip addr list
21: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
23: tapd0eab10b-57: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:78:21:ca brd ff:ff:ff:ff:ff:ff
    inet 172.50.0.1/24 brd 172.50.0.255 scope global tapd0eab10b-57
    inet6 fe80::f816:3eff:fe78:21ca/64 scope link 
       valid_lft forever preferred_lft forever

この状態は以下、




■更にもうひとつのネットワークとサブネットを追加
$ quantum net-create add_net2
Created a new network:
+-----------------+--------------------------------------+
| Field           | Value                                |
+-----------------+--------------------------------------+
| admin_state_up  | True                                 |
| id              | d560db49-30a4-46bc-a60e-9738017eac83 |
| name            | add_net2                             |
| router:external | False                                |
| shared          | False                                |
| status          | ACTIVE                               |
| subnets         |                                      |
| tenant_id       | 215c241925f543a1a69b3013b474fdd9     |
+-----------------+--------------------------------------+

$ quantum subnet-create --ip-version 4 --gateway 172.100.0.254 d560db49-30a4-46bc-a60e-9738017eac83 172.100.0.0/24
Created a new subnet:
+------------------+--------------------------------------------------+
| Field            | Value                                            |
+------------------+--------------------------------------------------+
| allocation_pools | {"start": "172.100.0.1", "end": "172.100.0.253"} |
| cidr             | 172.100.0.0/24                                   |
| dns_nameservers  |                                                  |
| enable_dhcp      | True                                             |
| gateway_ip       | 172.100.0.254                                    |
| host_routes      |                                                  |
| id               | f8be447d-e169-4970-8c33-44dcb1569689             |
| ip_version       | 4                                                |
| name             |                                                  |
| network_id       | d560db49-30a4-46bc-a60e-9738017eac83             |
| tenant_id        | 215c241925f543a1a69b3013b474fdd9                 |
+------------------+--------------------------------------------------+

ブリッジの状態
$ sudo ovs-vsctl show
70d88f15-7f24-4fac-a509-3f3c30533cdb
    Bridge br-tun
        Port br-tun
            Interface br-tun   
                type: internal 
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-2"
            Interface "gre-2"  
                type: gre
                options: {in_key=flow, out_key=flow, remote_ip="172.26.0.101"}
    Bridge br-int
        Port "tap43e85241-7d"  
            tag: 4
            Interface "tap43e85241-7d"
                type: internal 
        Port "tapa9806132-f2"  
            tag: 1
            Interface "tapa9806132-f2"
                type: internal 
        Port "qr-01b9a7bf-ce"  
            tag: 1
            Interface "qr-01b9a7bf-ce"
                type: internal 
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapd0eab10b-57"  
            tag: 3
            Interface "tapd0eab10b-57"
                type: internal 
        Port br-int
            Interface br-int   
                type: internal 
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal 
        Port "qg-f616aed6-66"  
            Interface "qg-f616aed6-66"
                type: internal
    ovs_version: "1.4.0+build0"

追加されたネットワークネームスペース
$ sudo ip netns |sort
qdhcp-b91768ec-84e2-4741-b254-a41f5bc43919
qdhcp-cb2f35e8-2ac7-493d-8089-a302e3be5cee
qdhcp-d560db49-30a4-46bc-a60e-9738017eac83
qrouter-34d4f254-4bfe-4f28-9ad5-e762e7014e6f

ネームスペースの状態
$ sudo ip netns exec qdhcp-d560db49-30a4-46bc-a60e-9738017eac83 ip addr list
24: tap43e85241-7d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:38:6b:bc brd ff:ff:ff:ff:ff:ff
    inet 172.100.0.1/24 brd 172.100.0.255 scope global tap43e85241-7d
    inet6 fe80::f816:3eff:fe38:6bbc/64 scope link 
       valid_lft forever preferred_lft forever
25: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

ネットワークの状態




devstackが追加したネットワークとOVSの状態等が微妙に違うのがわかる。


長くなってきたので続きは次の記事で。

0 件のコメント:

コメントを投稿