How to track networking of instance in openstack ?

In this article, I am going to show you what happen when we are creating a instance on single node openstack (packstack) deployment from neutron perspective.

Step 1 : I have take a backup of the command before creating a instance.

[root@opens1 ~(keystone_admin)]# ovs-vsctl show >> /tmp/before.txt

Step 2 : Once the instance is in active state. I took the backup in new file.

[root@opens1 ~(keystone_admin)]# ovs-vsctl show >> /tmp/after_private.txt

Step 3 : Associated the floating IP with the instance and taken the backup in new file.

[root@opens1 ~(keystone_admin)]# ovs-vsctl show >> /tmp/after_public.txt

Checking the state of instance.

[root@opens1 ~(keystone_admin)]# virsh list
Id    Name                           State
—————————————————-
2     instance-00000005              running

[root@opens1 ~(keystone_admin)]# virsh domiflist 2
Interface  Type       Source     Model       MAC
——————————————————-
tape3702b67-3e bridge     qbre3702b67-3e virtio      fa:16:3e:92:ad:3a

Step 4 : Checking the difference in output of all the backups.

a) First comapring the “/tmp/before.txt” and “/tmp/after_private.txt” output.

[root@opens1 ~(keystone_admin)]# diff /tmp/before.txt /tmp/after_private.txt
15a16,18
>         Port “qvoe3702b67-3e”
>             tag: 1
>             Interface “qvoe3702b67-3e”
[root@opens1 ~(keystone_admin)]# diff /tmp/before.txt /tmp/after_public.txt

b) No difference is present in /tmp/after_private.txt and  /tmp/after_public.txt. Assigning public IP to instance has not changed anything at OVS level.

[root@opens1 ~(keystone_admin)]# diff /tmp/after_private.txt /tmp/after_public.txt

Step 5 : Lets dig deeper on the difference we have seen in the output of Step 4 (a).

a) When we are creating a new instance. It will create a new bridge qvb which will be connected to your OVS using veth pair.

[root@opens1 ~(keystone_admin)]# brctl show
bridge name    bridge id        STP enabled    interfaces
qbre3702b67-3e        8000.2eb4721ca5aa    no        qvbe3702b67-3e
tape3702b67-3e

Setup will look like starting from physical host to instance ethernet.

ens3 —> br-ex (phy-br-ex) —> (int-br-ex) br-int —> (qvoe3702b67-3e) br-int —> (qvbe3702b67-3e) qbre3702b67-3e —>  qbre3702b67-3e (tape3702b67-3e)  —> eth0 (instance)

b) Lets check the MAC address of the interface assigned to instance. Same has been verified using command “virsh domiflist 2”.

# ip a | awk ‘/eth0/ {getline var1; print $0,var1}’
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc pfifo_fast qlen 1000     link/ether fa:16:3e:92:ad:3a brd ff:ff:ff:ff:ff:ff
inet 10.0.0.7/24 brd 10.0.0.255 scope global eth0     inet6 fe80::f816:3eff:fe92:ad3a/64 scope link

Step 6 : As the bridge which is created after launching a instance is connected to br-int. Hence we are checking the ports of br-int to find the MAC addresses associated with them.

a) Checking the ports which are present on br-int.

[root@opens1 images(keystone_admin)]# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000fa5710368941
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
3(int-br-ex): addr:72:64:40:3c:94:9e
config:     0
state:      0
speed: 0 Mbps now, 0 Mbps max
4(patch-tun): addr:f6:a3:bf:f8:56:d7
config:     0
state:      0
speed: 0 Mbps now, 0 Mbps max
5(tap8885a021-43): addr:00:00:00:00:00:00
config:     PORT_DOWN
state:      LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
8(qr-ae7f75fa-85): addr:00:00:00:00:00:00
config:     PORT_DOWN
state:      LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
9(qvoe3702b67-3e): addr:0a:ba:a4:51:fd:c9             <<<< Other pair of veth pair usedconnected to bridge (qbre3702b67-3e), as shown in Step 5 (a).
config:     0
state:      0
current:    10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
LOCAL(br-int): addr:fa:57:10:36:89:41
config:     PORT_DOWN
state:      LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

b) Checking the MAC addreses registered with the ports.

[root@opens1 images(keystone_admin)]# ovs-appctl fdb/show br-int
port  VLAN  MAC                Age
8     1  fa:16:3e:16:0f:e4  198
9     1  fa:16:3e:92:ad:3a  198                   <<<< Mac address of the instance from 5(b) output. (fa:16:3e:92:ad:3a)

Now we know that our instance IP is registered with port 9 on br-int.

[root@opens1 images(keystone_admin)]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=15807.166s, table=0, n_packets=1020, n_bytes=106714, idle_age=381, priority=1 actions=NORMAL
cookie=0x0, duration=15807.041s, table=0, n_packets=611, n_bytes=26785, idle_age=411, priority=2,in_port=3 actions=drop
cookie=0x0, duration=15807.159s, table=23, n_packets=0, n_bytes=0, idle_age=15807, priority=0 actions=drop

Step 7 : If you want to check the connectivity of br-int with br-ex, you can map the same with help of Step 5(a).

a) Issue below command to list the status of external bridge (br-ex) to which our interface is connected.

[root@opens1 images(keystone_admin)]# ovs-ofctl show br-ex
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000c66a17f6fa42
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
1(phy-br-ex): addr:de:49:06:e9:c0:a1                                                                  <<<<<<<<<<<<<< This is connected to Port 3 of br-int. Refer the output 6(a) ==> 3(int-br-ex)
config:     0
state:      0
speed: 0 Mbps now, 0 Mbps max
2(ens3): addr:52:54:00:fe:1c:36                                                                       <<<<<< Physical ethernet connected directly to br-ex.
config:     0
state:      0
current:    100MB-FD AUTO_NEG
advertised: 10MB-HD 10MB-FD 100MB-HD 100MB-FD COPPER AUTO_NEG AUTO_PAUSE
supported:  10MB-HD 10MB-FD 100MB-HD 100MB-FD COPPER AUTO_NEG
speed: 100 Mbps now, 100 Mbps max
5(qg-6f3dc69b-8e): addr:00:00:00:00:00:00                                                        <<<<<< Router other end of veth pair.
config:     PORT_DOWN
state:      LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
LOCAL(br-ex): addr:c6:6a:17:f6:fa:42
config:     0
state:      0
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

b) Checking the namespaces. (Miscellaneous)

[root@opens1 images(keystone_admin)]# ip netns list
qdhcp-42ef38e7-2b55-477c-bafc-3cd5f267e826
qrouter-6f3070e7-ea2a-478e-9e66-c74017a2f749

c) Issuing the command in namespaces.

[root@opens1 images(keystone_admin)]# ip netns exec qrouter-6f3070e7-ea2a-478e-9e66-c74017a2f749 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
11: qr-ae7f75fa-85: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:16:0f:e4 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-ae7f75fa-85
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe16:fe4/64 scope link
valid_lft forever preferred_lft forever
12: qg-6f3dc69b-8e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:f3:ed:09 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.100/24 brd 192.168.122.255 scope global qg-6f3dc69b-8e
valid_lft forever preferred_lft forever
inet 192.168.122.101/32 brd 192.168.122.101 scope global qg-6f3dc69b-8e
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fef3:ed09/64 scope link
valid_lft forever preferred_lft forever

[root@opens1 images(keystone_admin)]# ip netns exec qdhcp-42ef38e7-2b55-477c-bafc-3cd5f267e826 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
10: tap8885a021-43: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:a7:d7:c1 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.2/24 brd 10.0.0.255 scope global tap8885a021-43
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fea7:d7c1/64 scope link
valid_lft forever preferred_lft forever
d) Checking the status of tun bridge.

[root@opens1 images(keystone_admin)]# ovs-ofctl show br-tun
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000ce1b35779b4c
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
2(patch-int): addr:e6:38:3b:ee:b6:52
config:     0
state:      0
speed: 0 Mbps now, 0 Mbps max
LOCAL(br-tun): addr:ce:1b:35:77:9b:4c
config:     PORT_DOWN
state:      LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s