How the vxlan traffic flow in openstack ?

In my previous article, I have shown how to add compute node to existing packstack all-in-one node. Today, I again got a chance to play with openstack setup this time I installed the openstack using three nodes. In which one node is all-in-one setup and other are two compute nodes. You need to make some changes to answer file for doing it.

Step 1 : Once my installation was completed I checked the status of “ovs-vsctl show” on my all machines. I saw two ports appeared as VXLAN interfaces.

From one of the compute node :

[root@compute1 ~]# ovs-vsctl show
6d9ea537-04dd-4a3f-8444-9b9e2d2668d1
Bridge br-tun
fail_mode: secure
Port “vxlan-c0a87aad”
Interface “vxlan-c0a87aad”
type: vxlan
options: {df_default=”true”, in_key=flow, local_ip=”192.168.122.118″, out_key=flow, remote_ip=”192.168.122.173″}
Port “vxlan-c0a87a15”
Interface “vxlan-c0a87a15″
type: vxlan
options: {df_default=”true”, in_key=flow, local_ip=”192.168.122.118″, out_key=flow, remote_ip=”192.168.122.21″}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port br-tun
Interface br-tun
type: internal
Bridge br-int
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port “qvo9c8cdbba-4f”
tag: 1
Interface “qvo9c8cdbba-4f”
ovs_version: “2.4.0”

Step 2 : This time I created two internal networks {private_network (10.10.1.0) and private_network1 (10.10.2.0) }  and one external network { external_network (192.168.122.0) }. These all are connected to same router.

[root@controller ~(keystone_admin)]# neutron net-list
+————————————–+——————+——————————————————-+
| id                                   | name             | subnets                                               |
+————————————–+——————+——————————————————-+
| b05b0579-06e1-41a8-a6e8-58f893482a47 | private_network1 | cdeeb004-72be-44a5-9350-d39725812eb8 10.10.2.0/24     |
| aeac246e-918f-410a-ab8d-9c6bb8ab24ef | private_network  | 7b46a6b0-f83b-40f6-93d1-6bf62ac07c00 10.10.1.0/24     |
| 22053ae2-cc87-4bb9-8c6c-eed3aed78c06 | external_network | 05ca424f-4355-4de2-ae39-f5dab3751e31 192.168.122.0/24 |
+————————————–+——————+——————————————————-+

Step 3 : Booted two instances using private_network on separate compute nodes. As they are on same subnet hence network node will not come into picture whey they are going to communicate.

[root@controller ~(keystone_admin)]# nova list
+————————————–+————+——–+————+————-+—————————-+
| ID                                   | Name       | Status | Task State | Power State | Networks                   |
+————————————–+————+——–+————+————-+—————————-+
| cd027e02-251c-4d5a-8d37-12c04062dcba | test1node1 | ACTIVE | –          | Running     | private_network=10.10.1.6  |
| 257b04dd-d6ab-4e31-a754-75de06f91447 | test2node1 | ACTIVE | –          | Running     | private_network=10.10.1.5  |

Step 4 : I started the ping between these two VMs and started capturing traffic at ens3 interface of compute1 which is bearing the IP 192.168.122.118

I have redirected that traffic in /tmp/tempo.pcap. Let’s start analyzing that file.

a) Filtering out the traffic for vxlan traffic and showing the IP addresses used. These are external IPs that means our VM IP’s are encapsulated by external IP as in physical world switches are not aware of virtual IP.

tshark -tad -n -r /tmp/tempo.pcap -Y ‘udp.dstport == 4789’ -T fields -e ip.src -e ip.dst | sort | uniq -c
38 192.168.122.118    192.168.122.21
38 192.168.122.21    192.168.122.118

b) I have applied the the “decode as” filter in wireshark as I was not able to found the equivalent in tshark.

c) Here we can see that all traffic for VM with in same network goes with VNI : 39

Step 5 : Let’s match this VNI with flow table rules.

a) First covert the decimal value 39 to HEX value.

[root@compute1 ~]# printf “%x\n” 39
27

In hexadecimal it becomes 0x27.

b) We can filter the rules for this particular VNI.

[root@compute1 ~]# ovs-ofctl dump-flows br-tun | grep ‘0x27’
cookie=0x0, duration=9251.432s, table=4, n_packets=4288, n_bytes=407839, idle_age=2429, priority=1,tun_id=0x27 actions=mod_vlan_vid:1,resubmit(,10)
cookie=0x0, duration=9251.436s, table=22, n_packets=71, n_bytes=4242, idle_age=2435, dl_vlan=1 actions=strip_vlan,set_tunnel:0x27,output:2,output:3

c) Checking what is connected port 2 and port3 of br-tun. These are VXLAN interfaces.

[root@compute1 ~]# ovs-ofctl show br-tun | grep “^ [0-9]”
1(patch-int): addr:32:05:e7:fa:10:0a
2(vxlan-c0a87aad): addr:ae:fd:e4:e4:c6:00
3(vxlan-c0a87a15): addr:a2:15:32:ee:c1:7f

Step 6 : Let’s make it more spicy by booting the instance using second private subnet i.e private_network1. This instance was booted on compute1 node.

[root@compute1 ~(keystone_admin)]# nova list
+————————————–+————+——–+————+————-+—————————-+
| ID                                   | Name       | Status | Task State | Power State | Networks                   |
+————————————–+————+——–+————+————-+—————————-+
| cd027e02-251c-4d5a-8d37-12c04062dcba | test1node1 | ACTIVE | –          | Running     | private_network=10.10.1.6  |
| 257b04dd-d6ab-4e31-a754-75de06f91447 | test2node1 | ACTIVE | –          | Running     | private_network=10.10.1.5  |
| 70b4ff74-6385-42c4-b42b-edd1287088f9 | test3node1 | ACTIVE | –          | Running     | private_network1=10.10.2.3 |
+————————————–+————+——–+————+————-+—————————-+

Step 7 : Pinging the test1node1 from test3node1. Note : As they both are in different subnet hence the traffic will be routed through network node.
a) This time I captured the traffic at ens3 interface of network node. It involves the compute1, compute2 and network node IP (192.168.122.173).

tshark -tad -n -r /tmp/tempo2.pcap -Y ‘udp.dstport == 4789’ -T fields -e ip.src -e ip.dst | sort | uniq -c
10 192.168.122.118    192.168.122.173
2 192.168.122.118    192.168.122.21
10 192.168.122.173    192.168.122.118

b) Once again I applied the “decode as” VXLAN filter in wireshark. This time VNI ID also seen is 63 along with  39.

printf “%x\n” 63
3f

hexadecimal value becomes 0x3f.

c) We can filter the flow rule for it as well.

[root@controller ~(keystone_admin)]# ovs-ofctl dump-flows br-tun | grep ‘0x3f’
cookie=0x0, duration=4274.040s, table=4, n_packets=124, n_bytes=11822, idle_age=3330, priority=1,tun_id=0x3f actions=mod_vlan_vid:2,resubmit(,10)
cookie=0x0, duration=4274.044s, table=22, n_packets=13, n_bytes=1102, idle_age=4260, dl_vlan=2 actions=strip_vlan,set_tunnel:0x3f,output:5,output:6

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s