How the network traffic flow in Openstack covering N/S traffic?

In this article I am going to show the flow of network traffic in openstack environment. Networking is a complex topic in openstack setup. Most of the users are not aware of the network flow inside openstack. Lot of virtual devices are involved in communication.

Test Lab setup : Covering N/S traffic. Instance reaching to external world.

RHEL OSP 7 oackstack installation on Linux machine as KVM.

Running test1 instance with fixed and floating IP.

Step 1 : Showing the configuration details. As I am having only one running instance hence one bridge is created with two interfaces (qvb3d6dc685-c3 and tap3d6dc685-c3)

brctl show
bridge name    bridge id        STP enabled    interfaces
qbr3d6dc685-c3        8000.fa3c9708fc47    no        qvb3d6dc685-c3
tap3d6dc685-c3

As you may already know instance ethernet device will be directly connected to  “tap” device which further connects to one end of  veth pair.

Step 2 : Initially when you are creating networks in your openstack setup few imp things happens in background which we generally don’t care.

In my case when I created internal network with CIDR notation 10.10.2.0/24. I found that default gw of this network was automatically configured in qrouter namespace. It was assigned to interface starting with “qr-“.

Similarly of floating IP range (external network) first IP address will be assigned to the interface starting with “qg-” which is also present in router namespace. You can check these IPs using network namespace commands like :

# ip netns list

# ip netns exec <namespace> ip a

Step 3 : Before initiating the ping from instance to outside network. I started capturing the tcpdump using below commands.

tcpdump -s0 -i tap3d6dc685-c3 -w /tmp/tap-external.pcap &
tcpdump -s0 -i qvb3d6dc685-c3 -w /tmp/qvb-external.pcap &
tcpdump -s0 -i qvo3d6dc685-c3 -w /tmp/qvo-external.pcap &
ip netns exec qrouter-ae5968fa-902e-4ddc-8516-1216d2bb4303 tcpdump -s0 -i qg-c988447e-c3 -w /tmp/qg-external.pcap &
ip netns exec qrouter-ae5968fa-902e-4ddc-8516-1216d2bb4303 tcpdump -s0 -i qr-ebd072c8-28 -w /tmp/qr-external.pcap &
tcpdump -s0 -i ens3 -w /tmp/ens3-external.pcap &

Acutally, you need not to collect the capture for qvb and qvo both because these are two ends of same tube. Make sense 🙂  Infact, we will be seeing the same traffic on tap, qvb and qvo.

Step 4 : All set. Let’s start pinging the IP (10.65.223.4) from instance. Notably instance (test1) is having two IPs (10.10.2.9) and floating ip (192.168.122.11).

Step 5 : Once some ping response received, I killed all the tcpdump processes started in Step 3.

Step 6 : Time to dig the tcpdump capture.

a) When we start the ping from instance (test1) it was not knowing the IP 10.65.223.4 hence it sent ARP request asking for the IP addres of default gateway i.e 10.10.2.1

Below output is truncated, it’s the capture from tap device.

~~~
tshark -tad -n -r tap-external.pcap
Running as user “root” and group “root”. This could be dangerous.
1 2015-12-09 20:58:04 fa:16:3e:a3:e2:cc -> ff:ff:ff:ff:ff:ff ARP 42 Who has 10.10.2.1?  Tell 10.10.2.9
2 2015-12-09 20:58:04 fa:16:3e:62:9f:76 -> fa:16:3e:a3:e2:cc ARP 42 10.10.2.1 is at fa:16:3e:62:9f:76
3 2015-12-09 20:58:04    10.10.2.9 -> 10.65.223.4  ICMP 98 Echo (ping) request  id=0x3101, seq=0/0, ttl=64
4 2015-12-09 20:58:04  10.65.223.4 -> 10.10.2.9    ICMP 98 Echo (ping) reply    id=0x3101, seq=0/0, ttl=63 (request in 3)

~~~

b) tap device, qvb, and qvo will show you the same traffic.

c) Let’s directly jump to the router interfaces i.e qr and qg.

At qr interface is also showing the same traffic.

~~~
tshark -tad -n -r qr-external.pcap
Running as user “root” and group “root”. This could be dangerous.
1 2015-12-09 20:58:04 fa:16:3e:a3:e2:cc -> ff:ff:ff:ff:ff:ff ARP 42 Who has 10.10.2.1?  Tell 10.10.2.9
2 2015-12-09 20:58:04 fa:16:3e:62:9f:76 -> fa:16:3e:a3:e2:cc ARP 42 10.10.2.1 is at fa:16:3e:62:9f:76
3 2015-12-09 20:58:04    10.10.2.9 -> 10.65.223.4  ICMP 98 Echo (ping) request  id=0x3101, seq=0/0, ttl=64
4 2015-12-09 20:58:04  10.65.223.4 -> 10.10.2.9    ICMP 98 Echo (ping) reply    id=0x3101, seq=0/0, ttl=63 (request in 3)

~~~

Main change started at qg interface because in external world no one knows about the private IP hence SNAT performed on private IP and traffic is going to the external world with floating IP. At this point MAC address of bridge configured on linux box will be the destination MAC and source MAC would be of the of qg interface.

~~~
tshark -tad -n -r qg-external.pcap
Running as user “root” and group “root”. This could be dangerous.
1 2015-12-09 20:58:04 192.168.122.11 -> 10.65.223.4  ICMP 98 Echo (ping) request  id=0x3101, seq=0/0, ttl=63
2 2015-12-09 20:58:04  10.65.223.4 -> 192.168.122.11 ICMP 98 Echo (ping) reply    id=0x3101, seq=0/0, ttl=64 (request in 1)
3 2015-12-09 20:58:05 192.168.122.11 -> 10.65.223.4  ICMP 98 Echo (ping) request  id=0x3101, seq=1/256, ttl=63
4 2015-12-09 20:58:05  10.65.223.4 -> 192.168.122.11 ICMP 98 Echo (ping) reply    id=0x3101, seq=1/256, ttl=64 (request in 3)
5 2015-12-09 20:58:06 192.168.122.11 -> 10.65.223.4  ICMP 98 Echo (ping) request  id=0x3101, seq=2/512, ttl=63

~~~

Step 7 : Let’s check for the MAC addresses involved in communication.

a) Initial traffic started with instance MAC as source and default GW MAC as destination address.

~~~
tshark -tad -n -r tap-external.pcap -T fields -e eth.src -e eth.dst | sort | uniq -c
Running as user “root” and group “root”. This could be dangerous.
9 fa:16:3e:62:9f:76    fa:16:3e:a3:e2:cc
8 fa:16:3e:a3:e2:cc    fa:16:3e:62:9f:76
1 fa:16:3e:a3:e2:cc    ff:ff:ff:ff:ff:ff
~~~

Above flow remain same on veth pairs and qr interface.

b) From qg interface onwards, source MAC address changed to qg interface and destination MAC address to bridge configured on physical Linux box.

~~~
tshark -tad -n -r qg-external.pcap -T fields -e eth.src -e eth.dst | sort | uniq -c
Running as user “root” and group “root”. This could be dangerous.
9 52:54:00:68:9d:b5    fa:16:3e:9e:8e:27
9 fa:16:3e:9e:8e:27    52:54:00:68:9d:b5
~~~

 

Reference :

http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s