How the network traffic flow in openstack covering E/W traffic ?

In previous post, I have covered the N/S traffic flowing from instance to external world. In this  post, I am covering traffic between two instances hosted in different private network hosted on same compute node.

Step 1 : I have created two instances each instance is booted using different internal network. I have created one more internal network (10.10.3.0/24) to achieve the same, and added that network tas an interface to router.

[root@vswitch1 tmp(keystone_admin)]# neutron router-interface-add router1 private1_subnet
Added interface 3115253c-8cc8-48c9-8a43-9b9d7d593af6 to router router1.
[root@vswitch1 tmp(keystone_admin)]# neutron net-list
+————————————–+——————+——————————————————-+
| id                                   | name             | subnets                                               |
+————————————–+——————+——————————————————-+
| 1f882510-5777-4344-a55c-839d34847979 | private_network  | 898a13a9-694e-4d23-aefc-1145d43277cf 10.10.2.0/24     |
| 64e86f08-ce73-4fdd-9581-da449e5f069b | private1         | 305e1819-4c89-40a6-ac9a-74fdd0b36699 10.10.3.0/24     |
| a912cec4-bce7-4984-a9d8-630c8e784f69 | external_network | 16af3f27-c5c1-4018-9de5-9e3600086031 192.168.122.0/24 |
+————————————–+——————+——————————————————-+

Step 2 : I have two instances running on separate private networks.

[root@vswitch1 tmp(keystone_admin)]# nova list
+————————————–+——-+——–+————+————-+——————————————-+
| ID                                   | Name  | Status | Task State | Power State | Networks                                  |
+————————————–+——-+——–+————+————-+——————————————-+
| a0f07082-e54b-4bc5-82f0-2f67d0e9b7f0 | test1 | ACTIVE | –          | Running     | private_network=10.10.2.9, 192.168.122.11 |
| 1611741a-840e-4c1c-92a0-0d5b46058273 | test3 | ACTIVE | –          | Running     | private1=10.10.3.3                        |
+————————————–+——-+——–+————+————-+——————————————-+

Step 3 : Checking the HW addresses of the interfaces attached to the instances.

virsh dumpxml 2 | egrep “test1|mac address”
<nova:name>test1</nova:name>
<mac address=’fa:16:3e:a3:e2:cc’/>

virsh dumpxml 4 | egrep “test3|mac address”
<nova:name>test3</nova:name>
<mac address=’fa:16:3e:b2:ad:34’/>

Step 4 : Lets see which path traffic will take if I am going to ping the instance test1 from test3 i.e pinging ip 10.10.2.9 from 10.10.3.3

Step 5 : Before starting capturing tcpdump and start pinging the IP. I want to show the gateway MAC addresses for both interfaces i.e in qrouter namespace.

ip netns exec qrouter-ae5968fa-902e-4ddc-8516-1216d2bb4303 ip a | awk ‘/qr-.*/ {print $0;}’
12: qr-ebd072c8-28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN      fa:16:3e:62:9f:76
inet 10.10.2.1/24 brd 10.10.2.255 scope global qr-ebd072c8-28
24: qr-3115253c-8c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN      fa:16:3e:44:8d:d7
inet 10.10.3.1/24 brd 10.10.3.255 scope global qr-3115253c-8c

Step 6 : I have used below command for capturing the traffic. I am skipping the veth pair because tap and veth pair will contain same traffic. I am using tap device of each instance and each network default gateway interface.

tcpdump -s0 -i tap3d6dc685-c3 -w /tmp/tap-test1-2-3.pcap &
tcpdump -s0 -i tapb4692153-45 -w /tmp/tap-test2-2-3.pcap &
ip netns exec qrouter-ae5968fa-902e-4ddc-8516-1216d2bb4303 tcpdump -s0 -i qr-ebd072c8-28 -w /tmp/qr-test1-2-3.pcap &
ip netns exec qrouter-ae5968fa-902e-4ddc-8516-1216d2bb4303 tcpdump -s0 -i qr-3115253c-8c -w /tmp/qr-test2-2.3.pcap &

Step 7 : Let’s start ping from instance test3 (10.10.3.3)  to test1 (10.10.2.9).

Just for quick reference :

test1  (10.10.2.9)  MAC address : fa:16:3e:a3:e2:cc

test3 (10.10.3.3)   MAC address : fa:16:3e:b2:ad:34

qr-ebd072c8-28 (10.10.2.1) MAC address : fa:16:3e:62:9f:76

qr-3115253c-8c  (10.10.3.1)  MAC address : fa:16:3e:44:8d:d7

a) It doesn’t know where the destination IP address is located hence it started looking the MAC address of gateway 10.10.3.1.

tshark -tad -n -r /tmp/tap-test2-2-3.pcap
Running as user “root” and group “root”. This could be dangerous.
1 2015-12-09 23:09:49 fa:16:3e:b2:ad:34 -> ff:ff:ff:ff:ff:ff ARP 42 Who has 10.10.3.1?  Tell 10.10.3.3
2 2015-12-09 23:09:49 fa:16:3e:44:8d:d7 -> fa:16:3e:b2:ad:34 ARP 42 10.10.3.1 is at fa:16:3e:44:8d:d7
3 2015-12-09 23:09:49    10.10.3.3 -> 10.10.2.9    ICMP 98 Echo (ping) request  id=0x5101, seq=0/0, ttl=64
4 2015-12-09 23:09:49    10.10.2.9 -> 10.10.3.3    ICMP 98 Echo (ping) reply    id=0x5101, seq=0/0, ttl=63 (request in 3)

b) Let’s see the MAC addresses involved here fa:16:3e:44:8d:d7  MAC address of default gw and fa:16:3e:b2:ad:34 MAC address of test3.

tshark -tad -n -r /tmp/tap-test2-2-3.pcap -T fields -e eth.src -e eth.dst | sort | uniq -c
Running as user “root” and group “root”. This could be dangerous.
7 fa:16:3e:44:8d:d7    fa:16:3e:b2:ad:34
6 fa:16:3e:b2:ad:34    fa:16:3e:44:8d:d7
1 fa:16:3e:b2:ad:34    ff:ff:ff:ff:ff:ff

c) Let’s check the frames at qr interface which is connection endpoint or junction. We can see that when ping request arrived here it started looking for the gareway 10.10.2.1 MAC

tshark -tad -n -r /tmp/qr-test1-2-3.pcap
Running as user “root” and group “root”. This could be dangerous.
1 2015-12-09 23:09:49    10.10.3.3 -> 10.10.2.9    ICMP 98 Echo (ping) request  id=0x5101, seq=0/0, ttl=63
2 2015-12-09 23:09:49 fa:16:3e:a3:e2:cc -> ff:ff:ff:ff:ff:ff ARP 42 Who has 10.10.2.1?  Tell 10.10.2.9
3 2015-12-09 23:09:49 fa:16:3e:62:9f:76 -> fa:16:3e:a3:e2:cc ARP 42 10.10.2.1 is at fa:16:3e:62:9f:76
4 2015-12-09 23:09:49    10.10.2.9 -> 10.10.3.3    ICMP 98 Echo (ping) reply    id=0x5101, seq=0/0, ttl=64 (request in 1)
5 2015-12-09 23:09:50    10.10.3.3 -> 10.10.2.9    ICMP 98 Echo (ping) request  id=0x5101, seq=1/256, ttl=63
6 2015-12-09 23:09:50    10.10.2.9 -> 10.10.3.3    ICMP 98 Echo (ping) reply    id=0x5101, seq=1/256, ttl=64 (request in 5)

d) Filtering the MAC addresses involved on qr interface. MAC address is of gateway (10.10.2.1 – fa:16:3e:62:9f:76) and MAC address of test3 (fa:16:3e:a3:e2:cc)

tshark -tad -n -r /tmp/qr-test1-2-3.pcap -T fields -e eth.src -e eth.dst | sort | uniq -c
Running as user “root” and group “root”. This could be dangerous.
7 fa:16:3e:62:9f:76    fa:16:3e:a3:e2:cc
6 fa:16:3e:a3:e2:cc    fa:16:3e:62:9f:76
1 fa:16:3e:a3:e2:cc    ff:ff:ff:ff:ff:ff

e) Checking the traffic on test1 instance.

tshark -tad -n -r /tmp/tap-test1-2-3.pcap
Running as user “root” and group “root”. This could be dangerous.
1 2015-12-09 23:09:49    10.10.3.3 -> 10.10.2.9    ICMP 98 Echo (ping) request  id=0x5101, seq=0/0, ttl=63
2 2015-12-09 23:09:49 fa:16:3e:a3:e2:cc -> ff:ff:ff:ff:ff:ff ARP 42 Who has 10.10.2.1?  Tell 10.10.2.9
3 2015-12-09 23:09:49 fa:16:3e:62:9f:76 -> fa:16:3e:a3:e2:cc ARP 42 10.10.2.1 is at fa:16:3e:62:9f:76
4 2015-12-09 23:09:49    10.10.2.9 -> 10.10.3.3    ICMP 98 Echo (ping) reply    id=0x5101, seq=0/0, ttl=64 (request in 1)
5 2015-12-09 23:09:50    10.10.3.3 -> 10.10.2.9    ICMP 98 Echo (ping) request  id=0x5101, seq=1/256, ttl=63

f) MAC addresses involved are same when the traffic leaves qr interface. Only MAC address change happened at qr interface because it has to reach the gateway of destination network

tshark -tad -n -r /tmp/tap-test1-2-3.pcap -T fields -e eth.src -e eth.dst | sort | uniq -c
Running as user “root” and group “root”. This could be dangerous.
7 fa:16:3e:62:9f:76    fa:16:3e:a3:e2:cc
6 fa:16:3e:a3:e2:cc    fa:16:3e:62:9f:76
1 fa:16:3e:a3:e2:cc    ff:ff:ff:ff:ff:ff

Step 8 : qg interface is not involved in this communication as the network traffic is between internal network.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s