Category Archives: fedora

How to use referrals in NFS ?

In this article, I am going to show you the usage of referrals in NFS. For theoretical information about referrals, I suggest you to refer the NFv4 RFC. (https://tools.ietf.org/html/rfc7530#section-4).

My lab setup :

–> NFS servers ( fedora 22).

–> NFS client ( RHEL 7)

Step 1 : I have exported on xfs filesystems from NFS server named server1.

[root@server1 AWK]# df -h /share2
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        5.0G   33M  5.0G   1% /share2

[root@server1 AWK]# cat /etc/exports
/share2 *(rw,no_root_squash)

Step 2 : I am giving the referral of the exported filesystem on another NFS server which is server2.

[root@server2 ~]# cat /etc/exports
/share2 *(no_root_squash,no_subtree_check,refer=/share2@192.168.111.163,sync)

Make sure that /share2 directory is present locally.

Step 3 : Mount the filesystem on client and capture the tcpdump while mounting.

[root@rheldesk ~]# tcpdump -s0 -i ens33 -w /tmp/referral1.pcap &
[1] 2244

In below command 192.168.111.164 is the IP address of server2.

[root@rheldesk ~]# mount -t nfs -o vers=4 192.168.111.164:/share2 /referal1/

In output of df on client you will see that filesystem is mounted from 192.168.111.163 which is the IP address of server1. Actual NFS server from which we have exported the filesystem.

Step 4 : Analyzing the tcpdump.

As per RFC we should get the errors like NFSERR_MOVED and we got the same in tcpdump.

Just a p recheck to see, how many IPs sending packets are captured in tcpdump. Totally three IPs, one of client, two of NFS servers.

[root@rheldesk tmp]# tshark -tad -n -r referral2.pcap -R nfs -T fields -e ip.src -e ip.dst | sort | uniq -c
tshark: -R without -2 is deprecated. For single-pass filtering use -Y.
Running as user “root” and group “root”. This could be dangerous.
21 192.168.111.123 192.168.111.163
18 192.168.111.123 192.168.111.164
21 192.168.111.163 192.168.111.123
18 192.168.111.164 192.168.111.123

a) Let’s check the NFS operations ending with error.

[root@rheldesk tmp]# tshark -tad -n -r referral2.pcap -R ‘nfs.status != 0’
tshark: -R without -2 is deprecated. For single-pass filtering use -Y.
Running as user “root” and group “root”. This could be dangerous.
144 2015-07-18 22:45:23.462655 192.168.111.164 -> 192.168.111.123 NFS 130 V4 Reply (Call In 143) LOOKUP | GETFH Status: NFS4ERR_MOVED
148 2015-07-18 22:45:23.464255 192.168.111.164 -> 192.168.111.123 NFS 130 V4 Reply (Call In 147) LOOKUP | GETFH Status: NFS4ERR_MOVED

b) As per RFC, if the client encounter NFS4ERR_MOVED then it should send the GETATTR with FS_LOCATIONS and MOUNTED_ON_FILEID

[root@rheldesk tmp]# tshark -tad -n -r referral2.pcap -Y ‘rpc.xid == 0x479edcd4’
Running as user “root” and group “root”. This could be dangerous.
145 2015-07-18 22:45:23.462924 192.168.111.123 -> 192.168.111.164 NFS 202 V4 Call LOOKUP DH: 0x62d40c52/share2
146 2015-07-18 22:45:23.463513 192.168.111.164 -> 192.168.111.123 NFS 230 V4 Reply (Call In 145) LOOKUP

Same thing is happening in call 145. It’s asking for filesystem location and mounted_on_fileid. Outputs are truncated.

In frame 145 GETATTR call is going with FSID and FS_LOCATION because of the error in 144.

Opcode: GETATTR (9)
Attr mask[0]: 0x01000100 (FSID, FS_LOCATIONS)
reqd_attr: FSID (8)
reco_attr: FS_LOCATIONS (24)
Attr mask[1]: 0x00800000 (MOUNTED_ON_FILEID)
reco_attr: MOUNTED_ON_FILEID (55)

In frame 146, we can see the reply coming with information of NFS server IP i.e 192.168.111.163

Opcode: GETATTR (9)
Status: NFS4_OK (0)
Attr mask[0]: 0x01000100 (FSID, FS_LOCATIONS)
reqd_attr: FSID (8)
fattr4_fsid
fsid4.major: 134217728
fsid4.minor: 134217728
reco_attr: FS_LOCATIONS (24)
fattr4_fs_locations
pathname components (1)
Name: share2
length: 6
contents: share2
fill bytes: opaque data
fs_location4:
num: 1
fs_location4
server:
num: 1
server: 192.168.111.163
length: 15
contents: 192.168.111.163
fill bytes: opaque data
pathname components (1)
Name: share2
length: 6
contents: share2
fill bytes: opaque data
Attr mask[1]: 0x00800000 (MOUNTED_ON_FILEID)
reco_attr: MOUNTED_ON_FILEID (55)
fileid: 0x000000000106bfe6

c) Now client will start communicating with NFS server 192.168.111.163 to mount the filesystem. In subsequent calls it will check filehandler type whether its volatile or permanent.

[root@rheldesk tmp]# tshark -tad -n -r referral2.pcap -Y ‘nfs && frame.number ge 160’
Running as user “root” and group “root”. This could be dangerous.
160 2015-07-18 22:45:23.467782 192.168.111.123 -> 192.168.111.163 NFS 262 V4 Call SETCLIENTID
161 2015-07-18 22:45:23.468550 192.168.111.163 -> 192.168.111.123 NFS 130 V4 Reply (Call In 160) SETCLIENTID
162 2015-07-18 22:45:23.468669 192.168.111.123 -> 192.168.111.163 NFS 170 V4 Call SETCLIENTID_CONFIRM
165 2015-07-18 22:45:23.470028 192.168.111.163 -> 192.168.111.123 NFS 114 V4 Reply (Call In 162) SETCLIENTID_CONFIRM
167 2015-07-18 22:45:23.470693 192.168.111.163 -> 192.168.111.123 NFS 142 V1 CB_NULL Call
169 2015-07-18 22:45:23.471066 192.168.111.123 -> 192.168.111.163 NFS 94 V1 CB_NULL Reply (Call In 167)

Advertisements

Why nfstest is not generating the traces (packet captures) ?

If you are running tests with nfstest, you may have noticed that its not generating the packet captures for every test.  Nothing to worry, now new option (–createtraces) has been added to create packet captures for each test when it’s specified.

You may use the below link to check the recent developments regarding nfstest.

http://git.linux-nfs.org/?p=mora/nfstest.git;a=summary

I have cloned the git link provided in above link. If you are not having git package install on system, you need to install it.

# git clone git://git.linux-nfs.org/projects/mora/nfstest.git

It will download a directory with name “nfstest”. cd into that directory and run the below command to make the tool work.

# python setup.py install

Now with each test just add the –createtraces option, you will get the packet capture.

Happy NFS Testing!!!

How to use nfstest for testing of NFS ?

In this article I am going to show the usage of nfstest tool for the testing of various NFS operations. You can test all the versions of NFS upto NFSv 4.1 using this tool. It includes the support for pnfs as well.

I have downloaded the latest version of nfstest tool using link. http://www.linux-nfs.org/~mora/nfstest/releases/NFStest-1.0.10.tar.gz

My Lab setup :

–> NFS server : fedora 22 (192.168.111.163)

–> NFS clients : RHEL 7 (192.168.111.123) fedora 22 (192.168.111.164).

Step 1 : Installing the nfstest package on both clients.

Once you download the nfstest zipped file using the above link, you need to extract that zipped file and then issue the command for installation.

[root@pnfsclient NFStest-1.0.10]# pwd
/root/NFStest-1.0.10

[root@pnfsclient NFStest-1.0.10]# python setup.py install

Prerequisite : You need to have the python packet (python > 2.6 ) installed on your client.

Step 2 : After that jumped into the directory test which contains various python scripts to run the test.

[root@pnfsclient NFStest-1.0.10]# cd test/
[root@pnfsclient test]# pwd
/root/NFStest-1.0.10/test

[root@pnfsclient test]# ll
total 308
-rwxrwxr-x 1 500 500 24287 Jun  6 07:02 nfstest_cache
-rwxrwxr-x 1 500 500 27834 Jun  6 07:02 nfstest_delegation
-rwxrwxr-x 1 500 500 70361 Jun  6 07:02 nfstest_dio
-rwxrwxr-x 1 500 500  7710 Jun  6 07:01 nfstest_io
-rwxrwxr-x 1 500 500 48370 Jun  6 07:01 nfstest_lock
-rwxrwxr-x 1 500 500 56162 Jun  6 07:02 nfstest_pnfs
-rwxrwxr-x 1 500 500 72978 Jun  6 07:02 nfstest_posix

Step 3 : Some requirements before running the test.

If you want to run the tests which involve two clients then both the clients should have password less connection. In my case I have established password connection between both clients {RHEL 7 (192.168.111.123) fedora 22 (192.168.111.164)}.

This is useful for running tests like below which require two NFS clients.

read_recall_write
Recall read delegation by writing from a second client

write_recall_write
Recall write delegation by writing from a second client

read_recall_write_lock
Recall read delegation by writing from a second client with file lock

write_recall_write_lock
Recall write delegation by writing from a second client with file lock

write_recall_read
Recall write delegation by reading from a second client

write_recall_read_lock
Recall write delegation by reading from a second client with file lock

Also, if you are running test without specifying mountpoint option then test will run on /mnt/t hence please create this directory on all client nodes and change the permission to 777.

[root@pnfsclient test]# mkdir /mnt/t
[root@pnfsclient test]#
[root@pnfsclient test]# chmod 777 /mnt/t
[root@pnfsclient test]#

Step 4 : Lets run our first test from fedora node.

a) I have ran the below test in verbose form and instructed to keep the traces and logs of the test. If you are not going to specify the minorversion test will run with nfs v4.1 but  I want to run the test with NFSv4.0 hence specified the option. export option is used to mention the filesystem exported from NFS Server. I have ran the test to verify the read_lock only if you want to run the multiple tests specify the other with comma separated. You may refer the man page for more info on it. I have specified the output directory to keep the log and trace files i.e (/var/tmp/read_lock.nfsv4).

[root@pnfsclient test]# ./nfstest_delegation –server 192.168.111.163 –export=/share1  –keeptraces –createlog –verbose all –nfsversion=4 –minorversion=0 –tmpdir=/var/tmp/read_lock.nfsv4 –runtest=read_lock

Okay, test has been ran successfully below is the truncated output from test.

Logfile: /var/tmp/read_lock.nfsv4/nfstest_delegation_3567_20150715165434.log

5 tests (5 passed, 0 failed)

Total time: 41.232920s

Lets jump into the directory to verify the content. As expected its having two files, one is the log file and another one packet capture file.

[root@pnfsclient test]# cd /var/tmp/read_lock.nfsv4/
[root@pnfsclient read_lock.nfsv4]# pwd
/var/tmp/read_lock.nfsv4
[root@pnfsclient read_lock.nfsv4]# ll
total 32
-rw-r–r– 1 tcpdump tcpdump 24540 Jul 15 16:54 nfstest_delegation_3567_20150715165434_1.cap
-rw-r–r– 1 root    root     4137 Jul 15 16:54 nfstest_delegation_3567_20150715165434.log

Lets filter some information from log file. If you are running multiple tests below command will help you to figure out the packet capture file for each test.

[root@pnfsclient read_lock.nfsv4]# egrep -i “INFO|TIME|trace_open|PASS|FAIL” nfstest_delegation_3567_20150715165434.log
INFO: SYSTEM: Linux pnfsclient 4.0.4-301.fc22.x86_64 #1 SMP Thu May 21 13:10:33 UTC 2015 x86_64
DBG1: Get routing info: ip route get 192.168.111.163
INFO: Running test: read_lock
DBG1: trace_open [/var/tmp/read_lock.nfsv4/nfstest_delegation_3567_20150715165434_1.cap]
PASS: READ delegation should be granted
PASS: OPEN’s should not be sent for the same file
PASS: READ stateid should be the DELEG stateid
PASS: READ’s should not be sent when reading delegated file from a different process
PASS: LOCK should not be sent to the server
TIME: 2.740170s
5 tests (5 passed, 0 failed)
Total time: 41.232920s

Step 5 : Lets run the test with involvement of the second client, remember the step 3. I have ran the test read_recall_write which is mentioned in Step 3 that we require second client to run this test. I have given the IP of second client 192.168.111.123.

[root@pnfsclient test]# ./nfstest_delegation –server 192.168.111.163 –client 192.168.111.123 –export=/share1  –keeptraces –createlog –verbose all –nfsversion=4 –minorversion=0 –tmpdir=/var/tmp –runtest=read_recall_write –mtpoint=/mytesting

Logfile: /var/tmp/nfstest_delegation_3482_20150715163345.log

13 tests (13 passed, 0 failed)

Total time: 47.139873s

Once again we filter out the necessary information from log file using below command.

[root@pnfsclient tmp]# egrep -i “INFO|TIME|trace_open|PASS|FAIL” nfstest_delegation_3482_20150715163345.log
INFO: SYSTEM: Linux pnfsclient 4.0.4-301.fc22.x86_64 #1 SMP Thu May 21 13:10:33 UTC 2015 x86_64
DBG1: Get routing info: ip route get 192.168.111.163
INFO: Running test: read_recall_write
DBG1: trace_open [/var/tmp/nfstest_delegation_3482_20150715163345_1.cap]
PASS: READ delegation should be granted
PASS: CB_RECALL should not be sent to the client after a READ OPEN is received from a second client
PASS: CB_RECALL should not be sent to the client after a second client is granted a READ delegation
PASS: OPEN (WRITE) should be sent from second client
PASS: CB_RECALL should be sent to the client after a conflicting OPEN (WRITE) is received from a second client
PASS: CB_RECALL should recall READ delegation granted to client
PASS: OPEN with CLAIM_DELEGATE_CUR is sent from client right before returning the READ delegation after CB_RECALL
PASS: OPEN stateid should be the same as the original OPEN stateid
PASS: Delegation should not be granted when re-opening the file right before returning the READ delegation after CB_RECALL
PASS: DELEGRETURN should be sent from client right after re-opening the file
PASS: DELEGRETURN should return the READ delegation being recalled
PASS: OPEN (WRITE) reply should be sent to the second client after the READ delegation has been returned
PASS: Delegation should not be granted for the second client
TIME: 8.542772s
13 tests (13 passed, 0 failed)
Total time: 47.139873s

Lets check which IPs are involved in trace captured after running the test. We can see the three IPs one is of NFS server (192.168.111.163) and other are of two clients.

[root@pnfsclient read_recall_write_secondclient]# tshark -tad -n -r nfstest_delegation_3482_20150715163345_1.cap -Y nfs -T fields -e ip.src -e ip.dst | sort | uniq
Running as user “root” and group “root”. This could be dangerous.
192.168.111.123 192.168.111.163
192.168.111.163 192.168.111.123
192.168.111.163 192.168.111.164
192.168.111.164 192.168.111.163
References :

http://www.unix.com/man-page/centos/1/nfstest/

https://fedoraproject.org/wiki/Features/NFStest

http://wiki.linux-nfs.org/wiki/index.php/NFStest

How to configure pnfs server in Fedora 22 ?

In this article I am going to show how to configure pnfs server in fedora. After exporting the share we will mount the filesystem on another fedora node.

I have downloaded fedora 22 ISO image from fedora website and issue dnf update after installing it.

Step 1 : It’s similar to simple export in case of RHEL only option which I have added is pnfs.

pnfsserver# cat /etc/exports
/test1 *(rw,no_root_squash,pnfs)

Step 2 : I have mounted the exported share on another fedora node using below command. Before mounting the share just checking the version supported on NFS client even though I am using the same version of fedora on client.

[root@pnfsclient ~]# cat /proc/fs/nfsd/versions
-2 +3 +4 +4.1 +4.2

I have issued below command in background to capture the tcpdump.

[root@pnfsclient ~]# tcpdump -s 0 -i ens33 host 192.168.111.163 -w /tmp/fedmountv4.1pcap &
[1] 2598

Finally, mounted the filesystem and killed the tcpdump background process.

[root@pnfsclient ~]# mount.nfs -o vers=4.1 192.168.111.163:/test1 /mnt

[root@pnfsclient ~]# kill 2598
[root@pnfsclient ~]# 58 packets captured
58 packets received by filter
0 packets dropped by kernel

[1]+  Done                    tcpdump -s 0 -i ens33 host 192.168.111.163 -w /tmp/fedmountv4.1pcap

Step 3 : Lets check the mounted filesystem options.

[root@pnfsclient ~]# grep -w nfs4 /proc/mounts
192.168.111.163:/test1 /mnt nfs4 rw,relatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.111.164,local_lock=none,addr=192.168.111.163 0 0

Hey, but I am not able to verify whether client is using pNFS or not, hold on, use below command to verify your client layout.

[root@pnfsclient ~]# grep LAYOUT /proc/self/mountstats
nfsv4:  bm0=0xfdffbfff,bm1=0x40f9be3e,bm2=0x803,acl=0x3,sessions,pnfs=LAYOUT_BLOCK_VOLUME
LAYOUTGET: 0 0 0 0 0 0 0 0
LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
LAYOUTRETURN: 0 0 0 0 0 0 0 0

At the time of writing this article, pnfs was supported only for block layout.

Step 4 : Lets have look into the tcpdump output.

a) I am reading tcpdump using command line instead of wireshark. I am interested only EXCHANGE_ID call and reply hence filtered it.

[root@pnfsclient ~]# tshark -tad -n -r /tmp/fedmountv4.1pcap | grep -i exchange
Running as user “root” and group “root”. This could be dangerous.
12 2015-07-05 21:42:23.183471 192.168.111.164 -> 192.168.111.163 NFS 310 V4 Call EXCHANGE_ID
13 2015-07-05 21:42:23.184525 192.168.111.163 -> 192.168.111.164 NFS 202 V4 Reply (Call In 12) EXCHANGE_ID

b) I am opening the frame 12 which is a call from client to pnfs server. Below output is truncated showing only NFS part for the sake of brevity. I have high-lightened the EXCHGID4_FLAG_USE_PNFS_DS and EXCHGID4_FLAG_USE_PNFS_MDS flags in below output.

[root@pnfsclient ~]# tshark -V -tad -n -r /tmp/fedmountv4.1pcap ‘frame.number == 12’

Network File System, Ops(1): EXCHANGE_ID
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Tag: <EMPTY>
length: 0
contents: <EMPTY>
minorversion: 1
Operations (count: 1): EXCHANGE_ID
Opcode: EXCHANGE_ID (42)
eia_clientowner
verifier: 0x5598b942166d2919
Data: <DATA>
length: 24
contents: <DATA>
flags: 0x00000101
0… …. …. …. …. …. …. …. = EXCHGID4_FLAG_CONFIRMED_R: Not set
.0.. …. …. …. …. …. …. …. = EXCHGID4_FLAG_UPD_CONFIRMED_REC_A: Not set
                …. …. …. .0.. …. …. …. …. = EXCHGID4_FLAG_USE_PNFS_DS: Not set
                …. …. …. ..0. …. …. …. …. = EXCHGID4_FLAG_USE_PNFS_MDS: Not set
…. …. …. …0 …. …. …. …. = EXCHGID4_FLAG_USE_NON_PNFS: Not set
…. …. …. …. …. …1 …. …. = EXCHGID4_FLAG_BIND_PRINC_STATEID: Set
…. …. …. …. …. …. …. ..0. = EXCHGID4_FLAG_SUPP_MOVED_MIGR: Not set
…. …. …. …. …. …. …. …1 = EXCHGID4_FLAG_SUPP_MOVED_REFER: Set
eia_state_protect: SP4_NONE (0)
eia_client_impl_id
Implementor DNS domain name(nii_domain): kernel.org
length: 10
contents: kernel.org
fill bytes: opaque data
Implementation product name(nii_name): Linux 4.0.4-301.fc22.x86_64 #1 SMP Thu May 21 13:10:33 UTC 2015 x86_64
length: 70
contents: Linux 4.0.4-301.fc22.x86_64 #1 SMP Thu May 21 13:10:33 UTC 2015 x86_64
fill bytes: opaque data
Build timestamp(nii_date)
seconds: 0
nseconds: 0
[Main Opcode: EXCHANGE_ID (42)]

c) Checking the reply  for previous call i.e from pnfs server to client. reply is present in frame 13. Once again output is truncated.

[root@pnfsclient ~]# tshark -V -tad -n -r /tmp/fedmountv4.1pcap ‘frame.number == 13’

Network File System, Ops(1): EXCHANGE_ID
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Status: NFS4_OK (0)
Tag: <EMPTY>
length: 0
contents: <EMPTY>
Operations (count: 1)
Opcode: EXCHANGE_ID (42)
Status: NFS4_OK (0)
clientid: 0x6bb7985501000000
seqid: 0x00000001
flags: 0x00020001
0… …. …. …. …. …. …. …. = EXCHGID4_FLAG_CONFIRMED_R: Not set
.0.. …. …. …. …. …. …. …. = EXCHGID4_FLAG_UPD_CONFIRMED_REC_A: Not set
                …. …. …. .0.. …. …. …. …. = EXCHGID4_FLAG_USE_PNFS_DS: Not set
                …. …. …. ..1. …. …. …. …. = EXCHGID4_FLAG_USE_PNFS_MDS: Set
…. …. …. …0 …. …. …. …. = EXCHGID4_FLAG_USE_NON_PNFS: Not set
…. …. …. …. …. …0 …. …. = EXCHGID4_FLAG_BIND_PRINC_STATEID: Not set
…. …. …. …. …. …. …. ..0. = EXCHGID4_FLAG_SUPP_MOVED_MIGR: Not set
…. …. …. …. …. …. …. …1 = EXCHGID4_FLAG_SUPP_MOVED_REFER: Set
eia_state_protect: SP4_NONE (0)
eir_server_owner
minor ID: 0
major ID: <DATA>
length: 21
contents: <DATA>
fill bytes: opaque data
server scope: <DATA>
length: 21
contents: <DATA>
fill bytes: opaque data
eir_server_impl_id
[Main Opcode: EXCHANGE_ID (42)]

I am not using any dataserver that’s why EXCHGID4_FLAG_USE_PNFS_DS flag is unset. If you see the EXCHGID4_FLAG_USE_PNFS_MDS is set because our pNFS server is acting like a metadata server.

I will come up with more articles on pNFS.