How to use nfstest for testing of NFS ?

In this article I am going to show the usage of nfstest tool for the testing of various NFS operations. You can test all the versions of NFS upto NFSv 4.1 using this tool. It includes the support for pnfs as well.

I have downloaded the latest version of nfstest tool using link. http://www.linux-nfs.org/~mora/nfstest/releases/NFStest-1.0.10.tar.gz

My Lab setup :

–> NFS server : fedora 22 (192.168.111.163)

–> NFS clients : RHEL 7 (192.168.111.123) fedora 22 (192.168.111.164).

Step 1 : Installing the nfstest package on both clients.

Once you download the nfstest zipped file using the above link, you need to extract that zipped file and then issue the command for installation.

[root@pnfsclient NFStest-1.0.10]# pwd
/root/NFStest-1.0.10

[root@pnfsclient NFStest-1.0.10]# python setup.py install

Prerequisite : You need to have the python packet (python > 2.6 ) installed on your client.

Step 2 : After that jumped into the directory test which contains various python scripts to run the test.

[root@pnfsclient NFStest-1.0.10]# cd test/
[root@pnfsclient test]# pwd
/root/NFStest-1.0.10/test

[root@pnfsclient test]# ll
total 308
-rwxrwxr-x 1 500 500 24287 Jun  6 07:02 nfstest_cache
-rwxrwxr-x 1 500 500 27834 Jun  6 07:02 nfstest_delegation
-rwxrwxr-x 1 500 500 70361 Jun  6 07:02 nfstest_dio
-rwxrwxr-x 1 500 500  7710 Jun  6 07:01 nfstest_io
-rwxrwxr-x 1 500 500 48370 Jun  6 07:01 nfstest_lock
-rwxrwxr-x 1 500 500 56162 Jun  6 07:02 nfstest_pnfs
-rwxrwxr-x 1 500 500 72978 Jun  6 07:02 nfstest_posix

Step 3 : Some requirements before running the test.

If you want to run the tests which involve two clients then both the clients should have password less connection. In my case I have established password connection between both clients {RHEL 7 (192.168.111.123) fedora 22 (192.168.111.164)}.

This is useful for running tests like below which require two NFS clients.

read_recall_write
Recall read delegation by writing from a second client

write_recall_write
Recall write delegation by writing from a second client

read_recall_write_lock
Recall read delegation by writing from a second client with file lock

write_recall_write_lock
Recall write delegation by writing from a second client with file lock

write_recall_read
Recall write delegation by reading from a second client

write_recall_read_lock
Recall write delegation by reading from a second client with file lock

Also, if you are running test without specifying mountpoint option then test will run on /mnt/t hence please create this directory on all client nodes and change the permission to 777.

[root@pnfsclient test]# mkdir /mnt/t
[root@pnfsclient test]#
[root@pnfsclient test]# chmod 777 /mnt/t
[root@pnfsclient test]#

Step 4 : Lets run our first test from fedora node.

a) I have ran the below test in verbose form and instructed to keep the traces and logs of the test. If you are not going to specify the minorversion test will run with nfs v4.1 but  I want to run the test with NFSv4.0 hence specified the option. export option is used to mention the filesystem exported from NFS Server. I have ran the test to verify the read_lock only if you want to run the multiple tests specify the other with comma separated. You may refer the man page for more info on it. I have specified the output directory to keep the log and trace files i.e (/var/tmp/read_lock.nfsv4).

[root@pnfsclient test]# ./nfstest_delegation –server 192.168.111.163 –export=/share1  –keeptraces –createlog –verbose all –nfsversion=4 –minorversion=0 –tmpdir=/var/tmp/read_lock.nfsv4 –runtest=read_lock

Okay, test has been ran successfully below is the truncated output from test.

Logfile: /var/tmp/read_lock.nfsv4/nfstest_delegation_3567_20150715165434.log

5 tests (5 passed, 0 failed)

Total time: 41.232920s

Lets jump into the directory to verify the content. As expected its having two files, one is the log file and another one packet capture file.

[root@pnfsclient test]# cd /var/tmp/read_lock.nfsv4/
[root@pnfsclient read_lock.nfsv4]# pwd
/var/tmp/read_lock.nfsv4
[root@pnfsclient read_lock.nfsv4]# ll
total 32
-rw-r–r– 1 tcpdump tcpdump 24540 Jul 15 16:54 nfstest_delegation_3567_20150715165434_1.cap
-rw-r–r– 1 root    root     4137 Jul 15 16:54 nfstest_delegation_3567_20150715165434.log

Lets filter some information from log file. If you are running multiple tests below command will help you to figure out the packet capture file for each test.

[root@pnfsclient read_lock.nfsv4]# egrep -i “INFO|TIME|trace_open|PASS|FAIL” nfstest_delegation_3567_20150715165434.log
INFO: SYSTEM: Linux pnfsclient 4.0.4-301.fc22.x86_64 #1 SMP Thu May 21 13:10:33 UTC 2015 x86_64
DBG1: Get routing info: ip route get 192.168.111.163
INFO: Running test: read_lock
DBG1: trace_open [/var/tmp/read_lock.nfsv4/nfstest_delegation_3567_20150715165434_1.cap]
PASS: READ delegation should be granted
PASS: OPEN’s should not be sent for the same file
PASS: READ stateid should be the DELEG stateid
PASS: READ’s should not be sent when reading delegated file from a different process
PASS: LOCK should not be sent to the server
TIME: 2.740170s
5 tests (5 passed, 0 failed)
Total time: 41.232920s

Step 5 : Lets run the test with involvement of the second client, remember the step 3. I have ran the test read_recall_write which is mentioned in Step 3 that we require second client to run this test. I have given the IP of second client 192.168.111.123.

[root@pnfsclient test]# ./nfstest_delegation –server 192.168.111.163 –client 192.168.111.123 –export=/share1  –keeptraces –createlog –verbose all –nfsversion=4 –minorversion=0 –tmpdir=/var/tmp –runtest=read_recall_write –mtpoint=/mytesting

Logfile: /var/tmp/nfstest_delegation_3482_20150715163345.log

13 tests (13 passed, 0 failed)

Total time: 47.139873s

Once again we filter out the necessary information from log file using below command.

[root@pnfsclient tmp]# egrep -i “INFO|TIME|trace_open|PASS|FAIL” nfstest_delegation_3482_20150715163345.log
INFO: SYSTEM: Linux pnfsclient 4.0.4-301.fc22.x86_64 #1 SMP Thu May 21 13:10:33 UTC 2015 x86_64
DBG1: Get routing info: ip route get 192.168.111.163
INFO: Running test: read_recall_write
DBG1: trace_open [/var/tmp/nfstest_delegation_3482_20150715163345_1.cap]
PASS: READ delegation should be granted
PASS: CB_RECALL should not be sent to the client after a READ OPEN is received from a second client
PASS: CB_RECALL should not be sent to the client after a second client is granted a READ delegation
PASS: OPEN (WRITE) should be sent from second client
PASS: CB_RECALL should be sent to the client after a conflicting OPEN (WRITE) is received from a second client
PASS: CB_RECALL should recall READ delegation granted to client
PASS: OPEN with CLAIM_DELEGATE_CUR is sent from client right before returning the READ delegation after CB_RECALL
PASS: OPEN stateid should be the same as the original OPEN stateid
PASS: Delegation should not be granted when re-opening the file right before returning the READ delegation after CB_RECALL
PASS: DELEGRETURN should be sent from client right after re-opening the file
PASS: DELEGRETURN should return the READ delegation being recalled
PASS: OPEN (WRITE) reply should be sent to the second client after the READ delegation has been returned
PASS: Delegation should not be granted for the second client
TIME: 8.542772s
13 tests (13 passed, 0 failed)
Total time: 47.139873s

Lets check which IPs are involved in trace captured after running the test. We can see the three IPs one is of NFS server (192.168.111.163) and other are of two clients.

[root@pnfsclient read_recall_write_secondclient]# tshark -tad -n -r nfstest_delegation_3482_20150715163345_1.cap -Y nfs -T fields -e ip.src -e ip.dst | sort | uniq
Running as user “root” and group “root”. This could be dangerous.
192.168.111.123 192.168.111.163
192.168.111.163 192.168.111.123
192.168.111.163 192.168.111.164
192.168.111.164 192.168.111.163
References :

http://www.unix.com/man-page/centos/1/nfstest/

https://fedoraproject.org/wiki/Features/NFStest

http://wiki.linux-nfs.org/wiki/index.php/NFStest