How to use referrals in NFS ?

In this article, I am going to show you the usage of referrals in NFS. For theoretical information about referrals, I suggest you to refer the NFv4 RFC. (https://tools.ietf.org/html/rfc7530#section-4).

My lab setup :

–> NFS servers ( fedora 22).

–> NFS client ( RHEL 7)

Step 1 : I have exported on xfs filesystems from NFS server named server1.

[root@server1 AWK]# df -h /share2
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        5.0G   33M  5.0G   1% /share2

[root@server1 AWK]# cat /etc/exports
/share2 *(rw,no_root_squash)

Step 2 : I am giving the referral of the exported filesystem on another NFS server which is server2.

[root@server2 ~]# cat /etc/exports
/share2 *(no_root_squash,no_subtree_check,refer=/share2@192.168.111.163,sync)

Make sure that /share2 directory is present locally.

Step 3 : Mount the filesystem on client and capture the tcpdump while mounting.

[root@rheldesk ~]# tcpdump -s0 -i ens33 -w /tmp/referral1.pcap &
[1] 2244

In below command 192.168.111.164 is the IP address of server2.

[root@rheldesk ~]# mount -t nfs -o vers=4 192.168.111.164:/share2 /referal1/

In output of df on client you will see that filesystem is mounted from 192.168.111.163 which is the IP address of server1. Actual NFS server from which we have exported the filesystem.

Step 4 : Analyzing the tcpdump.

As per RFC we should get the errors like NFSERR_MOVED and we got the same in tcpdump.

Just a p recheck to see, how many IPs sending packets are captured in tcpdump. Totally three IPs, one of client, two of NFS servers.

[root@rheldesk tmp]# tshark -tad -n -r referral2.pcap -R nfs -T fields -e ip.src -e ip.dst | sort | uniq -c
tshark: -R without -2 is deprecated. For single-pass filtering use -Y.
Running as user “root” and group “root”. This could be dangerous.
21 192.168.111.123 192.168.111.163
18 192.168.111.123 192.168.111.164
21 192.168.111.163 192.168.111.123
18 192.168.111.164 192.168.111.123

a) Let’s check the NFS operations ending with error.

[root@rheldesk tmp]# tshark -tad -n -r referral2.pcap -R ‘nfs.status != 0’
tshark: -R without -2 is deprecated. For single-pass filtering use -Y.
Running as user “root” and group “root”. This could be dangerous.
144 2015-07-18 22:45:23.462655 192.168.111.164 -> 192.168.111.123 NFS 130 V4 Reply (Call In 143) LOOKUP | GETFH Status: NFS4ERR_MOVED
148 2015-07-18 22:45:23.464255 192.168.111.164 -> 192.168.111.123 NFS 130 V4 Reply (Call In 147) LOOKUP | GETFH Status: NFS4ERR_MOVED

b) As per RFC, if the client encounter NFS4ERR_MOVED then it should send the GETATTR with FS_LOCATIONS and MOUNTED_ON_FILEID

[root@rheldesk tmp]# tshark -tad -n -r referral2.pcap -Y ‘rpc.xid == 0x479edcd4’
Running as user “root” and group “root”. This could be dangerous.
145 2015-07-18 22:45:23.462924 192.168.111.123 -> 192.168.111.164 NFS 202 V4 Call LOOKUP DH: 0x62d40c52/share2
146 2015-07-18 22:45:23.463513 192.168.111.164 -> 192.168.111.123 NFS 230 V4 Reply (Call In 145) LOOKUP

Same thing is happening in call 145. It’s asking for filesystem location and mounted_on_fileid. Outputs are truncated.

In frame 145 GETATTR call is going with FSID and FS_LOCATION because of the error in 144.

Opcode: GETATTR (9)
Attr mask[0]: 0x01000100 (FSID, FS_LOCATIONS)
reqd_attr: FSID (8)
reco_attr: FS_LOCATIONS (24)
Attr mask[1]: 0x00800000 (MOUNTED_ON_FILEID)
reco_attr: MOUNTED_ON_FILEID (55)

In frame 146, we can see the reply coming with information of NFS server IP i.e 192.168.111.163

Opcode: GETATTR (9)
Status: NFS4_OK (0)
Attr mask[0]: 0x01000100 (FSID, FS_LOCATIONS)
reqd_attr: FSID (8)
fattr4_fsid
fsid4.major: 134217728
fsid4.minor: 134217728
reco_attr: FS_LOCATIONS (24)
fattr4_fs_locations
pathname components (1)
Name: share2
length: 6
contents: share2
fill bytes: opaque data
fs_location4:
num: 1
fs_location4
server:
num: 1
server: 192.168.111.163
length: 15
contents: 192.168.111.163
fill bytes: opaque data
pathname components (1)
Name: share2
length: 6
contents: share2
fill bytes: opaque data
Attr mask[1]: 0x00800000 (MOUNTED_ON_FILEID)
reco_attr: MOUNTED_ON_FILEID (55)
fileid: 0x000000000106bfe6

c) Now client will start communicating with NFS server 192.168.111.163 to mount the filesystem. In subsequent calls it will check filehandler type whether its volatile or permanent.

[root@rheldesk tmp]# tshark -tad -n -r referral2.pcap -Y ‘nfs && frame.number ge 160’
Running as user “root” and group “root”. This could be dangerous.
160 2015-07-18 22:45:23.467782 192.168.111.123 -> 192.168.111.163 NFS 262 V4 Call SETCLIENTID
161 2015-07-18 22:45:23.468550 192.168.111.163 -> 192.168.111.123 NFS 130 V4 Reply (Call In 160) SETCLIENTID
162 2015-07-18 22:45:23.468669 192.168.111.123 -> 192.168.111.163 NFS 170 V4 Call SETCLIENTID_CONFIRM
165 2015-07-18 22:45:23.470028 192.168.111.163 -> 192.168.111.123 NFS 114 V4 Reply (Call In 162) SETCLIENTID_CONFIRM
167 2015-07-18 22:45:23.470693 192.168.111.163 -> 192.168.111.123 NFS 142 V1 CB_NULL Call
169 2015-07-18 22:45:23.471066 192.168.111.123 -> 192.168.111.163 NFS 94 V1 CB_NULL Reply (Call In 167)

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s