How to use rsync efficiently in RHEL ?

rsync command is the common usable command in life of system administrator. In this article I am going to show you the powerful rsync hacks. I have shown three cases in this  article.

  • Case 1 : rsync usage with and without -a option.
  • Case 2 : How to exclude specific files from rsync.
  • Case 3 : How to exclude long list of files and directories.

Case 1 : rsync usage with and without -a option.

Step 1 : I have created 5 files with name from file{1..5} for test purpose each is of size 10M.

[root@Node1 check]# for i in {1..5}
> do
> dd if=/dev/zero of=/check/file$i bs=1M count=10
> done
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.010488 s, 1000 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0109788 s, 955 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0187398 s, 560 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0334705 s, 313 MB/s

[root@Node1 check]# ll
total 92160
-rw-r–r–. 1 root root 10485760 Sep 30 04:07 file1
-rw-r–r–. 1 root root 10485760 Sep 30 04:07 file2
-rw-r–r–. 1 root root 10485760 Sep 30 04:07 file3
-rw-r–r–. 1 root root 10485760 Sep 30 04:07 file4
-rw-r–r–. 1 root root 10485760 Sep 30 04:07 file5

Step 2 : Using rsync to sync the content between two file systems.

Source is /check (ext4 file system)

Destination is /mygfs2 (gfs2 file system)

[root@Node1 check]# time rsync -av /check/* /mygfs2
sending incremental file list
file1
file2
file3
file4
file5

sent 94383806 bytes  received 183 bytes  37753595.60 bytes/sec
total size is 94371840  speedup is 1.00

real    0m1.745s
user    0m0.604s
sys     0m0.774s

Step 3 : Creating some more files in source file system (/check). Files from file{10..15}.

[root@Node1 check]# for i in {10..15}; do dd if=/dev/zero of=/check/file$i bs=1M count=10; done
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0436085 s, 240 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0294622 s, 356 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0316015 s, 332 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.034687 s, 302 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0307658 s, 341 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0291474 s, 360 MB/s

Step 4 : Again issue the rsync command to sync the contents between two file system, it will sync only the new files which are added in previous step. It happens because we have used the -a option with rsync.

[root@Node1 check]# time rsync -av /check/* /mygfs2
sending incremental file list
file10
file11
file12
file13
file14
file15

sent 62922625 bytes  received 126 bytes  41948500.67 bytes/sec
total size is 157286400  speedup is 2.50

real    0m1.096s
user    0m0.448s
sys     0m0.487s

Step 5 : Lets see what happens if we are not using -a option in rsync. I have deleted the content of destination file system.

[root@Node1 check]# cd /mygfs2/

[root@Node1 mygfs2]# rm -rf *

[root@Node1 mygfs2]# cd ..

Step 6 : Syncing the content without -a option.
[root@Node1 /]# time rsync -v /check/* /mygfs2
file1
file11
file12
file13
file14
file15
file2
file3
file4
file5

sent 157306332 bytes  received 297 bytes  62922651.60 bytes/sec
total size is 157286400  speedup is 1.00

real    0m2.317s
user    0m0.995s
sys     0m1.154s

Step 7 : Adding new files in source file system and start syncing the content again with destination. It will sync all the files again which are already synced in step 6.

[root@Node1 /]# for i in {15..20}; do dd if=/dev/zero of=/check/file$i bs=1M count=10; done
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0331174 s, 317 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0143051 s, 733 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.00992653 s, 1.1 GB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0109304 s, 959 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0103564 s, 1.0 GB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.011833 s, 886 MB/s

[root@Node1 /]# time rsync -v /check/* /mygfs2
file1
file11
file12
file13
file14
file15
file16
file17
file18
file19
file2
file20
file3
file4
file5

sent 209741779 bytes  received 392 bytes  59926334.57 bytes/sec
total size is 209715200  speedup is 1.00

real    0m3.383s
user    0m1.361s
sys     0m1.714s

Case 2 : How to exclude specific files from rsync.

Step 1 : I have cleared the content of destination file system. Added some more files with name vicky{1..5} in source file system.

[root@Node1 check]# touch vicky{1..5}
[root@Node1 check]# ll
total 92160
-rw-r–r–. 1 root root 10485760 Sep 30 04:07 file1
-rw-r–r–. 1 root root 10485760 Sep 30 04:07 file2
-rw-r–r–. 1 root root 10485760 Sep 30 04:07 file3
-rw-r–r–. 1 root root 10485760 Sep 30 04:07 file4
-rw-r–r–. 1 root root 10485760 Sep 30 04:07 file5
-rw-r–r–. 1 root root 10485760 Sep 30 04:07 file6
-rw-r–r–. 1 root root 10485760 Sep 30 04:07 file7
-rw-r–r–. 1 root root 10485760 Sep 30 04:07 file8
-rw-r–r–. 1 root root 10485760 Sep 30 04:07 file9
-rw-r–r–. 1 root root        0 Sep 30 05:25 vicky1
-rw-r–r–. 1 root root        0 Sep 30 05:25 vicky2
-rw-r–r–. 1 root root        0 Sep 30 05:25 vicky3
-rw-r–r–. 1 root root        0 Sep 30 05:25 vicky4
-rw-r–r–. 1 root root        0 Sep 30 05:25 vicky5

Step 2 : During syncing between source and destination I am excluding the files which start with “file”. Now it will sync other files except which are starting with “file”.

[root@Node1 mygfs2]# rsync -av –exclude=file* /check/ /mygfs2/
sending incremental file list
./
vicky1
vicky2
vicky3
vicky4
vicky5

sent 267 bytes  received 110 bytes  754.00 bytes/sec
total size is 0  speedup is 0.00

[root@Node1 mygfs2]# cd /mygfs2/

[root@Node1 mygfs2]# ll
total 40
-rw-r–r–. 1 root root 0 Sep 30 05:25 vicky1
-rw-r–r–. 1 root root 0 Sep 30 05:25 vicky2
-rw-r–r–. 1 root root 0 Sep 30 05:25 vicky3
-rw-r–r–. 1 root root 0 Sep 30 05:25 vicky4
-rw-r–r–. 1 root root 0 Sep 30 05:25 vicky5

Case 3 : How to exclude long list of files.

Step 1 : Suppose if I want to exclude lot of files which I don’t want to sync. For this I can create one file including list of files  and directories which I want to exclude. In this case I have created file named “exclude.file”

[root@Node1 /]# cat exclude.file
vicky*
file7
file9

Step 2 : I have cleared the content of destination file system and started the sync again by mentioning that I want to exclude the files and  directories present in /exclude.file.

[root@Node1 /]# rm -rf /mygfs2/*

[root@Node1 /]# rsync -av –exclude-from=/exclude.file /check/ /mygfs2/
sending incremental file list
./
file1
file2
file3
file4
file5
file6
file8

sent 73409649 bytes  received 148 bytes  48939864.67 bytes/sec
total size is 73400320  speedup is 1.00

References : https://access.redhat.com/solutions/223913

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s