Category Archives: Basic Usage

Usage of awk Command to print and filter the data.

In this article I am going to show the basic usage of awk commands. I am using the below file to show the various usages.

[root@node1 ~]# cat data1
Item1,200
Item2,500
Item3,900
Item2,800
Item1,600

Case 1 : If you want to print the string or line equal to number of records in file. 

[root@node1 ~]# awk ‘{print “Hello world”}’ data1
Hello world
Hello world
Hello world
Hello world
Hello world

Case 2 : Lets add some spice by adding 1 at end of above expression. It means by default true condition. It will print string and each Record of file alternatively.

[root@node1 ~]# awk ‘{print “Hello world”}1’ data1
Hello world
Item1,200
Hello world
Item2,500
Hello world
Item3,900
Hello world
Item2,800
Hello world
Item1,600

Case 3 : Using the BEGIN in awk command to print the message before starting with processing on file. Now see the below two commands and compare the output with Case 1 and Case 2.

[root@node1 ~]# awk ‘BEGIN{print “Hello World”}’ data1
Hello World

[root@node1 ~]# awk ‘BEGIN{print “Hello World”}1’ data1
Hello World
Item1,200
Item2,500
Item3,900
Item2,800
Item1,600

Case 4 : How to print the selective columns in awk. By default Column separator or Field separator is space We have modified it to “,” in below case. Because our input is separated by “,” instead of blank space. where NF stands for number of fields.

[root@node1 ~]# awk -F, ‘{print $1,$2,NF}’ data1
Item1 200 2
Item2 500 2
Item3 900 2
Item2 800 2
Item1 600 2

Case 5 : As we are having BEGIN similarly we are having END statement as well which is used to print the data after processing of the file.

[root@node1 ~]# awk -F, ‘{print $1,$2;} END { print NR, ” records are processed”;}’ data1
Item1 200
Item2 500
Item3 900
Item2 800
Item1 600
5 records are processed

Case 6 : If we are changing NF to $NF then it is printing the value of last column.

[root@node1 ~]# awk -F, ‘{print $1,$2,$NF}’ data1
Item1 200 200
Item2 500 500
Item3 900 900
Item2 800 800
Item1 600 600

Case 7 : As I mentioned earlier by default filed separator is space we have changed it to “,”. In this case we are going to change it to new line. RS stands for record separator by default it is “\n” it has been changed also. Now it will show us 5 fields or columns are present which are separated by “\n”

[root@node1 ~]# awk ‘BEGIN { FS=”\n”; RS=””} {print $1,NF}’ data1
Item1,200 5

[root@node1 ~]# awk ‘BEGIN { FS=”\n”; RS=””} {print $1,NR}’ data1
Item1,200 1

[root@node1 ~]# awk ‘BEGIN { FS=”\n”; RS=””} {print $1,$2}’ data1
Item1,200 Item2,500

Case 8 : Similar to FS there is OFS (Output field separator) which by default is space and ORS (Output Record Separator) which by default is “\n”.

[root@node1 ~]# awk ‘BEGIN { FS=”\n”; RS=””; OFS=”,”} {print $1,$2}’ data1
Item1,200,Item2,500

[root@node1 ~]# awk ‘BEGIN {ORS=” “} {print $1} ‘ data1
Item1,200 Item2,500 Item3,900 Item2,800 Item1,600 [root@node1 ~]#

[root@node1 ~]# awk ‘BEGIN {ORS=”\n\n”} {print $1} ‘ data1
Item1,200

Item2,500

Item3,900

Item2,800

Item1,600

  •  Relational operator help to filter the output. I have put the output of vmstat command into file named data11.

[root@node1 ~]# cat data11
procs ———–memory———- —swap– —–io—- –system– —–cpu—–
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 91592 120576 250632 0 0 22 20 82 175 0 1 98 1 0
0 0 0 91568 120576 250636 0 0 0 0 70 154 0 0 100 0 0
0 0 0 91568 120576 250636 0 0 0 38 77 164 0 1 100 0 0
0 0 0 91568 120576 250636 0 0 0 30 78 185 0 1 93 6 0

Case 1 : Here I am applying filter on number of records greater than 2 and number of fields greater than 6. By filtering output in that way printing the first and fourth column.

[root@node1 ~]# awk ‘(NF>6 && NR > 2){print $1,$4}’ data11
0 91592
0 91568
0 91568
0 91568

Case 2 : Extending the above condition to CPU idle value.

[root@node1 ~]# awk ‘(NF>6 && NR > 2 && $15 < 95){print $1,$4}’ data11
0 91568

 

Advertisements

Dealing with multiline pattern using sed

In this post I am going to show you in simple way the difference while dealing with single line and multiline patterns.

I am taking the below file as example to show you the differences.

[root@node1 ~]# cat data7
hello world
my name is vikrant
i am happy

Case 1 : Captial N will read the new line of input and append it to contents of pattern space. They are separated by \n (newline).

In below example I am searching of vikrant at end of every line in data7 file. I found it on second line and after that I read the next line using N I substitue the \n with space option by which newline and line selected as per pattern are concatenated, to give the better understanding.

[root@node1 ~]# sed ‘/vikrant$/{N;s/\n/ /;}’ data7
hello world
my name is vikrant i am happy

Now if I am going to do the same thing by replacing N with small n output is different. Because n will print the pattern space then It will read the new line. It doesn’t created multiline pattern space as in case of N

[root@node1 ~]# sed ‘/vikrant$/{n;s/\n/ /;}’ data7
hello world
my name is vikrant
i am happy

Case 2 : Captial “P” is used to output the first portion of a multiliner pattern space up to embedded newline.

Continuing for previous case. Showing the usage of P vs p. after reading the line “my name is vikrant \n i am happy” But due to P option It is displaying only first part again i.e. “my name is vikrant”

[root@node1 ~]# sed ‘/vikrant$/{N;P}’ data7
hello world
my name is vikrant
my name is vikrant
i am happy

But p will display the whole thing again “my name is vikrant \n i am happy”

On other hand small “p” will display the whole portion included embedded new line.

[root@node1 ~]# sed ‘/vikrant$/{N;p}’ data7
hello world
my name is vikrant
i am happy
my name is vikrant
i am happy

Case 3 : Now moving with D and d.

Extending scenario from Case 2. As P will print only the first pattern space not the embedded one. In same way D will delete only the first pattern space. Now combination of N,P,D can be used to work on multiline.

[root@node1 ~]# sed ‘/vikrant$/{N;P;D;}’ data7
hello world
my name is vikrant
i am happy

If we are trying to do the same thing with d it will delete the whole line included the embedded one.

[root@node1 ~]# sed ‘/vikrant$/{N;P;d;}’ data7
hello world
my name is vikrant

 

How to delete lines with sed

In this post I am going to show you the use of delete in sed. I am using file named as data2 below are the contents of that file. None of the change in below cases is in place means the change will not get reflect in file content It will be shown on standard output only. To make the changes in place use -i option after sed in all commands.

[root@node1 network]# cat data2
I am unix engineer
Unix
UNIX
Linux
linux
LINUX
I work on Linux also.

Case 1 : How to delete specific lines from file.

[root@node1 network]# sed ‘1d;3d;5d’ data2
Unix
Linux
LINUX
I work on Linux also.

Case 2 : If want to delete range of lines.

[root@node1 network]# sed ‘1,5d’ data2
LINUX
I work on Linux also.

Case 3 : Deleting every line except 1 and 2.

[root@node1 network]# sed ‘1,2!d’ data2
I am unix engineer
Unix

Case 4 : How to delete line containing particular word. But it will not do case sensitive check.

[root@node1 network]# sed ‘/Linux/d’ data2
I am unix engineer
Unix
UNIX
linux
LINUX

Case 5 : Ignoring the case while deleting

[root@node1 network]# sed ‘/Linux/Id’ data2
I am unix engineer
Unix
UNIX

Case 6 : Will delete only lines starting with linux not those who containted word in line.

[root@node1 network]# sed ‘/^linux/Id’ data2
I am unix engineer
Unix
UNIX
I work on Linux also.

Case 7 : Using or in sed command Will delete lines containing “unix” or “linux” Point to be noted here it is not case sensitive.

[root@node1 network]# sed ‘/unix\|linux/d’ data2
Unix
UNIX
Linux
LINUX
I work on Linux also.

Case 8 : It will delete the unix (ignore case) starting from second line. Left the unix in first line as it is.

[root@node1 network]# sed ‘2,/unix/Id’ data2
I am unix engineer
Linux
linux
LINUX
I work on Linux also.

Case 9 : Delete the lines from 1 to 3 if they contain pattern unix or linux ignore case.

[root@node1 network]# sed ‘1,3{/unix\|linux/Id;}’ data2
Linux
linux
LINUX
I work on Linux also.

Case 10 : Delete the last line if it contains linux word in ignore case.

[root@node1 network]# sed ‘${/linux/Id;}’ data2
I am unix engineer
Unix
UNIX
Linux
linux
LINUX

 

How to use sed to append, delete and insert lines in file

In this post i am going to show you the basic usage of sed command while inserting, deleting and appending lines to file.

I have created one basic file name as data1. Below is the content of that. I am going to do operations on this file. One thing to remember here is that changes made by sed are not in place means they will not reflect in file. We have to use -i option to reflect them in file. Here -i means in place.

[root@node1 network]# cat data1
vikrant 8800
danish 10000
venky 12000

Case 1 : If we want to insert adding heading to file. Compare the output with previous one. Here heading is added before it start reading the file.

[root@node1 network]# sed ‘1i Employee Salary’ data1
Employee Salary
vikrant 8800
danish 10000
venky 12000

Case 2 : Above change will not be in file If we want to reflect that in file then use -i option.

[root@node1 network]# sed -i ‘1i Employee Salary’ data1
[root@node1 network]# cat data1
Employee Salary
vikrant 8800
danish 10000
venky 12000

Case 3 : How to add line after the first line. 

[root@node1 network]# sed -i ‘1a ————–‘ data1
[root@node1 network]# cat data1
Employee Salary
————–
vikrant 8800
danish 10000
venky 12000

Case 4 : How to append the line at end of file. It will read the whole file at end of last line It will add the mentioned line.

[root@node1 network]# sed -i ‘$a ————–‘ data1
[root@node1 network]# cat data1
Employee Salary
————–
vikrant 8800
danish 10000
venky 12000
————–

Case 5 : How to insert line before any specific line.  Here I am adding new line before line starting with venky.

[root@node1 network]# cat data1
Employee Salary
————–
vikrant 8800
danish 10000
venky 12000
————–
[root@node1 network]# sed -i ‘/venky/i madhu 6000’ data1
[root@node1 network]# cat data1
Employee Salary
————–
vikrant 8800
danish 10000
madhu 6000
venky 12000

Case 6 : Deleting the newly added line.

[root@node1 network]# sed -i ‘/madhu/d’ data1
[root@node1 network]# cat data1
Employee Salary
————–
vikrant 8800
danish 10000
venky 12000
————–

Case 7 : Using the same scenario of 5. Adding the line after specified line.

[root@node1 network]# sed -i ‘/venky/a madhu 6000’ data1
[root@node1 network]# cat data1
Employee Salary
————–
vikrant 8800
danish 10000
venky 12000
madhu 6000
————–