Monday, May 20, 2013

SED Command Examples in UNIX and Linux, Find and Replace using Regular Expression

SED command in UNIX  is stands for stream editor and it can perform lot's of function on file like, searching, find and replace, insertion or deletion. Though most common use of SED command in UNIX is for substitution or for find and replace. By using SED you can edit files even without opening it, which is much quicker way to find and replace something in file, than first opening that file in VI Editor and then changing it. In this SED command tutorial we will see some practical examples of SED command in UNIX based systems e.g. Linux. I must say having a good grip on find, grep, sort, vi editor and SED can take you next level of UNIX and Linux working experience. These are very powerful UNIX command and helps with lot of different tasks in server. By the way we will use following text file for our SED common example. As I have said before, best way to learn any UNIX command is to use them in your day to day task, and a good example is a good start. This file contains details of some popular android and iPhone smartphones, e.g. Model, company, price etc, separated by colon. You can also use any CSV file for this example.

$ cat list-of-smartphones-2011.txt
Model:Company:Price:Camera:3G
IPhone4:Apple:1000$:Yes:Yes
Galaxy:Samsung:900$:Yes:Yes
Optimus:LG:800$:Yes:Yes
Sensation:HTC:400$:Yes:Yes
IPhone4S:Apple:1100$:Yes:Yes
N9:Nokia:400$:Yes:Yes

Few things to keep in mind about  SED in UNIX
1) SED is a powerful text stream editor. Can do insertion, deletion, search and replace(substitution).
2) SED command in unix supports regular expression which allows it perform complex pattern matching.

SED command to Remove a Line from File in UNIX:

SED command in UNIX to search string in fileUsing SED command to delete lines or remove lines is one of the popular use of SED command in unix. most of the time you either delete first line or last line in unix. "d" flag is used to delete lines in SED. Following example of SED command in unix will show how to delete first and last line in UNIX based system e.g. Linux:

sed command to delete first line

$ sed '1d' list-of-smartphones-2011.txt

sed command to delete last line

$ sed '$d' list-of-smartphones-2011.txt

sed command to delete from line 1,4

$ sed '1,4d' list-of-smartphones-2011.txt
Sensation:HTC:400$:Yes:Yes
IPhone4S:Apple:1100$:Yes:Yes
N9:Nokia:400$:Yes:Yes

quote around "d" is not mandatory but its good practice and increase readability.

SED Command to Remove blank lines

One of the common usage of SED command is to delete empty lines or remove blank lines from files, without opening them. Unix sysadmin often use SED command in Linux to clean up files by removing empty lines. Following example of UNIX SED command will show you how to remove blank lines using SED in unix:

sed '/^$/d' list-of-smartphones-2011.txt

here text between /--/ is a regular expression for matching matching empty lines i.e. “^$”. Since “^” denote start of line and “$” denote end of line, “^$” means empty lines.

I have inserted 3 blank line in our example file and now number of lines are 10

$ cat list-of-smartphones-2011.txt | wc -l
10

$ sed '/^$/d' list-of-smartphones-2011.txt | wc -l
7

with this same technique we can remove spaces, or any lines which is matching to a string by using sed command in unix.

SED Command to Remove Matching String in UNIX

In this SED command example, we will learn how to remove lines which are matching to particular text.

$ sed '/Apple/d' list-of-smartphones-2011.txt
Model:Company:Price:Camera:3G
Galaxy:Samsung:900$:Yes:Yes
Optimus:LG:800$:Yes:Yes
Sensation:HTC:400$:Yes:Yes
N9:Nokia:400$:Yes:Yes

Above sed command in unix has already removed all lines which matches string "Apple" in it. You can see that, there is no line which contains “Apple”, because they are already removed.

UNIX SED Command to Find and Replace text from File

One of my favorite use of SED command in UNIX is, to find and replace strings or patterns without opening file. Though you can also do find and replace in VI, sed is much faster for such operation, especially if you are dealing with large files which can take relative long time to open in Vi editor. Linux SED command support regular expression which allows you to peform sophisticated find and replace in unix. "s" flag is used for substitution in sed and it works mostly like in VI.

SED command to find and replace:

sed 's/Apple/Microsoft/' list-of-smartphones-2011.txt

In below example of unix SED command we will find "Apple" in line and replace it with "Microsoft" string.

$ sed 's/Apple/Microsoft/' list-of-smartphones-2011.txt
Model:Company:Price:Camera:3G
IPhone4:Microsoft:1000$:Yes:Yes Apple
Galaxy:Samsung:900$:Yes:Yes
Optimus:LG:800$:Yes:Yes
Sensation:HTC:400$:Yes:Yes
IPhone4S:Microsoft:1100$:Yes:Yes
N9:Nokia:400$:Yes:Yes

An important thing is to note here is that it will only replace first occurrence of matching string on each line. Suppose if you have two matching string in one line , second will not be deleted. You can see this in second line of above SED command example, Apple is not replaced by Microsoft, to remove all occurrence of matching string use "g" at the end.

$ sed 's/Apple/Microsoft/g' list-of-smartphones-2011.txt
Model:Company:Price:Camera:3G
IPhone4:Microsoft:1000$:Yes:Yes Microsoft
Galaxy:Samsung:900$:Yes:Yes
Optimus:LG:800$:Yes:Yes
Sensation:HTC:400$:Yes:Yes
IPhone4S:Microsoft:1100$:Yes:Yes
N9:Nokia:400$:Yes:Yes


That’s all on How to use SED command in UNIX based operating systems like Linux or Solaris. We have seen some of the most common examples of SED command, e.g. doing find and replace in file without opening it.

3 comments :

Anonymous said...

What is the best way to find and replace from large files in UNIX? For example, you need to find all occurrences of word "INFO" and has to replace with "DEBUG", how will you do it and which is the fastest way to do that? This was the complete question asked on one UNIX interviews. My answer was by using SED command, e.g.

sed s/INFO/DEBUG/g application.log

But I want to know from experts, whether that was right answer or not, with explanation?

Anonymous said...

sed 's/INFO/DEBUG/g' application.log

Anonymous said...

Insert a row as a first row into a large files in UNIX.

Post a Comment