How to Remove CTRL-M characters From a File in UNIX and Linux

I wanted to transfer some files from Windows to Unix using FileZilla, but the problem arises when these files are transferred (Ascii or Binary mode both) and opened using VI we get ^M characters, also known as CTRL-M characters. I searched about this but the solutions were to remove these ^M characters when files are transferred using utilities. Is there any way that these ^M characters do not appear in the first place? Well, my search continues but I will share the solution which worked for me for removing control M characters i.e. CTRL-M or ^M characters. There are several UNIX commands e.g. dos2unix which can be used to convert a Windows or DOS generated files to UNIX one. You can also use sed command (stream editor) to remove CTRL-M characters without opening the file, very useful if you are removing CTRL-M characters from a large file. Alternatively, you can use VI command to open the file and replace ^M characters with nothing.

Unix uses a single Line Feed (LF) character as the line break. Windows/DOS uses 2 characters: Carriage Return/Line Feed (CR/LF). The control M characters appear in a file when you transfer then from Windows to UNIX (see How Linux works). So the answer is no, you cannot eliminate the CR\LF (unless you write all your text in one line) but you can still remove those CTRL-M characters by using one of the two mentioned ways:



2 ways to remove Control-M characters from a file in UNIX

Here is a couple of ways to delete control-M characters from a file, in the first way we try to use a command which converts a DOS file to UNIX format and in second way we literally remove control-M characters using UNIX commands e.g. sed or vi

1st Way 

$ dos2unix abc.txt

As the name suggest the dos2unix command convert a DOS or Windows generated file to UNIX file i.e. it replaces all CRLF to LF. See, The Linux Command Line: A Complete Introduction to learning more about dos2unix command.




2nd Way

You may need to do this when you import a text file from MS-DOS or (Window 8, 8.1, 10 or any other version), and forget to transfer it in ASCII or text mode. Here are a couple ways to do it, you can choose the one you like.

The easiest way is probably to use the stream editor sed to remove the ^M characters as shown in the following example

$ sed -e "s/^M//" filename > newfilename

To enter ^M, type CTRL-V, then CTRL-M. I mean, hold down the CTRL key then press V and M in succession. The sed command is really helpful in case of removing ^M characters from large and huge files because you don't need to open the files into VI or any other editor.

Btw, here is a screenshot of a how Control-M character looks like inside a file in UNIX, I am sure you have seen it before:



Btw, you can also remove ^M characters by opening the file in VI editor and replacing them with nothing as shown in the following command:

$ vi filename
Inside vi [in ESC mode] type: :%s/^M//g

To enter ^M, type CTRL-V, then CTRL-M i.e. hold down the CTRL key then press V and M in succession.


If you are an Emacs fan then don't get disappointed because you can also remove CTRL M characters in Emacs by following the below steps:

Go to the beginning of the document
Type: M-x replace-string RET C-q C-m RET RET
where "RET" means and C-q and C-m mean .

See Practical Guide to Linux Commands, Editors, and Shell Programming 3rd Edition by Mark G. Sobell to learn more about UNIX commands and editors.

How to Remove CTRL-M characters From a File in UNIX and Linux


In short, control-m characters will get appended to a file when a file is transferred from windows to UNIX machine. There are four ways it can be removed.


Using vi editor:
:%s/^M//g

Using col command:
$ cat filename | col -b > newfilename

Using sed command:
$ sed 's/^M//g' filename > newfilename

Using dos2unix command:
$ dos2unix filename newfilename

Now, let's see a complete example of removing control m characters from a file. You can see below we have a file with ^M characters, after using sed command those will be removed and verified again by using the cat command.

$ cat -v jsk
hi^M
jsk^M
$ cat jsk | col -b > jsk.new
$ cat -v jsk.new
hi
jsk
$ sed 's/^M//g' jsk > jsk.new2
$ cat -v jsk.new2
hi
jsk

Note: Remember how to type control M characters in UNIX, just hold the control key and then press v and m to get the control-m character.

Other UNIX command tutorials you may like
  • How to create complex directory structure using mkdir -p option (example)
  • 10 Examples of lsof command in UNIX (examples)
  • How to find IP address from hostname in UNIX and Linux? (commands)
  • How to find total size of a directory in UNIX? (command)
  • 10 example of xargs command in UNIX? (see here)
  • 5 example of sort command in UNIX? (see here)
  • 10 basic networking commands in UNIX? (see here)
  • How to close the telnet terminal in UNIX and Windows? (see here)

References
DOS vs Unix Line endings



2 comments :

Joel Frederick said...

The problem with dos2unix is that it isn't installed natively on all systems (not sure it is in any but only had to install it on one).

I had to look it up but you can also use the tr command as follows:

tr -d '^M' file2
or
tr -d '\015' file2

A couple of additional ways found at http://www.theunixschool.com/2011/03/different-ways-to-delete-m-character-in.html

Xakov said...

Hello!
Another way with vi:

:set ff=unix

And you can save your files without ^M

Regards

Post a Comment