The Linux I/O Redirection: An Introduction

Here we're continue with the series on Linux Basics. This article is the final article on our very important series. The series contains the three most important basics to learn Linxu and here is the fourth one that every newbie should read and understand.

---

You are recommended to read Articles of the series on Linux basics:

• The Linux Terminal : An Introduction

• Basic Linux Navigation And File Management

• The Linux Permissions: An Introduction

Introduction

The redirection capabilities built into Linux provide you with a robust set of tools used to make all sorts of tasks easier to accomplish. Whether you're writing complex software or performing file management through the command line, knowing how to manipulate the different I/O streams in your environment will greatly increase your productivity.

Streams

Input and output in the Linux environment is distributed across three streams. These streams are:

• standard input (stdin)
  
  
• standard output (stdout)
  
  
• standard error (stderr)
  
  

The streams are also numbered:

• stdin (0)
  
  
• stdout (1)
  
  
• stderr (2)
  
  

During standard interactions between the user and the terminal, standard input is transmitted through the user's keyboard. Standard output and standard error are displayed on the user's terminal as text. Collectively, the three streams are referred to as the standard streams.

Standard Input

The standard input stream typically carries data from a user to a program. Programs that expect standard input usually receive input from a device, such as a keyboard. Standard input is terminated by reaching EOF (end-of-file). As described by its name, EOF indicates that there is no more data to be read.

To see standard input in action, run the cat program. Cat stands for concatenate, which means to link or combine something. It is commonly used to combine the contents of two files. When run on its own, cat opens a looping prompt.

 cat  

After opening cat, type a series of numbers as it is running.

 1 
2 
3 
ctrl-d  

When you type a number and press enter, you are sending standard input to the running cat program, which is expecting said input. In turn, the cat program is sending your input back to the terminal display as standard output.

EOF can be input by the user by pressing ctrl-d. After the cat program receives EOF, it stops.

Standard Output

Standard output writes the data that is generated by a program. When the standard output stream is not redirected, it will output text to the terminal. Try the following example:

 echo Sent to the terminal through standard output  

When used without any additional options, the echo command displays any argument that is passed to it on the command line. An argument is something that is received by a program.

Run echo without any arguments:

 echo  

It will return an empty line, since there are no arguments.

Standard Error

Standard error writes the errors generated by a program that has failed at some point in its execution. Like standard output, the default destination for this stream is the terminal display.

When a program's standard error stream is piped to a second program, the piped data (consisting of program errors) is simultaneously sent to the terminal as well.

Let's see a basic example of standard error using the ls command. ls lists a directory's contents.

When run without an argument, ls lists the contents within the current directory. If ls is run with a directory as an argument, it will list the contents of the provided directory.

 ls %   

Since % is not an existing directory, this will send the following text to standard error:

 ls: cannot access %: No such file or directory  

Stream Redirection

Linux includes redirection commands for each stream. These commands write standard output to a file. If a non-existent file is targetted (either by a single-bracket or double-bracket command), a new file with that name will be created prior to writing.

Commands with a single bracket overwrite the destination's existing contents.

Overwrite

• > - standard output
  
  
• < - standard input
  
  
• 2> - standard error
  
  

Commands with a double bracket do not overwrite the destination's existing contents.

Append

• >> - standard output
  
  
• << - standard input
  
  
• 2>> - standard error
  
  

Let's see an example:

  cat > write_to_me.txt  a 
b 
c 
ctrl-d  

Here, cat is being used to write to a file, which is created as a result of the loop.

View the contents of writetome.txt using cat:

 cat write_to_me.txt  

It should have the following contents:

 a
b
c  

Redirect cat to writetome.txt again, and enter three numbers.

 cat > write_to_me.txt  1 
2 
3 
ctrl-d  

When you use cat to view writetome.txt, you will see the following:

 1 
2 
3 
 

The prior contents are no longer there, as the file was overwritten by the single-bracket command.

Do one more cat redirection, this time using double brackets:

 cat >> write_to_me.txt  a 
b 
c 
ctrl-d 

Open writetome.txt again, and you will see this:

 1 
2 
3 
a 
b 
c  

The file now contains text from both uses of cat, as the second one did not override the first one.

Pipes

Pipes are used to redirect a stream from one program to another. When a program's standard output is sent to another through a pipe, the first program's data, which is received by the second program, will not be displayed on the terminal. Only the filtered data returned by the second program will be displayed.

The Linux pipe is represented by a vertical bar.

 *|*  

An example of a command using a pipe:

 ls | less  

This takes the output of ls, which displays the contents of your current directory, and pipes it to the lessprogram. less displays the data sent to it one line at a time.

ls normally displays directory contents across multiple rows. When you run it through less, each entry is placed on a new line.

Though the functionality of the pipe may appear to be similar to that of > and >> (standard output redirect), the distinction is that pipes redirect data from one command to another, while > and >> are used to redirect exclusively to files.

Filters

Filters are commands that alter piped redirection and output. Note that filter commands are also standard Linux commands that can be used without pipes.

• find - Find returns files with filenames that match the argument passed to find.
  
  
• grep - Grep returns text that matches the string pattern passed to grep.
  
  
• tee - Tee redirects standard input to both standard output and one or more files.
  
  
• tr - tr finds-and-replaces one string with another.
  
  
• wc - wc counts characters, lines, and words.
  
  

Example

Now that you have been introduced to redirection, piping, and basic filters, let's look at some basic redirection patterns and examples.

command > file

This pattern redirects the standard output of a command to a file.

ls ~ > root_dir_contents.txt

The command above passes the contents of your system's root directory as standard output, and writes the output to a file named rootdircontents.txt. It will delete any prior contents in the file, as it is a single-bracket command.

command > /dev/null

/dev/null is a special file that is used to trash any data that is redirected to it. It is used to discard standard output that is not needed, and that might otherwise interfere with the functionality of a command or a script. Any output that is sent to /dev/null is discarded.
In the future, you may find the practice of redirecting standard output and standard error to /dev/null when writing shell scripts.

 ls > /dev/null  

This command discards the standard output stream returned from the command ls by passing it to /dev/null.

command 2> file

This pattern redirects the standard error stream of a command to a file, overwriting existing contents.

 mkdir '' 2> mkdir_log.txt  

This redirects the error raised by the invalid directory name '', and writes it to log.txt. Note that the error is still sent to the terminal and displayed as text.

command >> file

This pattern redirects the standard output of a command to a file without overwriting the file's existing contents.

 echo Written to a new file > data.txt 
echo Appended to an existing file's contents >> data.txt  

This pair of commands first redirects the text inputted by the user through echo to a new file. It then appends the text received by the second echo command to the existing file, without overwriting its contents.

command 2>> file

The pattern above redirects the standard error stream of a command to a file without overwriting the file's existing contents. This pattern is useful for creating error logs for a program or service, as the log file will not have its previous content wiped each time the file is written to.

 find '' 2> stderr_log.txt
wc '' 2>> stderr_log.txt 

The above command redirects the error message caused by an invalid find argument to a file named stderr_log.txt. It then appends the error message caused by an invalid wc argument to the same file.

command | command

Redirects the standard output from the first command to the standard input of the second command.

 find /var lib | grep deb  

This command searches through /var and its subfolders for filenames and extensions that match the stringdeb, and returns the file paths for the files, with the matching portion in each path highlighted in red.

command | tee file

This pattern (which includes the tee command) redirects the standard output of the command to a file and overwrites its contents. Then, it displays the redirected output in the terminal. It creates a new file if the file does not already exist.

In the context of this pattern, tee is typically used to view a program's output while simultaneously saving it to a file.

 wc /etc/magic | tee magic_count.txt  

This pipes the counts for characters, lines, and words in the magic file (used by the Linux shell to determine file types) to the tee command, which then splits wc's output in two directions, and sends it to the terminal display and the magic_count.txt file. For the tee command, imagine the letter T. The bottom part of the letter is the initial data, and the top part is the data being split in two different directions (standard output and the terminal).

Multiple pipes can be used to redirect output across multiple commands and/or filters.

command | command | command >> file

This pattern predirects the standard output of the first command and filters it through the next two commands. It then appends the final result to a file.

 ls ~ | grep *tar | tr e E >> ls_log.txt  

This begins by running ls in your root directory (~) and piping the result to the grep command. In this case, grep returns a list of files containing tar in their filename or extension.

The results from grep are then piped to tr, which replaces occurrences of the letter e with E, since e is being passed as the first argument (the string to search for), and E is passed as the second argument (the string that replaces any matches for the first argument). This final result is then appended to the file ls_log.txt, which is created if it does not already exist).

Conclusion

Learning how to use the redirection capabilities built into the Linux command line can be a bit daunting, but you are well on your way to mastering this skillset after completing this tutorial. Now that you have seen the basics of how redirections and pipes work, you'll be able to begin your foray into the world of shell scripting, which makes frequent use of the programs and patterns highlighted in this guide.

If you would like to dig deeper into the commands that were introduced in this tutorial, you can do so withman command | less. For example:

 man tee | less  

This will show you the full list of commands available for the tee program. You can use this pattern to display information and usage options for any Linux command or program.

Googling for specific commands, or for something that you would like to do in the command line (e.g. "delete all files in a directory that begin with an uppercase letter") can also prove helpful when you need to accomplish a specific task using the command line.

Tagged under:

linux basics

ubuntu