Operating systems recognize a couple special file descriptor IDs:
0- Input usally coming in from keyboard.
1- Output from the application that is meant to consumed by the user, stored in a file, or piped to another application for parsing. Typically goes to terminal.
2- Used for info, debug, and error messages to the user that are not intended to specifically be part of the application output. Typically goes to terminal.
This guide will look at how you can redirect and pipe these streams for better application development and system administration.
STDOUT versus STDERR
To give an example of when to use STDOUT versus STDERR, consider an application that generates a CSV file.
You want to make sure the CSV output is in the proper format so other applications can parse it
properly. Well, what if there are some warnings you want to inform the user about during the
CSV generation process. For example, the user is calling a deprecated method, or they
didn't provide enough values for a row, so you fill them in with empty values.
The warning messages should go to STDERR and the actual CSV contents should go
to STDOUT. This way the user can redirect STDOUT to a file like
leaving STDERR to print out to the terminal for them to review, or piping
Simple piping uses
| character to send the STDOUT
from one application to the STDIN of the next application.
# Send STDOUT from `cat` to the STDIN of `grep` cat file.txt | grep "some text" # You can pipe together many commands echo "hello world" | grep "world" | grep "wor" | grep "wo"
One nice thing about piping in Linux is that each application that is executed is run in parallel, so each application is processing its STDIN and sending its STDOUT as soon as it is received. It does not wait for the first application to completely finish before the next application runs. This makes it very efficient as all applications runs concurrently.
Another benefit is that you can avoid creating intermediary files.
For example, to create a
.tar.gz file, you can do it all it one
# Tarball up the `.py` and `.txt` files, gzip them # and send the final gzipped contents to a `.tar.gz` file. # The dash `-` means output to STDOUT instead of a file tar cf - *.py *.txt | gzip > text_and_py_files.tar.gz
To pipe both STDOUT and STDERR you can use the
echo "both stdout and stderr will get piped out" |& cat # Equivalent to: echo "both stdout and stderr going to same place" 2>&1 | cat
To redirect the standard output to a file, you can run a program like this with the
> angle bracket
followed by the file name. This works in Windows, Mac, and Linux.
# Redirect STDOUT to a file python hello_world.py > output.txt
The example above will overwrite and re-create the file each time it runs. If you use two angle brackets, it will append instead of overwriting the file.
# Append to output.txt python hello_world.py >> output.txt
Redirecting STDERR is similar to redirect STDOUT except
you must specify the file descriptor ID of
# Redirect STDERR to `debug.log` python hello.py 2>debug.log
# Redirect STDOUT to `stdout.log` # Redirect STDERR to `debug.log` python hello.py 1>stdout.log 2>debug.log
You can also redirect one stream to another.
In this example, we first redirect STDOUT (
to a file, and then we redirect to STDERR (
to the new address of
1 (the file).
Now both STDOUT and STDERR are going to
# Redirect STDOUT to `all_output.log` # then redirect STDERR to STDOUT (which goes to the file now) python hello.py 1>all_output.log 2>&1
You can also redirect the input, so instead of coming from your keyboard it comes from a file. Anything you entered in the file will be treated as if you typed it in yourself manually.
For example, if you had a Python 3 script like this:
# pyin.py name = input("What is your name?") email = input("What is your email?") print("Your name is %s and email is %s" % (name, email))
Then in your
input.txt file you had:
You could run the application like this:
python3 pyin.py < input.txt
This is essentially the same thing as running:
cat input.txt | python3 pyin.py
You can even redirect both STDIN and STDOUT at the same time.
python3 pyin.py < input.txt > output.txt
After reading this, you should know the difference between STDOUT and STDERR, and how to redirect them individually to output files. You should also know how to redirect STDIN to come from a file to provide input to an application.