Redirect standard error standard output

Redirections from script himself

You could plan redirections from the script itself:

#!/bin/bash

exec 1>>logfile.txt
exec 2>&1

/bin/ls -ld /tmp /tnt

Running this will create/append logfile.txt, containing:

/bin/ls: cannot access '/tnt': No such file or directory
drwxrwxrwt 2 root root 4096 Apr  5 11:20 /tmp

Or

#!/bin/bash

exec 1>>logfile.txt
exec 2>>errfile.txt

/bin/ls -ld /tmp /tnt

While create or append standard output to logfile.txt and create or append errors output to errfile.txt.

Log to many different files

You could create two different logfiles, appending to one overall log and recreating another last log:

#!/bin/bash

if [ -e lastlog.txt ] ;then
    mv -f lastlog.txt lastlog.old
fi
exec 1> >(tee -a overall.log /dev/tty >lastlog.txt)
exec 2>&1

ls -ld /tnt /tmp

Running this script will

• if lastlog.txt already exist, rename them to lastlog.old (overwriting lastlog.old if they exist).
• create a new lastlog.txt.
• append everything to overall.log
• output everything to the terminal.

Simple and combined logs

#!/bin/bash

[ -e lastlog.txt ] && mv -f lastlog.txt lastlog.old
[ -e lasterr.txt ] && mv -f lasterr.txt lasterr.old

exec 1> >(tee -a overall.log combined.log /dev/tty >lastlog.txt)
exec 2> >(tee -a overall.err combined.log /dev/tty >lasterr.txt)

ls -ld /tnt /tmp

So you have

• lastlog.txt last run log file
• lasterr.txt last run error file
• lastlog.old previous run log file
• lasterr.old previous run error file
• overall.log appended overall log file
• overall.err appended overall error file
• combined.log appended overall error and log combined file.
• still output to the terminal

And for interactive session, use stdbuf:

Regarding Fonic’ comment and after some test, I have to agree: with tee, stdbuf is useless. But …

If you plan to use this in *interactive* shell, you must tell `tee` to not buffering his input/output:

# Source this to multi-log your session
[ -e lasterr.txt ] && mv -f lasterr.txt lasterr.old
[ -e lastlog.txt ] && mv -f lastlog.txt lastlog.old
exec 2> >(exec stdbuf -i0 -o0 tee -a overall.err combined.log /dev/tty >lasterr.txt)
exec 1> >(exec stdbuf -i0 -o0 tee -a overall.log combined.log /dev/tty >lastlog.txt)

ls -ld /tnt /tmp

More complex sample

From my 3 remarks about how to Convert Unix timestamp to a date string

I’ve used more complex command to parse and reassemble squid‘s log in real time: As each line begin by an UNIX EPOCH with milliseconds, I split the line on 1st dot, add @ symbol before EPOCH SECONDS to
pass them to date -f - +%F %T then reassemble date‘s output and the rest of line with a dot by using paste -d ..

exec {datesfd}<> <(:)
tail -f /var/log/squid/access.log |
    tee >(
        exec sed -u 's/^([0-9]+)..*/@1/'|
            stdbuf -o0 date -f - +%F %T >&$datesfd
    ) |
        sed -u 's/^[0-9]+.//' |
        paste -d . /dev/fd/$datesfd -

With date, stdbuf was required…

Some explanations about exec and stdbuf commands:

• Running forks by using $(...) or <(...) is done by running subshell wich will execute binaries in another subshell (subsubshell). The exec command tell shell that there are not further command in script to be run, so binary (stdbuf ... tee) will be executed as replacement process, at same level (no need to reserve more memory for running another sub-process).

  From bash‘s man page (man -P'less +/^ *exec ' bash):

      exec [-cl] [-a name] [command [arguments]]
             If  command  is  specified,  it  replaces the
             shell.  No new process is created....
  

  This is not really needed, but reduce system footprint.

• From stdbuf‘s man page:

  NAME
         stdbuf  -  Run COMMAND, with modified buffering
         operations for its standard streams.
  

  This will tell system to use unbuffered I/O for tee command. So all outputs will be updated immediately, when some input are coming.

First thing to note is that there’s couple of ways depending on your purpose and shell, therefore this requires slight understanding of multiple aspects. Additionally, certain commands such as time and strace write output to stderr by default, and may or may not provide a method of redirection specific to that command

Basic theory behind redirection is that a process spawned by shell (assuming it is an external command and not shell built-in) is created via fork() and execve() syscalls, and before that happens another syscall dup2() performs necessary redirects before execve() happens. In that sense, redirections are inherited from the parent shell. The m&>n and m>n.txt inform the shell on how to perform open() and dup2() syscall (see also How input redirection works, What is the difference between redirection and pipe, and What does & exactly mean in output redirection )

Shell redirections

Most typical, is via 2> in Bourne-like shells, such as dash (which is symlinked to /bin/sh) and bash; first is the default and POSIX-compliant shell and the other is what most users use for interactive session. They differ in syntax and features, but luckily for us error stream redirection works the same (except the &> non standard one). In case of csh and its derivatives, the stderr redirection doesn’t quite work there.

Let’s come back to 2> part. Two key things to notice: > means redirection operator, where we open a file and 2 integer stands for stderr file descriptor; in fact this is exactly how POSIX standard for shell language defines redirection in section 2.7:

[n]redir-op word

For simple > redirection, the 1 integer is implied for stdout, i.e. echo Hello World > /dev/null is just the same as echo Hello World 1>/dev/null. Note, that the integer or redirection operator cannot be quoted, otherwise shell doesn’t recognize them as such, and instead treats as literal string of text. As for spacing, it’s important that integer is right next to redirection operator, but file can either be next to redirection operator or not, i.e. command 2>/dev/null and command 2> /dev/null will work just fine.

The somewhat simplified syntax for typical command in shell would be

 command [arg1] [arg2]  2> /dev/null

The trick here is that redirection can appear anywhere. That is both 2> command [arg1] and command 2> [arg1] are valid. Note that for bash shell, there there exists &> way to redirect both stdout and stderr streams at the same time, but again — it’s bash specific and if you’re striving for portability of scripts, it may not work. See also Ubuntu Wiki and What is the difference between &> and 2>&1.

Note: The > redirection operator truncates a file and overwrites it, if the file exists. The 2>> may be used for appending stderr to file.

If you may notice, > is meant for one single command. For scripts, we can redirect stderr stream of the whole script from outside as in myscript.sh 2> /dev/null or we can make use of exec built-in. The exec built-in has the power to rewire the stream for the whole shell session, so to speak, whether interactively or via script. Something like

#!/bin/sh
exec 2> ./my_log_file.txt
stat /etc/non_existing_file

In this example, the log file should show stat: cannot stat '/etc/non_existing_file': No such file or directory.

Yet another way is via functions. As kopciuszek noted in his answer, we can write function declaration with already attached redirection, that is

some_function(){
    command1
    command2
} 2> my_log_file.txt

Commands writing to stderr exclusively

Commands such as time and strace write their output to stderr by default. In case of time command, the only viable alternative is to redirect output of whole command , that is

time echo foo 2>&1 > file.txt

alternatively, synchronous list or subshell could be redirected if you want to separate the output ( as shown in related post ):

{ time sleep 1 2> sleep.stderr ; } 2> time.txt

Other commands, such as strace or dialog provide means to redirect stderr. strace has -o <filename.txt> option which allows specifying filename where output should be written. There is also an option for writing a textfile for each subprocess that strace sees. The dialog command writes the text user interface to stdout but output to stderr, so in order to save its output to variable ( because var=$(...) and pipelines only receives stderr ) we need to swap the file descriptors

result=$(dialog --inputbox test 0 0 2>&1 1>/dev/tty);

but additionally, there is --output-fd flag, which we also can utilize. There’s also the method of named pipes. I recommend reading the linked post about the dialog command for thorough description of what’s happening.

# Append vs Truncate

# Truncate >

1. Create specified file if it does not exist.
2. Truncate (remove file’s content)
3. Write to file

# Append >>

1. Create specified file if it does not exist.
2. Append file (writing at end of file).

# Redirecting both STDOUT and STDERR

File descriptors like 0 and 1 are pointers. We change what file descriptors point to with redirection. >/dev/null means 1 points to /dev/null.

First we point 1 (STDOUT) to /dev/null then point 2 (STDERR) to whatever 1 points to.

This can be further shortened to the following:

However, this form may be undesirable in production if shell compatibility is a concern as it conflicts with POSIX, introduces parsing ambiguity, and shells without this feature will misinterpret it:

NOTE: &> is known to work as desired in both Bash and Zsh.

# Redirecting standard output

> redirect the standard output (aka STDOUT) of the current command into a file or another descriptor.

These examples write the output of the ls command into the file file.txt

The target file is created if it doesn’t exists, otherwise this file is truncated.

The default redirection descriptor is the standard output or 1 when none is specified.
This command is equivalent to the previous examples with the standard output explicitly indicated:

Note: the redirection is initialized by the executed shell and not by the executed command, therefore it is done before the command execution.

# Using named pipes

Sometimes you may want to output something by one program and input it into another program, but can’t use a standard pipe.

You could simply write to a temporary file:

This works fine for most applications, however, nobody will know what tempFile does and someone might remove it if it contains the output of ls -l in that directory. This is where a named pipe comes into play:

myPipe is technically a file (everything is in Linux), so let’s do ls -l in an empty directory that we just created a pipe in:

The output is:

Notice the first character in the permissions, it’s listed as a pipe, not a file.

Now let’s do something cool.

Open one terminal, and make note of the directory (or create one so that cleanup is easy), and make a pipe.

Now let’s put something in the pipe.

You’ll notice this hangs, the other side of the pipe is still closed. Let’s open up the other side of the pipe and let that stuff through.

Open another terminal and go to the directory that the pipe is in (or if you know it, prepend it to the pipe):

You’ll notice that after hello from the other side is output, the program in the first terminal finishes, as does that in the second terminal.

Now run the commands in reverse. Start with cat < myPipe and then echo something into it. It still works, because a program will wait until something is put into the pipe before terminating, because it knows it has to get something.

Named pipes can be useful for moving information between terminals or between programs.

Pipes are small. Once full, the writer blocks until some reader reads the contents, so you need to either run the reader and writer in different terminals or run one or the other in the background:

More examples using named pipes:

# Redirecting multiple commands to the same file

# Print error messages to stderr

Error messages are generally included in a script for debugging purposes or for providing rich user experience. Simply writing error message like this:

may work for simple cases but it’s not the usual way. In this example, the error message will pollute the actual output of the script by mixing both errors and successful output in stdout.

In short, error message should go to stderr not stdout. It’s pretty simple:

Another example:

In the above example, the success message will be printed on stdout while the error message will be printed on stderr.

A better way to print error message is to define a function:

Now, when you have to print an error:

# Redirection to network addresses

Bash treats some paths as special and can do some network communication by writing to /dev/{udp|tcp}/host/port. Bash cannot setup a listening server, but can initiate a connection, and for TCP can read the results at least.

For example, to send a simple web request one could do:

and the results of www.google.com‘s default web page will be printed to stdout.

Similarly

would send a UDP message containing HIn to a listener on 192.168.1.1:6666

# Redirecting STDIN

< reads from its right argument and writes to its left argument.

To write a file into STDIN we should read /tmp/a_file and write into STDIN i.e 0</tmp/a_file

Note: Internal file descriptor defaults to 0 (STDIN) for <

# Redirecting STDERR

2 is STDERR.

Definitions:

echo_to_stderr is a command that writes "stderr" to STDERR

# STDIN, STDOUT and STDERR explained

Commands have one input (STDIN) and two kinds of outputs, standard output (STDOUT) and standard error (STDERR).

For example:

STDIN

Standard input is used to provide input to a program.
(Here we’re using the read builtin (opens new window) to read a line from STDIN.)

STDOUT

Standard output is generally used for «normal» output from a command. For example, ls lists files, so the files are sent to STDOUT.

STDERR

Standard error is (as the name implies) used for error messages. Because this message is not a list of files, it is sent to STDERR.

STDIN, STDOUT and STDERR are the three standard streams. They are identified to the shell by a number rather than a name:

0 = Standard in
1 = Standard out
2 = Standard error

By default, STDIN is attached to the keyboard, and both STDOUT and STDERR appear in the terminal. However, we can redirect either STDOUT or STDERR to whatever we need. For example, let’s say that you only need the standard out and all error messages printed on standard error should be suppressed. That’s when we use the descriptors 1 and 2.

Redirecting STDERR to /dev/null
Taking the previous example,

In this case, if there is any STDERR, it will be redirected to /dev/null (a special file which ignores anything put into it), so you won’t get any error output on the shell.

# Syntax

• command </path/to/file # Redirect standard input to file
• command >/path/to/file # Redirect standard output to flie
• command file_descriptor>/path/to/file # Redirect output of file_descriptor to file
• command >&file_descriptor # Redirect output to file_descriptor
• command file_descriptor>&another_file_descriptor # Redirect file_descriptor to another_file_descriptor
• command <&file_descriptor # Redirect file_descriptor to standard input
• command &>/path/to/file # Redirect standard output and standard error to file

# Parameters

UNIX console programs have an input file and two output files (input and output streams, as well as devices, are treated as files by the OS.) These are typically the keyboard and screen, respectively, but any or all of them can be redirected to come from — or go to — a file or other program.

STDIN is standard input, and is how the program receives interactive input. STDIN is usually assigned file descriptor 0.

STDOUT is standard output. Whatever is emitted on STDOUT is considered the «result» of the program. STDOUT is usually assigned file descriptor 1.

STDERR is where error messages are displayed. Typically, when running a program from the console, STDERR is output on the screen and is indistinguishable from STDOUT. STDERR is usually assigned file descriptor 2.

The order of redirection is important

Redirects both (STDOUT and STDERR) to the file.

Redirects only STDOUT, because the file descriptor 2 is redirected to the file pointed to by file descriptor 1 (which is not the file file yet when the statement is evaluated).

Each command in a pipeline has its own STDERR (and STDOUT) because each is a new process. This can create surprising results if you expect a redirect to affect the entire pipeline. For example this command (wrapped for legibility):

will print «Python error!» to the console rather than the log file. Instead, attach the error to the command you want to capture:

When we use > to redirect Standard Output, CMD.EXE interprets this as 1>, as can be seen by writing and running this one-line batch file «test.bat»:

DIR > NUL

Now run test.bat in CMD.EXE and watch the result:

C:>test.bat

C:>DIR  1>NUL

C:>_

It looks like CMD.EXE uses 1 for Standard Output and 2 for Standard Error.
We’ll see how we can use this later.

Nothing is impossible, not even redirecting the Console output.

Unfortunately, it can be done only in the old MS-DOS versions that came with a CTTY command.

The general idea was this:

CTTY NUL
ECHO Echo whatever you want, it won't be displayed on screen no matter what.
ECHO By the way, did I warn you that the keyboard doesn't work either?
ECHO I suppose that's why CTTY is no longer available on Windows systems.
ECHO The only way to get control over the computer again is a cold reboot,
ECHO or the following command:
CTTY CON

A pause or prompt for input before the CTTY CON command meant one had to press the reset button!

Besides being used for redirection to the NUL device, with CTTY COM1 the control could be passed on to a terminal on serial port COM1.

A safer way to redirect STARTed commands’ output would be to create and run a «wrapper» batch file that handles the redirection.
The batch file would look like this:

command > logfile

and the command line would be:

START batchfile

page last modified: 2016-09-19