Redirect error output bash

Redirections from script himself

You could plan redirections from the script itself:

#!/bin/bash

exec 1>>logfile.txt
exec 2>&1

/bin/ls -ld /tmp /tnt

Running this will create/append logfile.txt, containing:

/bin/ls: cannot access '/tnt': No such file or directory
drwxrwxrwt 2 root root 4096 Apr  5 11:20 /tmp

Or

#!/bin/bash

exec 1>>logfile.txt
exec 2>>errfile.txt

/bin/ls -ld /tmp /tnt

While create or append standard output to logfile.txt and create or append errors output to errfile.txt.

Log to many different files

You could create two different logfiles, appending to one overall log and recreating another last log:

#!/bin/bash

if [ -e lastlog.txt ] ;then
    mv -f lastlog.txt lastlog.old
fi
exec 1> >(tee -a overall.log /dev/tty >lastlog.txt)
exec 2>&1

ls -ld /tnt /tmp

Running this script will

• if lastlog.txt already exist, rename them to lastlog.old (overwriting lastlog.old if they exist).
• create a new lastlog.txt.
• append everything to overall.log
• output everything to the terminal.

Simple and combined logs

#!/bin/bash

[ -e lastlog.txt ] && mv -f lastlog.txt lastlog.old
[ -e lasterr.txt ] && mv -f lasterr.txt lasterr.old

exec 1> >(tee -a overall.log combined.log /dev/tty >lastlog.txt)
exec 2> >(tee -a overall.err combined.log /dev/tty >lasterr.txt)

ls -ld /tnt /tmp

So you have

• lastlog.txt last run log file
• lasterr.txt last run error file
• lastlog.old previous run log file
• lasterr.old previous run error file
• overall.log appended overall log file
• overall.err appended overall error file
• combined.log appended overall error and log combined file.
• still output to the terminal

And for interactive session, use stdbuf:

Regarding Fonic’ comment and after some test, I have to agree: with tee, stdbuf is useless. But …

If you plan to use this in *interactive* shell, you must tell `tee` to not buffering his input/output:

# Source this to multi-log your session
[ -e lasterr.txt ] && mv -f lasterr.txt lasterr.old
[ -e lastlog.txt ] && mv -f lastlog.txt lastlog.old
exec 2> >(exec stdbuf -i0 -o0 tee -a overall.err combined.log /dev/tty >lasterr.txt)
exec 1> >(exec stdbuf -i0 -o0 tee -a overall.log combined.log /dev/tty >lastlog.txt)

ls -ld /tnt /tmp

More complex sample

From my 3 remarks about how to Convert Unix timestamp to a date string

I’ve used more complex command to parse and reassemble squid‘s log in real time: As each line begin by an UNIX EPOCH with milliseconds, I split the line on 1st dot, add @ symbol before EPOCH SECONDS to
pass them to date -f - +%F %T then reassemble date‘s output and the rest of line with a dot by using paste -d ..

exec {datesfd}<> <(:)
tail -f /var/log/squid/access.log |
    tee >(
        exec sed -u 's/^([0-9]+)..*/@1/'|
            stdbuf -o0 date -f - +%F %T >&$datesfd
    ) |
        sed -u 's/^[0-9]+.//' |
        paste -d . /dev/fd/$datesfd -

With date, stdbuf was required…

Some explanations about exec and stdbuf commands:

• Running forks by using $(...) or <(...) is done by running subshell wich will execute binaries in another subshell (subsubshell). The exec command tell shell that there are not further command in script to be run, so binary (stdbuf ... tee) will be executed as replacement process, at same level (no need to reserve more memory for running another sub-process).

  From bash‘s man page (man -P'less +/^ *exec ' bash):

      exec [-cl] [-a name] [command [arguments]]
             If  command  is  specified,  it  replaces the
             shell.  No new process is created....
  

  This is not really needed, but reduce system footprint.

• From stdbuf‘s man page:

  NAME
         stdbuf  -  Run COMMAND, with modified buffering
         operations for its standard streams.
  

  This will tell system to use unbuffered I/O for tee command. So all outputs will be updated immediately, when some input are coming.

# this executes the cat-command and redirects its error messages (stderr) to the bit bucket
cat some_file.txt 2>/dev/null

# this executes the echo-command and redirects its normal output (stdout) to the standard error target
echo "There was an error" 1>&2

cat foo.txt bar.txt >new.txt
cat >new.txt foo.txt bar.txt
>new.txt cat foo.txt bar.txt

Valid redirection targets and sources

N > TARGET

N >> TARGET

Redirecting output and error output

&> TARGET

>& TARGET

> TARGET 2>&1

Appending redirected output and error output

>> FILE 2>&1

&>> FILE

Transporting stdout and stderr through a pipe

COMMAND1 2>&1 | COMMAND2

COMMAND1 |& COMMAND2

N < SOURCE

<<TAG
...
TAG

<<-TAG
...
TAG

# display help

cat <<EOF
Sorry...
No help available yet for $PROGRAM.
Hehe...
EOF

cat <<"EOF"
This won't be expanded: $PATH
EOF

<<< WORD

cat <<< "Hello world... $NAME is here..."

# { echo OUTPUT; echo ERRORS >&2; } is to simulate something that outputs to STDOUT and STDERR
# you can test with it
{ echo OUTPUT; echo ERRORS >&2; } 2>&1 1>/dev/null

{ echo OUTPUT; echo ERRORS >&2; } 1>/dev/null 2>&1

command >/dev/null 2>&1

# Append vs Truncate

# Truncate >

1. Create specified file if it does not exist.
2. Truncate (remove file’s content)
3. Write to file

# Append >>

1. Create specified file if it does not exist.
2. Append file (writing at end of file).

# Redirecting both STDOUT and STDERR

File descriptors like 0 and 1 are pointers. We change what file descriptors point to with redirection. >/dev/null means 1 points to /dev/null.

First we point 1 (STDOUT) to /dev/null then point 2 (STDERR) to whatever 1 points to.

This can be further shortened to the following:

However, this form may be undesirable in production if shell compatibility is a concern as it conflicts with POSIX, introduces parsing ambiguity, and shells without this feature will misinterpret it:

NOTE: &> is known to work as desired in both Bash and Zsh.

# Redirecting standard output

> redirect the standard output (aka STDOUT) of the current command into a file or another descriptor.

These examples write the output of the ls command into the file file.txt

The target file is created if it doesn’t exists, otherwise this file is truncated.

The default redirection descriptor is the standard output or 1 when none is specified.
This command is equivalent to the previous examples with the standard output explicitly indicated:

Note: the redirection is initialized by the executed shell and not by the executed command, therefore it is done before the command execution.

# Using named pipes

Sometimes you may want to output something by one program and input it into another program, but can’t use a standard pipe.

You could simply write to a temporary file:

This works fine for most applications, however, nobody will know what tempFile does and someone might remove it if it contains the output of ls -l in that directory. This is where a named pipe comes into play:

myPipe is technically a file (everything is in Linux), so let’s do ls -l in an empty directory that we just created a pipe in:

The output is:

Notice the first character in the permissions, it’s listed as a pipe, not a file.

Now let’s do something cool.

Open one terminal, and make note of the directory (or create one so that cleanup is easy), and make a pipe.

Now let’s put something in the pipe.

You’ll notice this hangs, the other side of the pipe is still closed. Let’s open up the other side of the pipe and let that stuff through.

Open another terminal and go to the directory that the pipe is in (or if you know it, prepend it to the pipe):

You’ll notice that after hello from the other side is output, the program in the first terminal finishes, as does that in the second terminal.

Now run the commands in reverse. Start with cat < myPipe and then echo something into it. It still works, because a program will wait until something is put into the pipe before terminating, because it knows it has to get something.

Named pipes can be useful for moving information between terminals or between programs.

Pipes are small. Once full, the writer blocks until some reader reads the contents, so you need to either run the reader and writer in different terminals or run one or the other in the background:

More examples using named pipes:

# Redirecting multiple commands to the same file

# Print error messages to stderr

Error messages are generally included in a script for debugging purposes or for providing rich user experience. Simply writing error message like this:

may work for simple cases but it’s not the usual way. In this example, the error message will pollute the actual output of the script by mixing both errors and successful output in stdout.

In short, error message should go to stderr not stdout. It’s pretty simple:

Another example:

In the above example, the success message will be printed on stdout while the error message will be printed on stderr.

A better way to print error message is to define a function:

Now, when you have to print an error:

# Redirection to network addresses

Bash treats some paths as special and can do some network communication by writing to /dev/{udp|tcp}/host/port. Bash cannot setup a listening server, but can initiate a connection, and for TCP can read the results at least.

For example, to send a simple web request one could do:

and the results of www.google.com‘s default web page will be printed to stdout.

Similarly

would send a UDP message containing HIn to a listener on 192.168.1.1:6666

# Redirecting STDIN

< reads from its right argument and writes to its left argument.

To write a file into STDIN we should read /tmp/a_file and write into STDIN i.e 0</tmp/a_file

Note: Internal file descriptor defaults to 0 (STDIN) for <

# Redirecting STDERR

2 is STDERR.

Definitions:

echo_to_stderr is a command that writes "stderr" to STDERR

# STDIN, STDOUT and STDERR explained

Commands have one input (STDIN) and two kinds of outputs, standard output (STDOUT) and standard error (STDERR).

For example:

STDIN

Standard input is used to provide input to a program.
(Here we’re using the read builtin (opens new window) to read a line from STDIN.)

STDOUT

Standard output is generally used for «normal» output from a command. For example, ls lists files, so the files are sent to STDOUT.

STDERR

Standard error is (as the name implies) used for error messages. Because this message is not a list of files, it is sent to STDERR.

STDIN, STDOUT and STDERR are the three standard streams. They are identified to the shell by a number rather than a name:

0 = Standard in
1 = Standard out
2 = Standard error

By default, STDIN is attached to the keyboard, and both STDOUT and STDERR appear in the terminal. However, we can redirect either STDOUT or STDERR to whatever we need. For example, let’s say that you only need the standard out and all error messages printed on standard error should be suppressed. That’s when we use the descriptors 1 and 2.

Redirecting STDERR to /dev/null
Taking the previous example,

In this case, if there is any STDERR, it will be redirected to /dev/null (a special file which ignores anything put into it), so you won’t get any error output on the shell.

# Syntax

• command </path/to/file # Redirect standard input to file
• command >/path/to/file # Redirect standard output to flie
• command file_descriptor>/path/to/file # Redirect output of file_descriptor to file
• command >&file_descriptor # Redirect output to file_descriptor
• command file_descriptor>&another_file_descriptor # Redirect file_descriptor to another_file_descriptor
• command <&file_descriptor # Redirect file_descriptor to standard input
• command &>/path/to/file # Redirect standard output and standard error to file

# Parameters

UNIX console programs have an input file and two output files (input and output streams, as well as devices, are treated as files by the OS.) These are typically the keyboard and screen, respectively, but any or all of them can be redirected to come from — or go to — a file or other program.

STDIN is standard input, and is how the program receives interactive input. STDIN is usually assigned file descriptor 0.

STDOUT is standard output. Whatever is emitted on STDOUT is considered the «result» of the program. STDOUT is usually assigned file descriptor 1.

STDERR is where error messages are displayed. Typically, when running a program from the console, STDERR is output on the screen and is indistinguishable from STDOUT. STDERR is usually assigned file descriptor 2.

The order of redirection is important

Redirects both (STDOUT and STDERR) to the file.

Redirects only STDOUT, because the file descriptor 2 is redirected to the file pointed to by file descriptor 1 (which is not the file file yet when the statement is evaluated).

Each command in a pipeline has its own STDERR (and STDOUT) because each is a new process. This can create surprising results if you expect a redirect to affect the entire pipeline. For example this command (wrapped for legibility):

will print «Python error!» to the console rather than the log file. Instead, attach the error to the command you want to capture: