Unix pipe error

So I wanted to contribute an answer like lesmana’s, but I think mine is perhaps a little simpler and slightly more advantageous pure-Bourne-shell solution:

# You want to pipe command1 through command2:
exec 4>&1
exitstatus=`{ { command1; printf $? 1>&3; } | command2 1>&4; } 3>&1`
# $exitstatus now has command1's exit status.

I think this is best explained from the inside out — command1 will execute and print its regular output on stdout (file descriptor 1), then once it’s done, printf will execute and print icommand1’s exit code on its stdout, but that stdout is redirected to file descriptor 3.

While command1 is running, its stdout is being piped to command2 (printf’s output never makes it to command2 because we send it to file descriptor 3 instead of 1, which is what the pipe reads). Then we redirect command2’s output to file descriptor 4, so that it also stays out of file descriptor 1 — because we want file descriptor 1 free for a little bit later, because we will bring the printf output on file descriptor 3 back down into file descriptor 1 — because that’s what the command substitution (the backticks), will capture and that’s what will get placed into the variable.

The final bit of magic is that first exec 4>&1 we did as a separate command — it opens file descriptor 4 as a copy of the external shell’s stdout. Command substitution will capture whatever is written on standard out from the perspective of the commands inside it — but since command2’s output is going to file descriptor 4 as far as the command substitution is concerned, the command substitution doesn’t capture it — however once it gets «out» of the command substitution it is effectively still going to the script’s overall file descriptor 1.

(The exec 4>&1 has to be a separate command because many common shells don’t like it when you try to write to a file descriptor inside a command substitution, that is opened in the «external» command that is using the substitution. So this is the simplest portable way to do it.)

You can look at it in a less technical and more playful way, as if the outputs of the commands are leapfrogging each other: command1 pipes to command2, then the printf’s output jumps over command 2 so that command2 doesn’t catch it, and then command 2’s output jumps over and out of the command substitution just as printf lands just in time to get captured by the substitution so that it ends up in the variable, and command2’s output goes on its merry way being written to the standard output, just as in a normal pipe.

Also, as I understand it, $? will still contain the return code of the second command in the pipe, because variable assignments, command substitutions, and compound commands are all effectively transparent to the return code of the command inside them, so the return status of command2 should get propagated out — this, and not having to define an additional function, is why I think this might be a somewhat better solution than the one proposed by lesmana.

Per the caveats lesmana mentions, it’s possible that command1 will at some point end up using file descriptors 3 or 4, so to be more robust, you would do:

exec 4>&1
exitstatus=`{ { command1 3>&-; printf $? 1>&3; } 4>&- | command2 1>&4; } 3>&1`
exec 4>&-

Note that I use compound commands in my example, but subshells (using ( ) instead of { } will also work, though may perhaps be less efficient.)

Commands inherit file descriptors from the process that launches them, so the entire second line will inherit file descriptor four, and the compound command followed by 3>&1 will inherit the file descriptor three. So the 4>&- makes sure that the inner compound command will not inherit file descriptor four, and the 3>&- will not inherit file descriptor three, so command1 gets a ‘cleaner’, more standard environment. You could also move the inner 4>&- next to the 3>&-, but I figure why not just limit its scope as much as possible.

I’m not sure how often things use file descriptor three and four directly — I think most of the time programs use syscalls that return not-used-at-the-moment file descriptors, but sometimes code writes to file descriptor 3 directly, I guess (I could imagine a program checking a file descriptor to see if it’s open, and using it if it is, or behaving differently accordingly if it’s not). So the latter is probably best to keep in mind and use for general-purpose cases.

Seeing «Broken pipe» in this situation is rare, but normal.

When you run type rvm | head -1, bash executes type rvm in one process, head -1 in another.¹ The stdout of type is connected to the «write» end of a pipe, the stdin of head to the «read» end. Both processes run at the same time.

The head -1 process reads data from stdin (usually in chunks of 8 kB), prints out a single line (according to the -1 option), and exits, causing the «read» end of the pipe to be closed. Since the rvm function is quite long (around 11 kB after being parsed and reconstructed by bash), this means that head exits while type still has a few kB of data to write out.

At this point, since type is trying to write to a pipe whose other end has been closed – a broken pipe – the write() function it caled will return an EPIPE error, translated as «Broken pipe». In addition to this error, the kernel also sends the SIGPIPE signal to type, which by default kills the process immediately.

(The signal is very useful in interactive shells, since most users do not want the first process to keep running and trying to write to nowhere. Meanwhile, non-interactive services ignore SIGPIPE – it would not be good for a long-running daemon to die on such a simple error – so they find the error code very useful.)

However, signal delivery is not 100% immediate, and there may be cases where write() returns EPIPE and the process continues to run for a short while before receiving the signal. In this case, type gets enough time to notice the failed write, translate the error code and even print an error message to stderr before being killed by SIGPIPE. (The error message says «-bash: type:» since type is a built-in command of bash itself.)

This seems to be more common on multi-CPU systems, since the type process and the kernel’s signal delivery code can run on different cores, literally at the same time.

It would be possible to remove this message by patching the type builtin (in bash’s source code) to immediately exit when it receives an EPIPE from the write() function.

However, it’s nothing to be concerned about, and it is not related to your rvm installation in any way.

Updated on 5/5/2022 – Have you ever encountered a situation in which you were unable to download any package on your Linux machine ? Or you might have probably seen an error like package not installed? This kind of error can easily be fixed with a command like “sudo apt install –f”. On rare occasions, you may have experienced a broken pipe error.

A pipe in Linux / Unix connects two processes, one of them has read-end of the file and the other one has the write-end of the file. When a process writes to a Pipe, it gets stored in a buffer and gets retrieved by the other process. Broken pipe occurs when a process prematurely exits from either end and the other process has not yet closed the pipe.

Example use case:

A user has just recently reinstalled RVM (Ruby Version Manager) after he performed a fresh install of Ubuntu.

He then opened up the terminal and issued the command:

type rvm | head -1

This issued the following error:

rvm is a function -bash: type: write error: Broken pipe

What happened here is that when the user runs the command type rvm | head -1, bash has actually executed type rvm in one process and  head -1 in another process. The stdout of the type part is connected to the “write” end of a pipe whereas the stdin of the head part is hooked up to the “read” end. Note that the two processes have run concurrently ( at the same time ).

The head -1 process has carried out a read operation of data from stdin , then prints out a single line (as dictated by the -1 option) before exiting, causing  therefore the “read” end of the pipe to be closed. Since the rvm function has quite a long data stream (about 11 kB after having been bash parsed and reconstructed), which means that head exits yet at the same time type still has some data to write out (few KB).

Since type is trying to carry out a  write operation to a pipe whose other end has therefore been closed – a brokenpipe routine or the write() function that it invoked,  will return an EPIPE error which is known as “Broken pipe”.

Inspecting the Command

In most cases, this might not be the case but the first step you should check is whether the command issued was right or not. You should reissue the command and check whether it gets executed or not. You can also try issuing commands like “sudo apt update” and “sudo apt install –f” as these commands are not destructive in nature. If your problem still persists, try rebooting the machine and see whether the problem was resolved or not.

Fixing a Problem with File System

When you have issued the commands mentioned earlier multiple times and you still get the error, check whether the error reads something like “read-only file system” in the terminal output. This may be caused when your boot partition gets mounted as read-only for some reason. The problem could be caused by some faulty software installation when the system decides that it is not safe to write to the drive.

Read: How to create a Sudo user on Ubuntu

The other cause might be when you try to install something from apt and the installer needs to access some resource in read mode, but cannot perform the read operation properly. It may throw an error like “sudo: cannot mount”. This error occurs because most of the ‘entities’ in Linux are files and in order to read a resource, Linux would need to open that file and read it. If however another process is currently using that resource, then it may not be possible to read the file. Also, when the reading process exits abruptly and does not close the file, it may corrupt the file until the next boot.

If you still cannot access the files even after rebooting, then the problem could be bigger than anticipated. You may have a broken file system. To resolve this issue, you may need a stable Linux environment in order to work on the broken system. The best way to do this is to boot from a Live Linux USB drive and work from it.

This is the right moment to backup all your data. Although the following steps are safe, you should make sure to store your data on a secure device.

Once you boot into a Live USB drive you should start to check for the partitions with a corrupt file system. To do so, issue the following command:

sudo fsck.ext4 -fv /dev/sdaX”

Note that here X stands for the partition that you are trying to scan. Note that this command is for partitions of type ext4. If you have a partition of type ext3 or ext2 you will need to replace the command with “fsck.ext3” and “fsck.ext2” respectively. This will scan your drive and print the output on the terminal (note the -v flag). Alternatively, you can specify a -c flag to surface-scan the drive; it will look for bad sectors on the drive.

Once you have done this, your partition should hopefully been fixed. Now Boot into your machine and issue the command:

sudo mount -o rw,remount /

This will restore the read/write permission on your drive and will therefore solve the broken pipe issue.

Read: How to manage permissions in Linux : guide for beginners

Afterwords

You have just seen one solution to resolve the broken pipe issue, but a broken pipe is not a problem, it could be a symptom of a larger problem more often than not. You can have a broken pipe whenever you are trying to read a stream like resource that gets closed prematurely..