How is shell redirection implemented?

Shell redirection allows you to redirect a process’ standard streams to paths other than the controlling terminal. This post will focus on explaining how it is implemented by shells (such as bash). If you are unfamiliar with redirection and would like to familiarize yourself with the concept and syntax (as far as bash is concerned), then GNU Bash Reference Manual - Redirections is a good place to start.

The fundamentals

A running shell process maintains a set of open standard file descriptors for stdout (1), stderr(2) and stdin (0) that are tied to the controlling terminal. This can be observed by querying the /proc virtual filesystem. For example, in bash:

1$ ls -l /proc/$$/fd

1Permissions Size User   Group  Date Modified Name
2lrwx------    64 bogdan bogdan 17 Jul 16:22  0 -> /dev/pts/3
3lrwx------    64 bogdan bogdan 17 Jul 16:22  1 -> /dev/pts/3
4lrwx------    64 bogdan bogdan 17 Jul 16:22  2 -> /dev/pts/3
5lrwx------    64 bogdan bogdan 17 Jul 16:22  255 -> /dev/pts/3

Above, $$ is a bash variable that expands to the PID of the running shell process, and 0, 1, 2 are file nodes in the virtual filesystem, all symlinked to the running terminal character device.

If I start another instance of my terminal emulator, and in the shell I issue the following command:

1$ echo "Hello from another terminal!" >/dev/pts/3

I will see the text pop up in my original terminal window. This fact doesn’t have much to do with the shell, or redirection, but it’s still pretty neat to recognize that the terminal is interfaceable with via a node on the filesystem.

Redirection

When you instruct the shell to spawn a new process, whatever process that may be, it does so using the usual fork + execve (or rather clone + execve) combination of syscalls. After the clone, the new process is granted a copy of the file descriptor table of the shell. At that point, before it yields control to the process it was instructed to spawn, or rather before it calls execve, the new process has a chance to redirect any of its file descriptors to a new destination. It would do this using the dup2(2) system call.

dup2’s signature is very simple:

1int dup2(int oldfd, int newfd);

It accepts two file descriptors, oldfd and newfd - and modifies newfd such that it points to the same destination as oldfd. It is thus very easy to see how the shell makes use of this syscall: when it is instructed to redirect one of the standard fds to a particular destination, it opens that destination (using open(2)), and runs dup2(2), passing it the fd of one of the standard streams and the fd of the new, opened destination.

Example

Let’s see it in action. Below is a simple Python script, print.py, that outputs two strings: one to stdout and one to stderr. It is marked as executable and has the appropriate shebang:

1#!/usr/bin/python3
2
3from sys import stderr, stdout
4
5print("To STDOUT, from Python.", file=stdout)
6print("To STDERR, from Python.", file=stderr)

What we’ll do is run bash, instructing it to execute the Python script, and additionally to redirect both stdout and stderr to separate files in the current directory. We will trace all syscalls that happen using strace, and then observe the output:

1$ strace -f bash -c './print.py 1>pyout.txt 2>pyerr.txt'

The -f switch is necessary so that strace also tracks any forked processes that the main process spawns. Remember that the sycalls we are looking for happen in the forked child process. The interesting portion(s) of the output can be seen below:

 1...
 22103644 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f020f1faa10) = 2103645
 3...
 42103645 openat(AT_FDCWD, "pyout.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
 52103645 dup2(3, 1)                      = 1
 62103645 close(3)                        = 0
 72103645 openat(AT_FDCWD, "pyerr.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
 82103645 dup2(3, 2)                      = 2
 92103645 close(3)                        = 0
102103645 execve("./print.py", ["./print.py"], 0x55e27ce84140 /* 41 vars */) = 0
11...
  • First, the parent bash process clones itself, and the child PID is returned.
  • Then, the child process opens the file pyout.txt, receiving file descriptor 3.
  • It calls dup2(3, 1), redirecting its file descriptor 1 (stdout) to point to the same resource as file descriptor 3 (pyout.txt).
  • It closes fd 3.
  • It opens the file pyerr.txt, receving file descriptor 3 again (remember, fd 3 was closed, and now the lowest unused one is returned by open).
  • It calls dup2(3, 2) to redirect file descriptor 2 (stderr) to pyerr.txt
  • Finally, it closes fd 3, and - as both standard file descriptors have now been redirected, it calls execve, running the Python script.
    • When this new Python process writes to “stdout” and “stderr” file descriptors, it will instead be writing to pyout.txt and pyerr.txt respectively.

Of particular interest to me was the fact that the forked process closes the original opened file descriptor once it performs the dup2 operation. See, a per-process “file descriptor” is just a reference to an “open file description”, an entry in a kernel-wide table of open files. So long as at least one per-process file descriptor remains open and associated with an open file description, the latter continues to exist. When the last open file descriptor associated with a certain open file description is closed, the latter is destroyed as well. This excellent article by Viacheslav Biriukov sheds light on and solidifies the difference between the two related concepts.