Shell glue

The shell has the power to glue commands together to make the impossible possible! Let's use some gluing magic!

Piping

We will start with pipes. In the last task of the last day, you did see the usage of the symbol | between two separate commands.

By entering cowsay "Hello" | lolcat, the output of the first command cowsay is sent to the second command lolcat.

lolcat takes the input, colors it and outputs it again!

Many Linux commands support handling input of another command.

You might have seen in the manual of wc in day 1 that the file as an argument is only optional. How could you use wc without arguments?

You might have guessed it now, make some wc pipes. OK, I admit that the naming is not the best 🚾😂

Let's get some data to work with. To do so, we will use the command curl which fetches content from the internet.

Let's count the number of lines of the HTML file of the homepage of this book:

$ curl -s https://dev-tools.mo8it.com | wc -l
254

The option -s tells curl to be silent and not show progress information.

You can see that wc did count the number of lines. We did just combine two completely different tools using a pipe!

How about counting the number of lines that contain the word "Linux" on the homepage? To do so, we will add a new pipe inbetween!

grep is a command that searches for matches of a specified pattern. Each line with a match is printed in a new line.

To demonstrate grep, here is one usage example:

$ curl --help | grep "silent"
 -s, --silent    Silent mode

We did just filter the output of the help message of a command. This is one way to search quickly for command options!

Back to the main example:

$ curl -s https://dev-tools.mo8it.com | grep "Linux" | wc -l
6

You can see that you can use multiple pipes. This allows for almost infinite combinations!

Being able to combine commands is the reason why many commands are simple. They do one thing and do it well! To do more, combine them!

This is much more flexible and powerful than a program that tries to do many things at once.

Input, output

Before going any further, we need to understand an important concept in Linux.

A command accepts input and generates two types of output. The input is called standard input (stdin). The output is split to standard output (stdout) and standard error (stderr). They actually have numbers that will be relevant later:

0: stdin
1: stdout
2: stderr

Normal output is sent to the standard output. Errors (and sometimes output that is not important) are sent to the standard error.

Redirections

You can redirect the standard output or the standard error of a command to a file!

If you just run curl -s https://dev-tools.mo8it.com, you will see the HTML file printed to the terminal. Let's redirect the output to an HTML file on the disk:

$ curl -s https://dev-tools.mo8it.com >dev-tools.html

Now, view the content of the new file dev-tools.html. You will see the same output from the terminal without redirection.

Now, try this command:

$ curl https://non-existent-site.mo8it.com >test.html
(…)
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

$ cat test.html

You will see that the file is empty since curl did not find a page to show as normal output. The error message was displayed using stderr.

If you are using this command in a script, then it might be wise to redirect the error to a log file:

$ curl https://non-existent-site.mo8it.com 2>curl.log

$ cat curl.log
curl: (60) SSL certificate problem: self-signed certificate
(…)

You can see that the error is not shown anymore after running the curl command. It was redirected to the file curl.log.

Did you notice the number 2 before the redirection symbol 2>?

The last section did mention that the number of the standard error is 2. Therefore, 2 has to be specified to redirect it.

If you don't specify a number, then it is equivalent to 1 which stands for the standard output. This means that > is equivalent to 1>.

What if a command produces output using stdout and stderr? Take a look at the following example:

$ touch some_file.txt

$ # Produces output to stdout and stderr.
$ ls some_file.txt does_not_exist.txt
ls: cannot access 'does_not_exist.txt': No such file or directory
some_file.txt

$ # Redirect only stdout to a file. stderr is shown.
$ ls some_file.txt does_not_exist.txt >stdout.txt
ls: cannot access 'does_not_exist.txt': No such file or directory

$ cat stdout.txt
some_file.txt

$ # Redirect only stderr to a file. stdout is shown.
$ ls some_file.txt does_not_exist.txt 2>stderr.txt
some_file.txt

$ cat stderr.txt
ls: cannot access 'does_not_exist.txt': No such file or directory

$ # Redirect both stdout and stderr to different files.
$ ls some_file.txt does_not_exist.txt >stdout.txt 2>stderr.txt

$ cat stdout.txt
some_file.txt

$ cat stderr.txt
ls: cannot access 'does_not_exist.txt': No such file or directory

$ # Redirect stdout and stderr to the same file with `&>`.
$ ls some_file.txt does_not_exist.txt &>mixed.txt

$ cat mixed.txt
ls: cannot access 'does_not_exist.txt': No such file or directory
some_file.txt

Appending

So far, we did redirect output using the operators >, 1>, 2> and &>. But these operators overwrite the files they are redirected to if they already exist. If you want to append to a file instead, use the operators above but with double >, for example &>>.

Discarding

There is a special file that you see some command redirect to: /dev/null.

This file is like a black hole. Everything redirected to that file is discarded.

For example, if you don't care about the errors that some command throughs, then you can redirected its stderr to /dev/null using 2>/dev/null

More details

We discussed the most important cases for redirections. But there are some less important details like the following:

command &>filename is equivalent to command >filename 2>&1, but not to command 2>&1 >filename because the order matters.
0<filename command or command 0<filename can be used to redirect a file to the stdin of a command.
A pipe command1 | command2 redirects only the stdout of command1 to the stdin of command2. But if you want to redirect stderr too, then you have to use command1 2>&1 | command2. To only redirect stderr, you have to use command 2>&1 >/dev/null | command2 (not command >/dev/null 2>&1 | command2 since the order matters).

You might have noticed how it can get complicated. Therefore, refer to the "Redirections" section in the Bash reference manual for more details.

Yes, the reference is about Bash, but Fish has the same behavior here.

Dev Tools