🔗 Lesson 11: Pipes, Redirection and Your First Script

Connect small commands into powerful chains, send output into files, and turn what you typed into a script you can run again

▶

Small tools, joined together

Every command you have learned so far does one small job: ls lists, grep searches, wc counts, sort sorts. The real power of the command line comes from joining them, so that the output of one becomes the input of the next. This lesson shows you how, and ends with the step that turns you from someone who types commands into someone who writes scripts: saving the commands in a file so the computer can run them again whenever you like.

Everything below was run for real on Ubuntu 24.04. File names, counts and sizes on your machine will differ.

The pipe: |

The | character (the pipe, usually on the key above Enter, with Shift) takes everything the left command prints and hands it to the right command instead of the screen.

ubuntu@practice:~$ ls /usr/bin | head -5
[
addpart
apt
apt-cache
apt-cdrom
ubuntu@practice:~$ ls /usr/bin | wc -l
301
ubuntu@practice:~$ cat /etc/passwd | wc -l
19
ubuntu@practice:~$ du -h --max-depth=1 /usr | sort -h
4.0K	/usr/games
4.0K	/usr/include
4.0K	/usr/lib64
4.0K	/usr/src
36K	/usr/libexec
40K	/usr/local
5.3M	/usr/sbin
6.0M	/usr/share
19M	/usr/bin
49M	/usr/lib
79M	/usr
ubuntu@practice:~$ du -h --max-depth=1 /usr | sort -rh | head -3
79M	/usr
49M	/usr/lib
19M	/usr/bin

• ls /usr/bin | head -5 lists a folder, but head -5 only keeps the first five lines.
• ls /usr/bin | wc -l counts the lines of that listing: wc -l counts lines, so this machine has 301 items in /usr/bin. This is the standard trick for "how many?".
• cat /etc/passwd | wc -l counts the lines of the user list from lesson 5: 19 accounts.
• du -h --max-depth=1 /usr | sort -h is the promise from lesson 9. du prints folder sizes in no particular order, and sort -h sorts them by human size (4.0K before 5.3M before 49M), so the biggest is at the bottom. Adding -r reverses the order, and head -3 keeps only the top three: /usr itself at 79 MB, then /usr/lib and /usr/bin.

You can chain as many as you like, each command reading from the one before. Nothing is changed on disk. A pipe only moves text from one program to the next. (This is also how you find one process among hundreds: ps aux | grep NAME, the pipe from lesson 7's promise; from the documentation, not run here.)

Redirection: > >> 2> <

Pipes join commands to each other. Redirection joins a command to a file.

ubuntu@practice:~$ echo "first line" > out.txt
ubuntu@practice:~$ echo "second line" >> out.txt
ubuntu@practice:~$ cat out.txt
first line
second line
ubuntu@practice:~$ echo "oops" > out.txt
ubuntu@practice:~$ cat out.txt
oops
ubuntu@practice:~$ ls /nonexistent
ls: cannot access '/nonexistent': No such file or directory
ubuntu@practice:~$ ls /nonexistent 2> errors.txt
ubuntu@practice:~$ cat errors.txt
ls: cannot access '/nonexistent': No such file or directory
ubuntu@practice:~$ ls /usr/share/doc /nonexistent > list.txt 2> errors.txt
ubuntu@practice:~$ wc -l list.txt
96 list.txt
ubuntu@practice:~$ cat errors.txt
ls: cannot access '/nonexistent': No such file or directory
ubuntu@practice:~$ ls /nonexistent 2> /dev/null
ubuntu@practice:~$ printf 'pear\napple\nfig\n' > fruit.txt
ubuntu@practice:~$ sort < fruit.txt
apple
fig
pear
ubuntu@practice:~$ sort fruit.txt > sorted.txt
ubuntu@practice:~$ cat sorted.txt
apple
fig
pear
ubuntu@practice:~$ wc -l < fruit.txt
3

• > sends the output into a file instead of the screen, and replaces whatever was in the file. echo "first line" > out.txt created it.
• >> appends: echo "second line" >> out.txt adds a line at the end, and cat out.txt shows both.
• Careful: a single > on an existing file wipes it out without asking. echo "oops" > out.txt left only oops: the two earlier lines are gone. When in doubt, use >>, or look first.
• Every program has two output streams: normal output and error messages. ls /nonexistent printed an error on the screen. 2> (the 2 is the number of the error stream) sends only the errors to a file: the screen stayed quiet, and cat errors.txt shows the message was saved.
• ls /usr/share/doc /nonexistent > list.txt 2> errors.txt splits the two: the 96 lines of listing went to list.txt and the one error went to errors.txt.
• 2> /dev/null throws errors away. /dev/null is a special file that swallows everything, a black hole. Use it for noise you truly do not care about, not to hide real problems.
• < goes the other way: it feeds a file into a command. sort < fruit.txt sorted the file's three lines, and wc -l < fruit.txt counted them. (Many commands, like sort fruit.txt > sorted.txt, also take the file name directly.)

A real pipeline: reading a log

Servers write logs, files recording what happened (lesson 8). They are the best practice for pipelines. We made a tiny pretend web log with seven requests, one per line: the visitor's address, the method, the page and the result (200 means found, 404 means not found).

ubuntu@practice:~$ echo "10.0.0.5 GET /index.html 200" > access.log
ubuntu@practice:~$ echo "10.0.0.9 GET /about.html 200" >> access.log
ubuntu@practice:~$ echo "10.0.0.5 GET /missing.html 404" >> access.log
ubuntu@practice:~$ echo "10.0.0.7 GET /index.html 200" >> access.log
ubuntu@practice:~$ echo "10.0.0.5 GET /index.html 200" >> access.log
ubuntu@practice:~$ echo "10.0.0.9 GET /old.html 404" >> access.log
ubuntu@practice:~$ echo "10.0.0.5 GET /contact.html 200" >> access.log
ubuntu@practice:~$ cat access.log
10.0.0.5 GET /index.html 200
10.0.0.9 GET /about.html 200
10.0.0.5 GET /missing.html 404
10.0.0.7 GET /index.html 200
10.0.0.5 GET /index.html 200
10.0.0.9 GET /old.html 404
10.0.0.5 GET /contact.html 200
ubuntu@practice:~$ grep -c 404 access.log
2
ubuntu@practice:~$ grep 404 access.log
10.0.0.5 GET /missing.html 404
10.0.0.9 GET /old.html 404
ubuntu@practice:~$ cut -d' ' -f1 access.log
10.0.0.5
10.0.0.9
10.0.0.5
10.0.0.7
10.0.0.5
10.0.0.9
10.0.0.5
ubuntu@practice:~$ cut -d' ' -f1 access.log | sort | uniq -c
      4 10.0.0.5
      1 10.0.0.7
      2 10.0.0.9
ubuntu@practice:~$ cut -d' ' -f1 access.log | sort | uniq -c | sort -rn
      4 10.0.0.5
      2 10.0.0.9
      1 10.0.0.7
ubuntu@practice:~$ grep 404 access.log | cut -d' ' -f3 | tee missing.txt | wc -l
2
ubuntu@practice:~$ cat missing.txt
/missing.html
/old.html

Now we ask it questions, building up one step at a time:

• grep -c 404 access.log counts the lines with 404: two. grep 404 access.log shows them.
• cut -d' ' -f1 access.log cuts each line into pieces at every space (-d' ' is the delimiter) and keeps field 1 (-f1): just the address.
• Adding | sort | uniq -c counts how often each address appears: uniq collapses neighbouring identical lines, which is why you must sort first, and -c prints the count. The visitor 10.0.0.5 made 4 requests.
• Adding | sort -rn sorts that by number, biggest first (-n numeric, -r reverse). Now you know who the busiest visitor is.
• The last command, grep 404 access.log | cut -d' ' -f3 | tee missing.txt | wc -l, finds the failed requests, keeps the page names, and passes them through **tee, which works like a T-junction in a pipe: it **saves a copy to a file and also passes the text on. The wc -l counted 2, and missing.txt holds the two missing pages.

That one idea, **start with a file, then keep adding | something and look at the result each time**, is how administrators answer almost every question about logs and data. On a real server the log would have thousands of lines, and the very same pipeline would work unchanged.

Your first script

A shell script is just a text file of commands, run from top to bottom. You already know how to create a file with nano (lesson 10). Open nano hello.sh and type:

  GNU nano 7.2                        hello.sh *
#!/bin/bash
# My first script
name="Linux learner"
echo "Hello, $name!"
echo "Today is $(date +%A)."

^G Help      ^O Write Out ^W Where Is  ^K Cut       ^T Execute   ^C Location
^X Exit      ^R Read File ^\ Replace   ^U Paste     ^J Justify   ^/ Go To Line

• The first line, #!/bin/bash, is called the shebang. It tells the system which program should run the file: bash.
• Lines starting with # are comments, notes for humans that the computer ignores.
• name="Linux learner" creates a variable, a named box holding text. There are no spaces around the =. You read it back with a dollar sign: $name.
• $(date +%A) runs a command and puts its output right there in the text: the day of the week.

We saved the file with Ctrl+O, Enter and Ctrl+X. Then:

ubuntu@practice:~$ cat hello.sh
#!/bin/bash
# My first script
name="Linux learner"
echo "Hello, $name!"
echo "Today is $(date +%A)."
ubuntu@practice:~$ bash hello.sh
Hello, Linux learner!
Today is Monday.
ubuntu@practice:~$ ./hello.sh
bash: ./hello.sh: Permission denied
ubuntu@practice:~$ ls -l hello.sh
-rw-r--r-- 1 ubuntu ubuntu 101 Oct  5 16:44 hello.sh
ubuntu@practice:~$ chmod +x hello.sh
ubuntu@practice:~$ ls -l hello.sh
-rwxr-xr-x 1 ubuntu ubuntu 101 Oct  5 16:44 hello.sh
ubuntu@practice:~$ ./hello.sh
Hello, Linux learner!
Today is Monday.
ubuntu@practice:~$

• bash hello.sh runs the file by handing it to bash. It works at once.
• ./hello.sh tries to run the file directly (./ means "in this folder", lesson 3), and fails with Permission denied. A new file is not executable: look at ls -l, which shows -rw-r--r--, with no x anywhere. You met this in lesson 5.
• chmod +x hello.sh adds the x (execute) permission, and ls -l now shows -rwxr-xr-x. After that, ./hello.sh runs like a program. Because of the shebang, it is bash that runs it.

Making decisions: arguments, if, and exit codes

Useful scripts react to their input. This one takes a folder name and counts what is inside it:

  GNU nano 7.2                        count.sh *
#!/bin/bash
# Count the items in a folder
folder="$1"
if [ -z "$folder" ]; then
    echo "Usage: $0 FOLDER"
    exit 1
fi
if [ ! -d "$folder" ]; then
    echo "$folder is not a folder"
    exit 2
fi
count=$(ls "$folder" | wc -l)
echo "$folder contains $count items"

^G Help      ^O Write Out ^W Where Is  ^K Cut       ^T Execute   ^C Location
^X Exit      ^R Read File ^\ Replace   ^U Paste     ^J Justify   ^/ Go To Line

• $1 is the first argument: the word typed after the script name. ($2 is the second, and $0 is the script's own name.)
• if [ -z "$folder" ]; then ... fi is a decision. -z is true when the text is empty, so this part says: "if no folder was given, show how to use the script and exit with code 1". The fi ("if" backwards) closes the block. Keep the spaces inside the [ ] brackets: they are required.
• [ ! -d "$folder" ] means "if it is not (!) a directory". Then we stop with a different exit code, 2.
• The quotes around "$folder" protect names containing spaces. Always put them there.
• If both checks pass, count=$(ls "$folder" | wc -l) stores the result of a pipeline in a variable, and echo reports it.

Run it, with echo $? after each run to show the exit code from lesson 4 (0 is success, anything else is a failure):

ubuntu@practice:~$ chmod +x count.sh
ubuntu@practice:~$ ./count.sh
Usage: ./count.sh FOLDER
ubuntu@practice:~$ echo $?
1
ubuntu@practice:~$ ./count.sh /usr/share
/usr/share contains 42 items
ubuntu@practice:~$ echo $?
0
ubuntu@practice:~$ ./count.sh /nonexistent
/nonexistent is not a folder
ubuntu@practice:~$ echo $?
2
ubuntu@practice:~$ ./count.sh /usr/share && echo "That worked"
/usr/share contains 42 items
That worked
ubuntu@practice:~$ ./count.sh /nonexistent || echo "That failed"
/nonexistent is not a folder
That failed
ubuntu@practice:~$ for f in *.txt; do echo "$f has $(wc -l < "$f") lines"; done
errors.txt has 1 lines
fruit.txt has 3 lines
list.txt has 96 lines
missing.txt has 2 lines
out.txt has 1 lines
sorted.txt has 3 lines
ubuntu@practice:~$

With no argument, the script printed its usage message and exited with 1. With a real folder it printed the count and 0. With a missing folder it complained and exited with 2. You choose the codes, and other programs (or other scripts) can read them.

The last lines show the two-way shortcuts: && means "and then, but only if the first command succeeded", and || means "or else, only if it failed". Exactly one of the two messages is printed. Finally, a loop: for f in *.txt; do ... done repeated the echo once for every .txt file in the folder, with $f holding each name in turn. Loops are how one short script handles ten files or ten thousand.

Where to go from here

You have now reached the end of Linux 101. You can find your way around, handle files and permissions, install software, manage processes and services, read logs, check your machine's health, edit files, and chain commands into scripts. The rest of the site is written for someone with exactly these skills:

• Shell scripting goes much deeper into what you just started.
• Command Line Tips has dozens of guides on individual tools.
• SysAdmin has runbooks and how-tos for real servers.

The cheat sheet

• A | B send the output of A into B. Useful partners: head, tail, wc -l, sort, uniq -c, grep, cut -d' ' -f1.
• > write to a file (replaces!). >> append. 2> errors only. 2> /dev/null discard errors. < read input from a file.
• tee FILE save a copy and keep the pipe flowing.
• Pipeline for counting: ... | sort | uniq -c | sort -rn.
• Script: first line #!/bin/bash, then chmod +x NAME.sh, then ./NAME.sh.
• name="text" and $name. $(command) inserts output. $1 the first argument.
• if [ ... ]; then ... fi, exit N, echo $? for the exit code, && after success, || after failure.
• for f in *.txt; do ...; done repeat for each file.

← Lesson 10 · ↑ All lessons · Lesson 12: The Red Hat Family: Meet Rocky Linux →