Working with files, text and pipes

Most of what a Linux administrator does is work with text. Configuration files are text, logs are text, the output of every command is text, and even the list of users is a text file. Linux provides a toolbox of small commands that each do one job with text: show it, search it, sort it, count it, cut it into columns. Their real power appears when they are joined together with pipes, so that the output of one becomes the input of the next. With a few of these tools Jordan can answer questions such as which coffees sold most online this week, which addresses are hammering the shop with requests, or how many orders failed, in a single line typed at the prompt.

This lesson starts with the everyday commands for creating, copying, moving and deleting files, then covers viewing and editing text, wildcards, the three standard streams and redirection, and finally the text-processing tools and pipelines that make the command line so productive.

Creating, copying, moving and deleting

A handful of commands handle files and directories. They are short to type and quick to run, and because they act immediately and silently, they reward care. There is no recycle bin on a Linux server: a file removed with rm is gone, and only a backup can bring it back.

> mkdir -p ~/practice/reports/2026
> cd ~/practice
> touch todo.txt
> cp todo.txt todo-copy.txt
> mv todo-copy.txt reports/
> ls -R
.:
reports  todo.txt

./reports:
2026  todo-copy.txt

./reports/2026:
> rm -r reports
> ls
todo.txt

The command rm -r removes a directory and everything below it without asking. Combined with -f, which forces removal and suppresses errors, it becomes rm -rf, the most feared command on Linux. A stray space can be disastrous: rm -rf / srv/old, with an accidental space after the slash, tells rm to remove the whole tree starting at /. Modern versions of rm refuse to work on / unless given an extra option, but that safeguard does not protect other directories. Good habits are to use -i when deleting by hand, to run ls with the same arguments first to see what will be removed, and to keep backups.

Sam sets Jordan a simple rule for ridge-web: practise in a directory under your home, never experiment with rm in /etc, /srv or /var, and copy a configuration file before editing it, for example cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak, so that you can always go back.

Viewing text

Several commands show the contents of text files, each suited to a different size of file and a different question.

Of these, tail -f is the one administrators reach for in a crisis. When Theo says a customer's order just failed, Jordan can run tail -f /var/log/nginx/access.log in one window, ask Theo to try the order again, and watch the request arrive. In less, typing a slash and a word searches forward, n jumps to the next match, G goes to the end and g back to the start, so even a log of millions of lines is easy to explore.

Editing text: nano and vim

Sooner or later every administrator must edit a configuration file on a server that has no graphical editor. Two text-mode editors are almost always available. GNU nano is simple: you type to insert text, and the commands are listed along the bottom of the screen, with the caret symbol meaning the Ctrl key, so Ctrl+O saves (writes Out) and Ctrl+X exits. Ubuntu opens nano by default when a command such as crontab -e needs an editor.

Vim, an improved version of the classic vi editor from 1976, is on virtually every Unix-like system, including tiny rescue environments where nano is missing. It is a modal editor: in normal mode keys are commands, and only in insert mode does typing insert text. Press i to start inserting, Esc to return to normal mode, then type :w to save, :q to quit, :wq to do both, or :q! to quit without saving. Vim is famously confusing at first, but those few keys are enough to survive, and many administrators come to prefer it for its speed.

To edit a file that belongs to root, put sudo in front: sudo nano /etc/hosts. A safer habit for some files is to use a dedicated command that checks your changes before saving them, such as visudo for the sudo rules, which lesson 4 introduces. A syntax error in some configuration files can lock everyone out.

Wildcards

When a command needs several file names, the shell can generate them from a pattern. These patterns are called wildcards or globs, and the shell expands them before the command runs, so ls *.csv receives the list of matching names, not the pattern itself. That is why the same patterns work with every command.

Python's fnmatch module uses the same rules as the shell, so you can test a pattern against a list of names safely before using it with a command like rm.

from fnmatch import fnmatch

names = ["roastlog-2026-11-01.sql.gz", "roastlog-2026-11-09.sql.gz", "roastlog-2026-10-31.sql.gz",
         "orders.csv", "roast_log.csv", "README"]
for pattern in ["*.csv", "roastlog-2026-11-0[1-7].sql.gz", "roast?log.csv", "*"]:
    hits = [n for n in names if fnmatch(n, pattern)]
    print(f"{pattern:32} {hits}")

*.csv                            ['orders.csv', 'roast_log.csv']
roastlog-2026-11-0[1-7].sql.gz   ['roastlog-2026-11-01.sql.gz']
roast?log.csv                    ['roast_log.csv']
*                                ['roastlog-2026-11-01.sql.gz', 'roastlog-2026-11-09.sql.gz', 'roastlog-2026-10-31.sql.gz', 'orders.csv', 'roast_log.csv', 'README']

Standard streams and redirection

Every program started from the shell is given three channels, called standard streams. Standard input, number 0, is where it reads input, normally the keyboard. Standard output, number 1, is where it writes its results, normally the screen. Standard error, number 2, is where it writes error messages, also normally the screen. Keeping errors separate from results means that when you save a command's output to a file, error messages still appear where you can see them and do not get mixed into the data.

Redirection changes where these streams go. A greater-than sign after a command sends standard output to a file, replacing whatever the file held. Two greater-than signs append to the file instead. Putting the number 2 in front redirects standard error. A less-than sign feeds a file into standard input. And the special file /dev/null swallows anything written to it, which is how you discard output you do not want.

> ls /srv/ridgeline /nonexistent > listing.txt
ls: cannot access '/nonexistent': No such file or directory
> cat listing.txt
/srv/ridgeline:
roastlog
shop
> date >> checks.log
> df -h /srv >> checks.log
> find /etc -name "*.conf" 2> /dev/null | wc -l
143

The greater-than sign is the most dangerous character in this section. Typing > when you meant >> erases the file's previous contents without any warning, and running a command such as sort roast.csv > roast.csv empties the file before sort even reads it, because the shell opens the output file first. Write to a new file and then rename it instead.

Pipes: joining commands together

A pipe, written with the vertical bar |, connects the standard output of one command to the standard input of the next. The commands run at the same time, with data flowing between them, and nothing is written to disk in between. Pipes let you build a custom tool on the spot out of simple parts. The idea came from Doug McIlroy at Bell Labs and was added to Unix in 1973, and it remains the heart of command-line work.

> grep "POST /checkout" /var/log/nginx/access.log | wc -l
312
> grep " 500 " /var/log/nginx/access.log | tail -n 3
> cut -d' ' -f1 /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -n 5
   1840 203.0.113.88
    411 198.51.100.17
    207 192.0.2.44
    190 192.0.2.61
    122 198.51.100.200
> grep -c "sunrise-blend" /var/log/nginx/access.log

Read the third pipeline from left to right. cut -d' ' -f1 splits each line of the web server log at spaces and keeps the first field, the client's address. sort puts identical addresses next to each other. uniq -c collapses each run of identical lines into one, with a count in front. sort -rn sorts those counts as numbers, largest first. head -n 5 keeps the top five. The answer, that one address made 1,840 requests, is a clue worth following up, and lesson 9 shows what Jordan found.

The text-processing toolbox

The tools used in pipelines are worth knowing individually. Each has many options, which man describes, but a few of each cover most needs.

The grep command is the one you will use most. Its name comes from a command in an early Unix editor, g/re/p, meaning globally search for a regular expression and print. A regular expression is a pattern language for text: a dot matches any character, a caret anchors the start of a line, square brackets match a set and an asterisk repeats what comes before. You can go a long way with plain words, though, and grep -i error /var/log/syslog is often all you need.

Here is the busiest-address pipeline written in Python, using the Counter class from the collections module. Seeing the same steps in both forms makes the shell version easier to remember, and the Python version is easier to extend when the question gets more complicated.

from collections import Counter

log = """203.0.113.88 - - [03/Nov/2026:08:01:12 +0000] "GET /shop HTTP/1.1" 200 5123
198.51.100.17 - - [03/Nov/2026:08:01:15 +0000] "GET /shop/sunrise-blend HTTP/1.1" 200 4210
203.0.113.88 - - [03/Nov/2026:08:01:16 +0000] "GET /wp-login.php HTTP/1.1" 404 162
192.0.2.44 - - [03/Nov/2026:08:02:40 +0000] "POST /checkout HTTP/1.1" 200 811
203.0.113.88 - - [03/Nov/2026:08:02:41 +0000] "GET /.env HTTP/1.1" 404 162
198.51.100.17 - - [03/Nov/2026:08:03:02 +0000] "POST /checkout HTTP/1.1" 500 97"""

lines = log.splitlines()
addresses = Counter(line.split()[0] for line in lines)        # cut | sort | uniq -c
for addr, count in addresses.most_common(2):                  # sort -rn | head
    print(count, addr)
statuses = Counter(line.split()[8] for line in lines)
print("By status:", dict(sorted(statuses.items())))
print("Checkouts:", sum("POST /checkout" in line for line in lines))   # grep | wc -l

3 203.0.113.88
2 198.51.100.17
By status: {'200': 3, '404': 2, '500': 1}
Checkouts: 2

def grep(lines, word, ignore_case=False, invert=False):
    return lines


log = ['GET /shop 200', 'POST /checkout 500', 'GET /Checkout 200']
print(grep(log, 'checkout'))

def top_addresses(log_text, n):
    return []


sample = '1.1.1.1 GET /\n2.2.2.2 GET /\n1.1.1.1 GET /shop'
print(top_addresses(sample, 2))

def wc(text):
    return (0, 0, 0)


print(wc('Sunrise Blend\nEthiopia Guji\n'))