Parsing logs and text

Every server writes logs: lines of text recording what happened, when and to whom. The web server behind Ridgeline's online shop writes a line for every page request, the operating system records logins and service restarts, and the shop software notes each order. Logs are how an administrator finds out what went wrong at 3 a.m., whether someone is trying to guess passwords, and which pages customers actually use. The trouble is volume: a busy day produces thousands of lines. This lesson shows how to turn log text into structured records with Python, using string methods for simple formats and regular expressions for the rest, and then how to count, filter and summarize the results.

All the log lines in this lesson are invented samples in the real formats. IP addresses come from the ranges reserved for documentation, such as 203.0.113.0 to 203.0.113.255, so none of them belong to a real computer.

What a log line holds

Most web servers write requests in a standard layout called the combined log format. Each line holds the client's IP address, two fields that are usually a dash, the date and time in square brackets, the request line in quotation marks (method, path and protocol), the status code, the size of the response in bytes, and, in quotation marks, the page the visitor came from and the browser's description of itself. Once you know where each piece sits, you can pull out exactly the fields a question needs.

Status codes are the most useful single field. Codes in the 200s mean success; 301 and 302 are redirects; 304 means the browser's cached copy is still good. The 400s are client errors: 401 means login required, 403 forbidden, and 404 not found, often a broken link. The 500s are server errors, the ones that mean the shop itself has a problem. A sudden rise in 500s, or a stream of 401s from one address, is exactly what Jordan's morning check is looking for.

Simple formats: split and partition

When fields are separated by a fixed character, string methods are the quickest and clearest tool. Many application logs write a timestamp, a level and a message separated by spaces, and the shop software's order log uses vertical bars. split with a maximum number of splits keeps the message whole even when it contains spaces, and partition splits once at a separator and always returns three parts, which avoids errors on lines that lack the separator.

app_log = """2026-11-03 08:01:12 INFO order 10512 placed, 2 items
2026-11-03 08:04:55 ERROR payment gateway timeout for order 10513
2026-11-03 08:05:30 INFO order 10513 placed, 1 item
2026-11-03 08:09:02 ERROR stock lookup failed: RC-104
"""
for line in app_log.splitlines():
    day, time, level, message = line.split(" ", 3)   # at most 3 splits: message stays whole
    if level == "ERROR":
        print(time, "|", message)

order = "10512|2026-11-03|RC-101|2|37.00"
number, date, sku, qty, total = order.split("|")
print(number, sku, int(qty) * 1, float(total))
key, sep, value = "retries=3".partition("=")
print(key, value, repr(sep))

08:04:55 | payment gateway timeout for order 10513
08:09:02 | stock lookup failed: RC-104
10512 RC-101 2 37.0
retries 3 '='

The combined web format is harder for split alone, because the request and the browser description contain spaces and sit inside quotation marks. You can still split carefully, but it gets fragile. That is where regular expressions come in.

Regular expressions with re

A regular expression, or regex, is a small pattern language for describing text. In Python, the re module applies patterns to strings. A few building blocks cover most log work: \d matches a digit and \S any character that is not whitespace; + means one or more of the previous item and * means zero or more; square brackets match any one of a set of characters; and parentheses capture part of the match so that you can extract it. Named groups, written (?P<name>...), let you pull out fields by name, which keeps patterns readable.

import re

LINE = re.compile(
    r'(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] '
    r'"(?P<method>\S+) (?P<path>\S+) [^"]*" (?P<status>\d{3}) (?P<size>\d+|-)'
)
line = '203.0.113.42 - - [03/Nov/2026:08:14:07 -0800] "GET /shop/beans HTTP/1.1" 200 5120 "-" "Mozilla/5.0"'
m = LINE.match(line)
print(m.group("ip"), m.group("method"), m.group("path"), m.group("status"))
record = m.groupdict()
print(record)

203.0.113.42 GET /shop/beans 200
{'ip': '203.0.113.42', 'time': '03/Nov/2026:08:14:07 -0800', 'method': 'GET', 'path': '/shop/beans', 'status': '200', 'size': '5120'}

Reading the pattern piece by piece: \S+ takes the IP address up to the first space; two more \S+ skip the dash fields; \[ and \] match literal square brackets, with [^\]]+ capturing everything up to the closing bracket; inside the quotation marks, the method and path are captured and [^"]* skips the rest of the request; \d{3} captures exactly three digits for the status; and the size is digits or a dash. Writing the pattern as a raw string, with r before the quotation mark, stops Python from treating the backslashes as escape characters. Compiling the pattern once with re.compile and reusing it is faster and clearer when you apply it to thousands of lines.

match only succeeds if the pattern matches at the start of the line, search finds the pattern anywhere, and findall returns every match as a list. A line that does not fit the pattern makes match return None, so always check before calling group: real logs contain odd lines from probes, truncated writes and software upgrades, and a parser that crashes on the first odd line is no use at 3 a.m.

import re

sample = '198.51.100.7 - - [03/Nov/2026:09:30:00 -0800] "POST /cart HTTP/1.1" 500 0 "-" "curl/8.5"'

def parse_line(line):
    return None

print(parse_line(sample))

Counting and summarizing

Once each line is a dictionary, questions about the log become questions about a list of records, and the tools from PYTH-101 apply. collections.Counter counts how often each value appears; most_common gives the top entries. Filtering with a condition and counting again answers questions such as which pages return 404 most often, or which addresses produce the most failed logins.

import re
from collections import Counter

LINE = re.compile(r'(?P<ip>\S+) \S+ \S+ \[[^\]]+\] "(?P<method>\S+) (?P<path>\S+) [^"]*" (?P<status>\d{3})')
log = """203.0.113.42 - - [03/Nov/2026:08:14:07 -0800] "GET /shop/beans HTTP/1.1" 200 5120 "-" "Mozilla/5.0"
203.0.113.42 - - [03/Nov/2026:08:14:09 -0800] "GET /shop/cart HTTP/1.1" 200 2048 "-" "Mozilla/5.0"
198.51.100.23 - - [03/Nov/2026:08:20:00 -0800] "GET /wp-login.php HTTP/1.1" 404 312 "-" "scanner"
198.51.100.23 - - [03/Nov/2026:08:20:01 -0800] "GET /.env HTTP/1.1" 404 312 "-" "scanner"
203.0.113.77 - - [03/Nov/2026:08:31:15 -0800] "POST /account/login HTTP/1.1" 401 180 "-" "Mozilla/5.0"
203.0.113.77 - - [03/Nov/2026:08:31:19 -0800] "POST /account/login HTTP/1.1" 401 180 "-" "Mozilla/5.0"
203.0.113.77 - - [03/Nov/2026:08:31:24 -0800] "POST /account/login HTTP/1.1" 200 950 "-" "Mozilla/5.0"
192.0.2.10 - - [03/Nov/2026:08:40:02 -0800] "POST /cart/checkout HTTP/1.1" 500 0 "-" "Mozilla/5.0"
this line was cut short by a restart
"""
records, bad = [], 0
for line in log.splitlines():
    m = LINE.match(line)
    if m:
        records.append(m.groupdict())
    else:
        bad += 1
status = Counter(r["status"] for r in records)
print("requests:", len(records), "| unparsed lines:", bad)
print("by status:", dict(sorted(status.items())))
print("404 paths:", Counter(r["path"] for r in records if r["status"] == "404").most_common())
print("failed logins by IP:", Counter(r["ip"] for r in records if r["status"] == "401").most_common())

requests: 8 | unparsed lines: 1
by status: {'200': 3, '401': 2, '404': 2, '500': 1}
404 paths: [('/wp-login.php', 1), ('/.env', 1)]
failed logins by IP: [('203.0.113.77', 2)]

Even this tiny sample tells a story. One address requested /wp-login.php and /.env, files that do not exist on Ridgeline's shop: that is an automated scanner probing for weaknesses common on other websites, which is normal background noise on the internet but worth watching. Another address failed to log in twice before succeeding, probably a customer who mistyped a password. And one checkout returned a 500 error, which is a real problem for the business and goes to the top of Jordan's list. Counting the lines that did not parse is a good habit too: a sudden jump in unparsed lines usually means the log format changed.

def status_counts(lines):
    counts = {}
    return counts

print(status_counts(lines))

Times, windows and large files

Questions about logs are often about time: how many errors in the last hour, or did failed logins come in a burst. The timestamp in square brackets can be turned into a datetime object with datetime.strptime and a format string that describes its layout, here %d/%b/%Y:%H:%M:%S %z for day, abbreviated month name, year, time and time zone offset. Datetimes can then be compared, subtracted and grouped, for example by hour, using the techniques from PYTH-101.

from datetime import datetime
from collections import Counter

stamps = ["03/Nov/2026:08:14:07 -0800", "03/Nov/2026:08:52:40 -0800",
          "03/Nov/2026:09:03:11 -0800", "03/Nov/2026:09:04:59 -0800", "03/Nov/2026:11:30:00 -0800"]
times = [datetime.strptime(s, "%d/%b/%Y:%H:%M:%S %z") for s in stamps]
print(times[0].isoformat())
print("gap between first two:", times[1] - times[0])
by_hour = Counter(t.strftime("%H:00") for t in times)
for hour, n in sorted(by_hour.items()):
    print(hour, "#" * n)

2026-11-03T08:14:07-08:00
gap between first two: 0:38:33
08:00 ##
09:00 ##
11:00 #

Real logs can be large, hundreds of megabytes on a busy site. The key habit is to read them line by line, looping over the open file, rather than reading the whole file into memory with read. Combined with a compiled pattern and a Counter, a loop like this handles millions of lines comfortably. Logs are also rotated: each day or week the current file is renamed, often compressed to access.log.1.gz, and a fresh one started. The gzip module opens compressed logs with the same interface, gzip.open(path, 'rt', encoding='utf-8'), so the same parsing loop works on old logs too.

import gzip, re
from collections import Counter
from pathlib import Path

STATUS = re.compile(r'" (\d{3}) ')
# make a small compressed log, as log rotation would
with gzip.open("access.log.1.gz", "wt", encoding="utf-8") as f:
    f.write('203.0.113.4 - - [02/Nov/2026:10:00:00 -0800] "GET / HTTP/1.1" 200 900 "-" "x"\n' * 3)
    f.write('203.0.113.5 - - [02/Nov/2026:10:01:00 -0800] "GET /x HTTP/1.1" 404 10 "-" "x"\n')

counts = Counter()
with gzip.open("access.log.1.gz", "rt", encoding="utf-8") as f:
    for line in f:                       # one line at a time, never the whole file
        m = STATUS.search(line)
        if m:
            counts[m.group(1)] += 1
print(dict(counts), "| file exists:", Path("access.log.1.gz").exists())

{'200': 3, '404': 1} | file exists: True

def top_missing(lines, n):
    return []

print(top_missing(lines, 2))