Reading web pages responsibly

Not every source of useful information offers an API. Northshore Green Coffee publishes its new arrivals on a public web page a few days before the lots appear in the weekly price feed, and Kai likes to know early when a good Guatemalan or Rwandan lot arrives. Reading information out of web pages with a program is called web scraping. It is a powerful technique and a responsibility: the page belongs to someone else, it runs on their servers at their cost, and it may contain material protected by copyright or personal information protected by privacy law. Done carelessly, scraping can break the site's rules, overload a small business's server or collect data nobody agreed to share.

This lesson starts with the questions to ask before scraping anything, then shows how to read robots.txt, how HTML is structured, how to extract data with the standard library's HTML parser and with the popular Beautiful Soup library, and how to write a scraper that is polite, cached and robust when the page changes.

Look for a better source first

Before writing a scraper, check whether the information is available in a form meant for programs. Many sites have an API, sometimes only mentioned in a footer link for developers. Many publish feeds: RSS or Atom for news and updates, CSV or JSON downloads for data, sitemaps listing their pages. Some will simply send you the data if you ask; when Jordan emailed Northshore, they offered to add new arrivals to the weekly CSV feed from next season. An official source is more reliable than a scraper, which breaks whenever the page layout changes, and it comes with clear permission to use the data.

When scraping is the only option, ask four questions before writing any code. Do the site's terms of use allow automated access? Does robots.txt allow it? Is the data personal information, such as names, emails or reviews tied to people? And what load will the scraper put on the server? The answers decide whether to go ahead, and how.

Terms, copyright and personal information

A site's terms of use often say whether automated access is allowed, and some forbid it outright. Breaking them can lead to an account being closed or, in serious cases, legal action. Facts such as a price or an arrival date are generally not protected by copyright, but the text, photographs and overall compilation of a page often are, so copying descriptions or images into your own site is a different matter from noting a price. Logging in to reach data, getting around paywalls, defeating CAPTCHAs or rate limits, or pretending to be a browser to avoid a block all signal that the owner does not want automated access, and a responsible scraper does not do any of them.

Personal information needs particular care. In the United States, there is no single national privacy law for businesses, but a growing number of states, led by California with the CCPA as amended by the CPRA, give people rights over personal information that businesses collect, and federal rules cover areas such as children's data and health information. Information being visible on a public page does not mean it may be collected and reused for any purpose. A script that gathers names, email addresses or photos of people should not be written without a clear, lawful purpose and advice on the rules that apply.

Kai's arrivals check passes all four tests: Northshore's terms allow personal and business use of the price information, robots.txt does not forbid the page, the data is about coffee lots rather than people, and one request a day is a negligible load.

robots.txt

Most sites publish a file at /robots.txt that tells automated clients, called robots or crawlers, which paths they may visit. It lists rules for user agents: Disallow lines block paths and Allow lines permit exceptions, and some sites add a Crawl-delay asking for a pause between requests. robots.txt is a convention rather than a lock, which is exactly why respecting it matters: it is the site owner's stated wish. Python's standard library can read and apply it with urllib.robotparser.

from urllib.robotparser import RobotFileParser

ROBOTS = """User-agent: *
Disallow: /account/
Disallow: /cart
Allow: /account/help
Crawl-delay: 10

User-agent: BadBot
Disallow: /
"""

rp = RobotFileParser()
rp.parse(ROBOTS.splitlines())        # a real script uses rp.set_url(".../robots.txt") and rp.read()
agent = "ridgeline-arrivals/1.0"
for path in ("/arrivals", "/account/orders", "/account/help", "/cart"):
    print(path, rp.can_fetch(agent, "https://northshore.example" + path))
print("crawl delay:", rp.crawl_delay(agent))
print("BadBot on /arrivals:", rp.can_fetch("BadBot", "https://northshore.example/arrivals"))

/arrivals True
/account/orders False
/account/help True
/cart False
crawl delay: 10
BadBot on /arrivals: False

A scraper should check can_fetch before every new path and honor crawl_delay when it is given. Reading robots.txt once at start-up and keeping the parser is enough for a short run.

def allowed_paths(robots_text, agent, paths):
    return [p for p in paths if 'account' not in p]

print(allowed_paths(ROBOTS, 'ridgeline-arrivals/1.0', ['/arrivals', '/account/help', '/search?q=kenya']))

How HTML is structured

A web page is an HTML document: a tree of elements written as tags, much like the XML of lesson 4 but more forgiving. A table is a table element containing tr rows, each with th header cells or td data cells. Elements can carry attributes, and two matter most for scraping: id, a name that should be unique on the page, and class, one or more labels used for styling, such as class="lot soldout". These attributes are the handles a scraper holds on to. Your browser's developer tools, opened with F12 or a right-click and Inspect, show the tree for any part of the page, which is the quickest way to find the elements and classes you need.

Here is the part of Northshore's arrivals page Kai cares about. The listing below stores it as a string so it can be parsed without a network; in the real script, the text comes from requests.get(url).text.

<html><head><title>Northshore Green Coffee: new arrivals</title></head>
<body>
<h1>New arrivals</h1>
<table id="arrivals">
  <tr><th>Lot</th><th>Origin</th><th>Price per kg</th></tr>
  <tr class="lot"><td>NS-2616</td><td>Guatemala, Huehuetenango</td><td class="price">8.75</td></tr>
  <tr class="lot"><td>NS-2617</td><td>Rwanda</td><td class="price">9.10</td></tr>
  <tr class="lot soldout"><td>NS-2618</td><td>Brazil</td><td class="price">6.95</td></tr>
</table>
<p class="updated">Updated 2026-11-09</p>
</body></html>

Parsing HTML with the standard library

Never pull data out of HTML with string searches or regular expressions over the whole page; attributes move, whitespace changes and the code breaks. Use a parser. The standard library's html.parser.HTMLParser reads a document and calls methods you define when it meets a start tag, a piece of text and an end tag. It takes a little more code than third-party libraries, but it works everywhere with nothing to install.

PAGE = '''<html><head><title>Northshore Green Coffee: new arrivals</title></head>
<body>
<h1>New arrivals</h1>
<table id="arrivals">
  <tr><th>Lot</th><th>Origin</th><th>Price per kg</th></tr>
  <tr class="lot"><td>NS-2616</td><td>Guatemala, Huehuetenango</td><td class="price">8.75</td></tr>
  <tr class="lot"><td>NS-2617</td><td>Rwanda</td><td class="price">9.10</td></tr>
  <tr class="lot soldout"><td>NS-2618</td><td>Brazil</td><td class="price">6.95</td></tr>
</table>
<p class="updated">Updated 2026-11-09</p>
</body></html>'''

from html.parser import HTMLParser

class ArrivalsParser(HTMLParser):
    """Collect the cells of rows with class "lot", skipping sold-out lots."""
    def __init__(self):
        super().__init__()
        self.rows, self.row, self.in_cell, self.skip = [], None, False, False
    def handle_starttag(self, tag, attrs):
        classes = (dict(attrs).get("class") or "").split()
        if tag == "tr" and "lot" in classes:
            self.row, self.skip = [], "soldout" in classes
        elif tag == "td" and self.row is not None:
            self.in_cell = True
            self.row.append("")
    def handle_data(self, data):
        if self.in_cell:
            self.row[-1] += data.strip()
    def handle_endtag(self, tag):
        if tag == "td":
            self.in_cell = False
        elif tag == "tr" and self.row is not None:
            if not self.skip:
                self.rows.append(tuple(self.row))
            self.row = None

parser = ArrivalsParser()
parser.feed(PAGE)
for lot, origin, price in parser.rows:
    print(f"{lot}  {origin:26} {price}")

NS-2616  Guatemala, Huehuetenango   8.75
NS-2617  Rwanda                     9.10

The parser tracks a little state: whether it is inside a lot row, whether that row is sold out, and whether it is inside a cell. handle_starttag receives attributes as a list of name-value pairs, which dict() turns into a dictionary. The result is a list of tuples that the rest of the program can check and use, exactly like rows read from a CSV file.

from decimal import Decimal

def price_cells(html):
    out = []
    for part in html.split('class="price">')[1:]:
        out.append(part[:4])
    return out

print(price_cells(PAGE))

Beautiful Soup

For anything more than a small table, most Python developers use Beautiful Soup, a third-party library installed with python -m pip install beautifulsoup4. It builds the whole tree and lets you search it with methods such as find and find_all or with CSS selectors, the same patterns web designers use to style pages. The scraper above shrinks to a few lines.

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://northshore.example/arrivals", timeout=10,
                    headers={"User-Agent": "ridgeline-arrivals/1.0 (kai@ridgelinecoffee.example)"})
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")
for row in soup.select("table#arrivals tr.lot:not(.soldout)"):
    lot, origin, price = (td.get_text(strip=True) for td in row.find_all("td"))
    print(lot, origin, price)

The CSS selector table#arrivals tr.lot:not(.soldout) says: inside the table with id arrivals, rows with class lot that do not also have class soldout. Selectors make scrapers short and readable, and they are where a scraper usually breaks when the page is redesigned, so keep them in one place, such as constants at the top of the script.

Some pages build their content with JavaScript after loading, so the HTML that requests receives is nearly empty. Before reaching for browser automation tools such as Playwright, which drive a real browser and are much heavier, open the developer tools' Network tab: such pages usually fetch their data from a JSON endpoint, which is easier and more reliable to read directly, subject to the same rules about permission.

Being a polite scraper

A responsible scraper behaves like a considerate visitor. It identifies itself with a User-Agent that names the script and gives a contact address, so the site owner can get in touch rather than block it. It goes slowly: one request at a time, with a pause of several seconds between requests, longer if robots.txt asks for a crawl delay. It requests only the pages it needs, never follows every link on a site, and runs at quiet times. It caches what it has fetched, so running the script twice in an hour does not fetch everything twice, and it uses conditional requests with If-Modified-Since or If-None-Match, which let the server answer 304 Not Modified without sending the page again. And it stops at the first sign of trouble: a 403, a 429 or a CAPTCHA page means the site does not want this traffic.

import time

class PoliteFetcher:
    """Wait at least `delay` seconds between requests to the same host."""
    def __init__(self, fetch, delay=10, clock=time.monotonic, sleep=time.sleep):
        self.fetch, self.delay, self.clock, self.sleep = fetch, delay, clock, sleep
        self.last = {}
    def get(self, host, path):
        now = self.clock()
        wait = self.last.get(host, -1e9) + self.delay - now
        if wait > 0:
            self.sleep(wait)
            now += wait
        self.last[host] = now
        return self.fetch(host, path)

t = [0.0]
log = []
fetcher = PoliteFetcher(lambda h, p: f"<html>{p}</html>", delay=10,
                        clock=lambda: t[0], sleep=lambda s: (log.append(round(s, 1)), t.__setitem__(0, t[0] + s)))
for path in ("/arrivals", "/arrivals/NS-2616", "/arrivals/NS-2617"):
    fetcher.get("northshore.example", path)
    t[0] += 2.5                                  # time spent processing each page
print("waited:", log)

waited: [7.5, 7.5]

The fetcher waits only as long as needed: the first request goes straight away, and each later one waits until ten seconds have passed since the previous request, counting the time spent processing the page. Passing in the clock and sleep functions, rather than calling time.sleep directly, lets the listing simulate a minute of polite crawling instantly, and lets tests do the same.

def start_times(ready, delay):
    return ready

print(start_times([0, 3, 30, 31], 10))

When the page changes

A scraper depends on a page layout that its owner can change at any moment without telling anyone. Assume it will happen and design for it. Keep a saved copy of the page in your tests, so you can check the parser without the network. Check the results for sanity: if the arrivals table suddenly has no rows, or every price is empty, that is far more likely to mean a redesign than a week with no coffee, and the script should raise an alert instead of reporting "no new arrivals". Kai's script emails Jordan when it finds zero lots twice in a row, which has caught one redesign so far.