Scope and closures

Every name in a Python program lives somewhere. When Jordan writes stock = 42 at the top of a script, or price = 18.50 inside a function, Python records that name in a particular namespace, and the rules for finding names decide which value an expression like stock * price actually uses. In small scripts this rarely matters. In a program with dozens of functions, it explains why a function can read a constant but not change a counter, why two functions can both use a variable called total without interfering, and how a function can remember a value between calls without any global variables at all.

This lesson covers namespaces and the LEGB rule that Python uses to look up names, the global and nonlocal statements and why they should be rare, nested functions, closures that capture values from an enclosing function, and function factories that build customized functions on demand. It ends with decorators, a widely used Python feature that is built entirely from closures.

Namespaces

A namespace is a mapping from names to objects, much like a dictionary whose keys are variable names. Python keeps several at once. The built-in namespace holds names such as print, len and ValueError, available everywhere. Each module has a global namespace for the names defined at its top level. And every call of a function creates a fresh local namespace for its parameters and the variables it assigns, which is thrown away when the call returns. The same name can exist in several namespaces at once, referring to different objects.

bags = 42                     # global name

def restock(added):
    bags = added + 5          # a new local name, unrelated to the global
    print("inside:", bags)
    return bags

restock(10)
print("outside:", bags)
print("restock" in globals(), "added" in globals())

inside: 15
outside: 42
True False

Assigning bags inside restock did not touch the global bags. Python decided, when it compiled the function, that bags is local to restock because the function assigns to it. The global is shadowed, hidden behind the local name, for the duration of the call. globals() returns the module's namespace as a dictionary; added is not in it because parameters are local.

The LEGB rule

When Python meets a name in an expression, it searches the namespaces in a fixed order and uses the first match. The order is remembered as LEGB: Local (the current function), Enclosing (any functions that contain the current one, from the inside out), Global (the module), and Built-in. If the name is not found in any of them, Python raises a NameError.

REORDER_POINT = 10                       # global

def low_report(stock):                   # stock is local to low_report
    def flag(code):                      # code is local to flag
        # stock comes from the enclosing function, REORDER_POINT from
        # the module, and len and print from the built-ins.
        return code + (" LOW" if stock[code] < REORDER_POINT else "")
    for code in sorted(stock):
        print(flag(code), len(code))

low_report({"RC-104": 9, "RC-101": 42})

RC-101 6
RC-104 LOW 6

Scope is decided by where code is written, not by where it is called from. flag can see stock because flag is written inside low_report. This is called lexical scoping, and it makes programs predictable: you can tell which variable a name refers to by reading the code around it.

One consequence surprises beginners. Because Python decides at compile time that a name is local if the function assigns to it anywhere, reading that name before the assignment fails, even if a global with the same name exists.

sold_today = 0

def record_sale(qty):
    try:
        sold_today = sold_today + qty    # reads the local before it exists
    except UnboundLocalError as err:
        print(type(err).__name__ + ":", err)
    return qty

record_sale(2)
print(sold_today)

UnboundLocalError: cannot access local variable 'sold_today' where it is not associated with a value
0

global and nonlocal

The global statement tells Python that a name inside a function refers to the module-level variable, so assigning it changes the global. It works, but it ties the function to that variable: the function cannot be tested on its own, two calls can interfere with each other, and a reader has to search the whole module to find out who changes what. Code reviewers treat global as a sign that a function should take a parameter and return a value instead.

sold_today = 0

def record_sale_global(qty):
    global sold_today
    sold_today += qty

def record_sale(total, qty):        # the better design
    return total + qty

record_sale_global(2)
record_sale_global(3)
print("global version:", sold_today)

total = 0
for qty in (2, 3):
    total = record_sale(total, qty)
print("parameter version:", total)

global version: 5
parameter version: 5

nonlocal is the equivalent for enclosing functions: it lets an inner function rebind a variable belonging to the function around it. Unlike global, nonlocal has a respectable use, because the variable it changes is private to one function call rather than shared by the whole program. That use is the closure.

Nested functions and closures

A function defined inside another function is a nested function. If the inner function uses names from the outer one, and the outer function returns the inner one, something remarkable happens: the inner function keeps access to those names even after the outer function has finished. The returned function, together with the variables it captured, is called a closure.

def make_tax_adder(rate):
    def add_tax(amount):
        return round(amount * (1 + rate), 2)   # rate comes from the enclosing call
    return add_tax

gst = make_tax_adder(0.05)
idaho = make_tax_adder(0.06)
print(gst(10.00), idaho(10.00))
print(gst(9.00), idaho(9.00))
print(gst.__closure__[0].cell_contents)

10.5 10.6
9.45 9.54
0.05

make_tax_adder has returned twice, and its local variable rate should have disappeared each time. It did not, because each returned add_tax function holds on to the rate from the call that created it. gst remembers 0.05 and idaho remembers 0.06. The last line shows where Python keeps the captured value, in a cell attached to the function; you never need to touch it, but it shows that closures are not magic.

A function that builds and returns other functions is called a factory. Factories are useful whenever you need several functions that differ only in a setting: tax adders for different regions, price formatters for different currencies, discount functions for different loyalty tiers. Each can be created with one call instead of being written out by hand.

Closures that remember state

With nonlocal, a closure can update the variables it captured, which gives it memory between calls. Kai wants a batch numbering function for the roast log: each call should return the next batch id, starting from a given number. A closure keeps the counter private, so no other code can reset it by accident.

def make_batch_numberer(start):
    count = start
    def next_batch():
        nonlocal count
        batch_id = f"B-{count:04d}"
        count += 1
        return batch_id
    return next_batch

next_batch = make_batch_numberer(412)
print(next_batch(), next_batch(), next_batch())
test_batches = make_batch_numberer(1)       # an independent counter
print(test_batches(), next_batch())

B-0412 B-0413 B-0414
B-0001 B-0415

Without nonlocal, count += 1 would make count local to next_batch, and the first call would fail with UnboundLocalError. With it, each numberer has its own count, which lives as long as the function does. In lesson 4 you will meet classes, which are the usual way to bundle data with functions when there is more than one piece of state. For a single counter or setting, a closure is lighter and perfectly professional.

def make_discount(rate):
    return round(20.00 * (1 - rate), 2)

roast = make_discount(0.10)
print(roast)

A classic closure trap: late binding

Closures capture variables, not values. If an inner function is created inside a loop and refers to the loop variable, every function sees the variable's final value, because they all look it up when they are called, after the loop has finished. This is called late binding.

adders = []
for rate in (0.05, 0.06, 0.13):
    adders.append(lambda amount: round(amount * (1 + rate), 2))
print([f(100) for f in adders])          # all use the last rate

adders = []
for rate in (0.05, 0.06, 0.13):
    adders.append(lambda amount, r=rate: round(amount * (1 + r), 2))
print([f(100) for f in adders])          # each keeps its own rate

[113.0, 113.0, 113.0]
[105.0, 106.0, 113.0]

The fix in the second loop uses a default argument, which is evaluated when the lambda is created and so freezes the current value. A factory function such as make_tax_adder avoids the problem too, because each call has its own rate variable. When a list of functions all give the same answer, late binding is the first thing to check.

Decorators

A decorator is a function that takes a function and returns a new function that adds some behaviour around it. Decorators are built from exactly the ideas in this lesson: functions as values, nested functions and closures. Python gives them a special syntax: writing @name on the line above a def means the function is defined and then immediately replaced by name(function).

import functools

def counted(func):
    """Count how many times func is called."""
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        wrapper.calls += 1
        return func(*args, **kwargs)
    wrapper.calls = 0
    return wrapper

@counted
def loyalty_tier(points):
    """Return the loyalty tier for a number of points."""
    return "Reserve" if points >= 300 else "Roast" if points >= 100 else "Bean"

for p in (40, 150, 320):
    print(p, loyalty_tier(p))
print("calls:", loyalty_tier.calls)
print(loyalty_tier.__name__, "-", loyalty_tier.__doc__)

40 Bean
150 Roast
320 Reserve
calls: 3
loyalty_tier - Return the loyalty tier for a number of points.

wrapper accepts *args and **kwargs so it can pass any arguments straight through to the original function, which is captured in the closure as func. functools.wraps copies the original function's name and docstring onto the wrapper, so help and error messages still describe loyalty_tier rather than a mysterious wrapper. Decorators are used throughout Python for logging, timing, checking permissions, caching and registering functions. You will write web routes with them in PYTH-220 and test fixtures with them in PYTH-230.

The standard library includes ready-made decorators. functools.lru_cache remembers the results of previous calls and returns them instantly when the same arguments come back, which can turn a slow function into a fast one at the cost of some memory. PYTH-250 returns to caching when it covers performance.

import functools

@functools.lru_cache(maxsize=None)
def shipping_rate(zone):
    print("  looking up zone", zone)        # pretend this is slow
    return {1: 8.00, 2: 11.50, 3: 15.25}[zone]

for zone in (1, 2, 1, 1, 3, 2):
    print(zone, shipping_rate(zone))
print(shipping_rate.cache_info().hits, "answers came from the cache")

  looking up zone 1
1 8.0
  looking up zone 2
2 11.5
1 8.0
1 8.0
  looking up zone 3
3 15.25
2 11.5
3 answers came from the cache

import functools

def positive_qty(func):
    @functools.wraps(func)
    def wrapper(code, qty, *args, **kwargs):
        if qty <= 0:
            raise ValueError(f"{func.__name__}: quantity must be positive")
        return func(code, qty, *args, **kwargs)
    return wrapper

stock = {"RC-104": 9}

@positive_qty
def sell(code, qty):
    stock[code] -= qty
    return stock[code]

@positive_qty
def restock(code, qty):
    stock[code] += qty
    return stock[code]

print(sell("RC-104", 2), restock("RC-104", 30))
for bad in (0, -3):
    try:
        sell("RC-104", bad)
    except ValueError as err:
        print("ValueError:", err)

7 37
ValueError: sell: quantity must be positive
ValueError: sell: quantity must be positive

import functools

def logged(func):
    return func

@logged
def add_points(a, b):
    return a + b

print(add_points(20, 15))
print(LOG)