Learning, memory and reward

Memory is not one thing. Remembering where you parked, knowing that Ottawa is a capital, riding a bicycle, flinching at a sound associated with pain and finding a song more pleasant after hearing it many times all depend on different brain systems. This lesson connects the synaptic mechanisms of lesson 6 to those systems: simple learning in the sea slug Aplysia, the hippocampus and its maps of space, consolidation and the role of sleep, emotional memory in the amygdala, and the dopamine signals that teach the brain what to expect. It ends with the circadian clock and sleep, which shape when and how well we learn.

Kinds of memory

Modern classification began with a patient. In 1953 Henry Molaison, known in the literature for decades as H.M., had much of his medial temporal lobes, including most of both hippocampi, removed to treat severe epilepsy. His seizures improved, but he could no longer form new long-term memories of facts and events. Brenda Milner, at the Montreal Neurological Institute, studied him for decades and showed that his intelligence, his short-term memory and his older memories were largely spared, and that he could learn new motor skills, such as tracing a star seen in a mirror, improving day by day while having no recollection of having practised. Memory systems were dissociable.

Declarative (explicit) memory, for facts (semantic memory) and personal events (episodic memory), depends on the medial temporal lobe, including the hippocampus and entorhinal cortex. Non-declarative (implicit) memory includes procedural skills and habits, which depend on the striatum and cerebellum; priming, which depends on sensory cortex; classical conditioning, which depends on the amygdala for emotional responses and the cerebellum for motor responses; and non-associative learning such as habituation and sensitization, which can occur in simple reflex circuits. Working memory, holding information in mind for seconds while using it, depends on prefrontal cortex.

Simple learning in Aplysia

Eric Kandel chose the marine snail Aplysia californica because its nervous system has about 20,000 large, identifiable neurons. Touching its siphon causes the gill to withdraw, a reflex mediated by a few dozen sensory and motor neurons. Repeated gentle touches cause habituation, a decline in the response, because transmitter release from the sensory neurons decreases. A noxious shock to the tail causes sensitization, an enhanced response to later siphon touches. Kandel's group traced sensitization to serotonin released by modulatory interneurons onto the sensory neuron terminals, activating cyclic AMP and protein kinase A, which close potassium channels, broaden action potentials and increase transmitter release. A single shock produced short-term sensitization lasting minutes; repeated shocks produced long-term sensitization lasting days, which required PKA to enter the nucleus, activate the transcription factor CREB, and drive new protein synthesis and the growth of new synaptic connections. The same molecules, cyclic AMP, PKA and CREB, turned out to be required for long-term memory in flies and mice, revealing a conserved molecular logic. Kandel shared the 2000 Nobel Prize.

Classical conditioning and prediction

In classical conditioning a neutral conditioned stimulus (CS), such as a tone, is paired with a biologically important unconditioned stimulus (US), such as food or a shock, until the CS alone evokes a response. Learning is not driven simply by pairing. In blocking, discovered by Leon Kamin in 1968, an animal that has already learned that a light predicts a shock fails to learn about a tone added alongside the light, even though the tone is paired with the shock many times. The shock is already predicted, so there is nothing new to learn. Robert Rescorla and Allan Wagner formalized this in 1972: the associative strength \(V\) of each stimulus present changes in proportion to the prediction error, the difference between what happened and what was predicted by all stimuli present:

where \(\lambda\) is the maximum associative strength the US supports and \(\alpha\) and \(\beta\) are learning rates for the CS and US.

Dopamine and reward prediction errors

In the 1990s Wolfram Schultz recorded dopamine neurons in the ventral tegmental area and substantia nigra of monkeys learning that a light predicted juice. Before learning, the neurons fired a burst when the juice arrived. After learning, they fired at the light and not at the expected juice. If the juice was then omitted, firing dipped below baseline at the moment juice should have arrived. Dopamine neurons were not signalling reward itself but the difference between received and predicted reward. Peter Dayan, Read Montague and Terry Sejnowski recognized in 1996 that this matched the temporal-difference (TD) learning signal from reinforcement learning theory (AI-330), which compares successive predictions of future reward:

The TD error \(\delta\) is positive when things turn out better than expected, negative when worse and zero when fully predicted, and it moves backward in time to the earliest reliable predictor. Dopamine released in the striatum is thought to strengthen recently active corticostriatal synapses when \(\delta\) is positive, through the dopamine-gated plasticity mentioned in lesson 6, so actions followed by unexpectedly good outcomes become more likely. Optogenetic stimulation of dopamine neurons can itself act as a reward and can unblock learning in a blocking experiment, strong causal evidence. The theory also helps explain addiction: drugs such as cocaine raise dopamine directly, generating a prediction error that can never be fully predicted away, so drug-associated cues and actions keep being reinforced.

# Temporal-difference learning reproduces the dopamine response pattern (cue then reward).
import numpy as np
T = 12                         # time steps in a trial; cue at step 2, reward at step 8
cue, reward_t = 2, 8
alpha, gamma = 0.2, 1.0
V = np.zeros(T + 1)            # value of each time step after the cue (states before cue stay 0)
def trial(V, rewarded=True, learn=True):
    delta = np.zeros(T)
    for t in range(T):
        r = 1.0 if (t == reward_t and rewarded) else 0.0
        v_now = V[t] if t >= cue else 0.0          # the cue is unpredictable, so pre-cue value is 0
        v_next = V[t + 1] if t + 1 >= cue else 0.0
        delta[t] = r + gamma * v_next - v_now
        if learn and t >= cue:
            V[t] += alpha * delta[t]
    return delta
def show(label, d):
    print(f"{label:22s}" + " ".join(f"{x:+.2f}" if abs(x) > 0.005 else "  .  " for x in d))
print("time step:            " + " ".join(f"{t:5d}" for t in range(T)))
show("trial 1 (naive)", trial(V))
for _ in range(200):
    trial(V)
show("after learning", trial(V, learn=False))
show("reward omitted", trial(V, rewarded=False, learn=False))

time step:                0     1     2     3     4     5     6     7     8     9    10    11
trial 1 (naive)         .     .     .     .     .     .     .     .   +1.00   .     .     .  
after learning          .   +1.00   .     .     .     .     .     .     .     .     .     .  
reward omitted          .   +1.00   .     .     .     .     .     .   -1.00   .     .     .

The simulated prediction error matches the recordings: before learning it appears at the reward; after learning it appears at the cue, the earliest predictor (step 1, the transition into the cue), and vanishes at the reward; and when reward is omitted, it dips negative at the expected time. Some details of the theory are debated, for example whether dopamine also signals novelty, salience or movement, and whether it encodes value or prediction errors about sensory features, but the reward prediction error account remains one of the clearest links between a computational theory and neural activity.

The hippocampus and spatial maps

The hippocampus has a distinctive circuit. Input from layer 2 of entorhinal cortex reaches the dentate gyrus through the perforant path; dentate granule cells project to CA3 through mossy fibres; CA3 neurons project to CA1 through Schaffer collaterals and also connect extensively with each other through recurrent collaterals; and CA1 projects back to entorhinal cortex, also receiving direct entorhinal input. The dentate gyrus is thought to separate similar experiences into distinct representations (pattern separation), and the recurrent network of CA3 to complete a stored pattern from a partial cue (pattern completion), much like the attractor networks of lesson 11. The dentate gyrus is also one of the few places where new neurons may be added in adults, abundantly in rodents; whether adult neurogenesis persists substantially in humans has been contested in a series of studies since 2018.

In 1971 John O'Keefe and Jonathan Dostrovsky found that pyramidal cells in the rat hippocampus fire when the animal is in a particular location, the cell's place field; different cells cover different places, and together they form a map of the environment, as O'Keefe and Lynn Nadel argued in their 1978 book The Hippocampus as a Cognitive Map. In 2005 May-Britt and Edvard Moser discovered grid cells in the medial entorhinal cortex, which fire at multiple locations forming a regular triangular lattice across an environment, providing a metric coordinate system; head direction cells, border cells and speed cells complete the navigation system. O'Keefe and the Mosers shared the 2014 Nobel Prize. Place cells also fire at progressively earlier phases of the ongoing theta rhythm (6 to 10 hertz) as the animal crosses the field, called phase precession, so the order of places is compressed into each theta cycle; during rest and sleep, sequences of place cells replay past paths at many times their original speed, during brief high-frequency events called sharp-wave ripples.

# A model place cell: firing rate map built from spikes during a simulated random walk in a 1 m arena.
import numpy as np
rng = np.random.default_rng(4)
dt, steps = 0.02, 30000                       # 10 minutes at 50 Hz
pos = np.zeros((steps, 2)); pos[0] = [0.5, 0.5]
heading = 0.0
for i in range(1, steps):                     # smooth random walk with wall reflection
    heading += rng.normal(0, 0.3)
    step = 0.25 * dt * np.array([np.cos(heading), np.sin(heading)])
    new = pos[i-1] + step
    if not (0 < new[0] < 1 and 0 < new[1] < 1):
        heading += np.pi; new = pos[i-1]
    pos[i] = new
field, width, peak = np.array([0.7, 0.3]), 0.10, 15.0    # field middle (m), width (m), peak rate (Hz)
rate = peak * np.exp(-np.sum((pos - field)**2, axis=1) / (2 * width**2))
spikes = rng.random(steps) < rate * dt
bins = 10
occ, _, _ = np.histogram2d(pos[:, 0], pos[:, 1], bins=bins, range=[[0, 1], [0, 1]])
spk, _, _ = np.histogram2d(pos[spikes, 0], pos[spikes, 1], bins=bins, range=[[0, 1], [0, 1]])
ratemap = np.where(occ > 0, spk / (occ * dt), 0)
print(f"total spikes: {spikes.sum()}, coverage: {np.mean(occ > 0) * 100:.0f}% of bins visited")
print("rate map (Hz), y increasing upward:")
for row in range(bins - 1, -1, -1):
    print(" ".join(f"{ratemap[col, row]:4.1f}" for col in range(bins)))
iy, ix = np.unravel_index(np.argmax(ratemap.T), ratemap.T.shape)
print(f"peak bin middle: x = {(ix + 0.5) / bins:.2f} m, y = {(iy + 0.5) / bins:.2f} m (true field 0.70, 0.30)")

total spikes: 379, coverage: 100% of bins visited
rate map (Hz), y increasing upward:
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.4  0.8  0.8  0.0  0.0
 0.0  0.0  0.0  0.2  0.3  1.2  4.9  4.0  1.6  0.2
 0.0  0.0  0.0  0.1  0.7  2.9  9.4 11.0  5.5  0.5
 0.0  0.0  0.0  0.6  0.6  2.0 10.7 12.4  5.4  0.9
 0.0  0.0  0.0  0.0  0.2  0.6  4.8  4.8  2.9  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.8  0.0  0.0  0.0
peak bin middle: x = 0.75 m, y = 0.25 m (true field 0.70, 0.30)

Dividing spike counts by time spent in each location (occupancy) is essential; without it, a cell would seem to prefer wherever the animal happened to spend most time. Human recordings from epilepsy patients navigating virtual environments have found place-like and grid-like cells, and in 2005 a famous study found single neurons in the human medial temporal lobe that responded to many different pictures of one person, such as a particular actor, suggesting highly selective, abstract representations sometimes called concept cells.

Consolidation, engrams and sleep

Declarative memories are initially dependent on the hippocampus but, according to the standard consolidation model, gradually become independent of it as they are reorganized in neocortex over weeks to years; patients like H.M. had graded retrograde amnesia, losing memories from the years just before surgery more than older ones. An alternative, multiple trace theory, holds that vivid episodic memories always need the hippocampus while semantic gist becomes cortical. Retrieval also changes memories: a reactivated memory can become labile and must be restabilized, a process called reconsolidation, which has been explored as a way to weaken traumatic memories.

Memory traces can now be manipulated directly. In 2012 Susumu Tonegawa's group labelled dentate gyrus neurons active during fear learning with channelrhodopsin, using a promoter driven by activity, and showed that reactivating them with light in a different context made mice freeze, as if recalling the fear. Later studies created false associations and recovered memories in mouse models of amnesia, suggesting that some forgotten memories are inaccessible rather than erased.

Sleep supports consolidation. During slow-wave sleep, hippocampal sharp-wave ripples coincide with cortical slow oscillations and thalamocortical sleep spindles, and this coupling is thought to transfer information to cortex; disrupting ripples after learning impairs memory in rats, and boosting slow oscillations with sounds timed to them improves memory in some human studies. Sleep may also downscale synapses strengthened during the day, according to the synaptic homeostasis hypothesis, and helps clear metabolites from the brain.

Emotional memory and the amygdala

In fear conditioning, a tone paired with a mild shock comes to elicit freezing, increased heart rate and stress hormones. The amygdala is essential. Tone and shock information converge in the lateral amygdala, where LTP strengthens tone inputs, as described in lesson 6; the central amygdala drives fear responses through the hypothalamus and brainstem. Extinction, the decline of fear when the tone is repeatedly presented without shock, is new learning that suppresses the original memory rather than erasing it; it depends on the ventromedial prefrontal cortex and is context-specific, which is why fear can return in a new setting. Exposure therapy for anxiety disorders and post-traumatic stress disorder builds on extinction. Noradrenaline and stress hormones released during emotional events enhance consolidation, which is why emotional memories are vivid.

The circadian clock

Nearly every cell has a molecular clock with a period of about 24 hours, built from a transcriptional–translational feedback loop. Jeffrey Hall, Michael Rosbash and Michael Young worked out the mechanism in fruit flies, receiving the 2017 Nobel Prize: the proteins PERIOD and TIMELESS accumulate, enter the nucleus and inhibit their own transcription, then degrade, letting the cycle restart. In mammals, CLOCK and BMAL1 drive transcription of Period and Cryptochrome genes, whose proteins feed back to inhibit CLOCK–BMAL1. The master clock is the suprachiasmatic nucleus (SCN) of the hypothalamus, about 20,000 neurons on each side, which receives light information from melanopsin-containing retinal ganglion cells (lesson 7) and synchronizes clocks throughout the body. Sleep is regulated by this circadian timer together with a homeostatic drive that builds during waking, partly through adenosine accumulation, which caffeine blocks. Shift work and jet lag misalign these processes, with measurable effects on learning, mood and metabolism.