Post-occupancy evaluation

A year after it opened, the Rail Yard Commons is busy. The library lends more books than any branch of its size in the city, the makerspace has a waiting list, the settlement offices see two hundred families a month and the gathering room is booked most evenings. The building has also produced a list of complaints: the reading room is too warm on sunny afternoons in May, the café's coffee grinder can be heard in the quiet study area, the classroom blinds are hard to reach, and the first year's metered energy use is about 12% above the model. Thibodeau's last session is held in the commons itself, with Natarajan, the branch manager, Anders Holm from the performance course and Priyanka Sethi, the energy modeller. 'Most architects never find out how their buildings work,' he says. 'They hand over the keys and move on to the next one. This is how you find out.'

Why evaluate a building in use

A post-occupancy evaluation, or POE, is a systematic study of a building after it has been occupied for long enough to settle, usually one to two years. It asks how well the building meets the needs of the people who use it and the goals set for it: the program, the comfort and health of its occupants, its energy and water use, its maintenance, its accessibility and its contribution to its community. It serves several readers. The owner learns how to operate and adjust the building and what to ask for in the next one. The occupants are heard, and their problems are fixed. The design team learns which of its decisions worked, which did not and why, which is the only reliable way an office improves over time. And the profession as a whole gains evidence that is otherwise scarce.

POE has a long history but a patchy record. Researchers developed the methods in the 1970s and 1980s; the book by Wolfgang Preiser and his colleagues in 1988 set out three levels of study, from a quick indicative review through an investigative study to a diagnostic one with detailed measurement. Since then, public owners and some universities and health systems have made evaluations routine, and frameworks such as Soft Landings, developed in the United Kingdom in the late 2000s, have tried to build evaluation and aftercare into the delivery process from the start. Many buildings are still never evaluated, mainly because no one is paid to do it. The city's agreement with Thibodeau's former office included a one-year evaluation as a fee item, which is the main reason this one is happening.

Methods

An evaluation combines several kinds of evidence, because each sees something the others miss. A walkthrough with the building's managers and staff, guided by a checklist drawn from the project's goals, finds obvious problems quickly. Occupant surveys, using a standard questionnaire so that results can be compared with other buildings, measure satisfaction with temperature, air quality, lighting, acoustics, cleanliness and layout; the Center for the Built Environment at the University of California, Berkeley, has run such a survey since about 2000 and holds results from many hundreds of buildings. Interviews and small group discussions explain what the survey numbers mean. Environmental measurements, with loggers for temperature, humidity, carbon dioxide, light and sound left in typical rooms for some weeks, show whether complaints match conditions. Metered energy and water use, compared with the design model and with similar buildings, show whether the building performs as intended. And observation of how spaces are actually used, sometimes by mapping where people sit and what they do at set times of day, shows whether the program was right.

Evaluation involves people, and the ethics are real. Surveys and interviews must be voluntary, with consent and a clear statement of how the results will be used; responses must be anonymous, especially where staff might fear criticizing their employer's building; and observations of public spaces must avoid identifying individuals. The city's evaluation team uses the city's own research approval process, and the survey collects no names.

What the occupants said

The survey of staff and regular users had 214 responses, about a third of those invited, which is a good rate for this kind of study. On a scale from minus three, very dissatisfied, to plus three, very satisfied, the results are strongly positive for daylight, the layout of the library and the commons, and the quality of the air; mixed for temperature; and negative for noise in the quiet study area. The code below summarizes the responses by topic and the share of respondents who were dissatisfied, which is how many evaluations report results, because an average can hide a group of very unhappy occupants.

# Summarize occupant survey responses by topic (scale -3 to +3).
responses = {  # topic: counts of answers from -3 to +3
    "daylight":            [1, 3, 6, 14, 41, 82, 67],
    "air quality":         [2, 4, 9, 27, 58, 71, 43],
    "temperature":         [8, 17, 29, 38, 54, 46, 22],
    "noise in quiet areas": [19, 31, 37, 33, 41, 35, 18],
    "layout":              [2, 5, 8, 21, 49, 77, 52],
}
scale = [-3, -2, -1, 0, 1, 2, 3]
print(f"{'topic':22s} {'n':>4s} {'mean':>5s} {'dissatisfied':>13s}")
for topic, counts in responses.items():
    n = sum(counts)
    mean = sum(s * c for s, c in zip(scale, counts)) / n
    dissatisfied = sum(counts[:3]) / n          # answers of -3, -2 or -1
    flag = "  investigate" if dissatisfied > 0.20 else ""
    print(f"{topic:22s} {n:4d} {mean:5.2f} {dissatisfied:12.0%}{flag}")

topic                     n  mean  dissatisfied
daylight                214  1.83           5%
air quality             214  1.43           7%
temperature             214  0.58          25%  investigate
noise in quiet areas    214  0.04          41%  investigate
layout                  214  1.57           7%

Temperature and noise both have more than a fifth of respondents dissatisfied, a common threshold in such surveys for a problem worth investigating. The interviews explain both. The reading room overheats on sunny afternoons in spring because the exterior blinds at the commons were deleted to save money during construction, as the change order lesson described, and the reading room's west glazing turned out to depend on them more than the energy model assumed. The noise in the quiet study area comes from the café: the coffee grinder and the dishwasher share a wall with the study area, and the acoustic separation the design specified was reduced when a door between them was changed to an open archway in a late owner request. Neither problem is mysterious once the history is known, which is why the evaluation team includes people who were on the project.

Energy and the performance gap

The building was designed for an energy use intensity of 63 kWh per square metre a year against a target of 65. The first year's metered use, adjusted to a typical weather year, was 71. Sethi compares the meters with her model month by month, using the calibration measures from ASHRAE Guideline 14 that the performance course taught, and finds that the gap is concentrated in a few places rather than spread evenly: the café's equipment runs longer hours than the program assumed, the makerspace dust collector and kilns are used heavily in the evenings, and the ventilation for the multipurpose wing runs at full flow whenever it is booked, even for small groups. None of these is a design failure in the usual sense; they are differences between assumed and actual use. Two can be fixed by the city's operators at no cost.

# Compare metered monthly energy with the model and check calibration (ASHRAE Guideline 14).
area = 3750                                   # square metres
model = [27.4, 24.5, 23.1, 18.4, 14.8, 13.7, 15.2, 14.8, 15.5, 19.3, 23.0, 26.6]  # MWh per month
meter = [29.6, 26.8, 25.6, 21.1, 17.6, 17.1, 17.7, 17.3, 18.1, 21.8, 25.1, 28.5]
n = len(model)
mean_meter = sum(meter) / n
nmbe = sum(m - s for m, s in zip(meter, model)) / (n * mean_meter)
cv = (sum((m - s) ** 2 for m, s in zip(meter, model)) / n) ** 0.5 / mean_meter
print(f"model {sum(model) * 1000 / area:.0f} kWh/m2, metered {sum(meter) * 1000 / area:.0f} kWh/m2")
print(f"NMBE {nmbe:.1%} (limit +/-5%), CV(RMSE) {cv:.1%} (limit 15%)")
# Two operating fixes: shorter ventilation runtime in the multipurpose wing,
# and the café equipment on timers. Estimated savings in MWh per month:
fix = [2.0, 2.0, 2.1, 2.3, 2.4, 2.5, 2.4, 2.4, 2.4, 2.2, 2.0, 2.0]
after = [m - f for m, f in zip(meter, fix)]
print(f"after operating fixes: {sum(after) * 1000 / area:.0f} kWh/m2 (target 65)")

model 63 kWh/m2, metered 71 kWh/m2
NMBE 11.3% (limit +/-5%), CV(RMSE) 11.4% (limit 15%)
after operating fixes: 64 kWh/m2 (target 65)

The comparison shows a model that is biased low, by more than the 5% Guideline 14 allows for a calibrated model, while its monthly pattern follows the meters closely. In other words the model got the shape of the year right and the size of some loads wrong. Sethi recalibrates the model with the actual hours of use, and the calibrated model shows that the two operating fixes bring the building back close to its target. The exterior blinds at the commons, if reinstated, would cut the spring overheating and some cooling energy, and the city adds them to its capital plan.

From findings to action

An evaluation is only worth its cost if its findings lead to action. The report sorts them into three groups. Fixes for this building: reprogramming the multipurpose ventilation, timers on the café equipment, a door between the café and the study area, lower blind controls in the classrooms, and the exterior blinds. Lessons for the owner's next projects: protect acoustic separations and solar control from late changes and savings, write actual hours of use into the program, and fund evaluation in every project. Lessons for the design office: test the sensitivity of comfort to deleted shading, flag acoustic separations as protected items in the change log, and model plug and process loads from interviews with future users rather than from defaults.

Thibodeau ends the course where it began, with the three boxes: owner, designer, builder. Every delivery method, every contract and every procedure the students have studied is a way of managing the relationships between them so that a building gets built. Evaluation is the one step that adds a fourth box, the people who use the building, and draws a line back from them to the other three. 'If you remember one thing from this course,' he says, 'make it this: the job is not finished at the handover. It is finished when you know whether the building works.'

Exercise 12: a one-year evaluation

• Prepare an evaluation plan for the Rail Yard Commons with goals, methods, schedule, ethics and the people involved.
• Run the survey code with the instructor's responses and identify the topics to investigate.
• Compare the metered and modelled energy, calculate the calibration measures and explain the gap.
• Write five findings, each with evidence, cause, fix and lesson, and present them to the class as you would to the owner.

Self-check

• Have I combined surveys, interviews, measurements, meters and observation?
• Are participants' consent and anonymity protected?
• Does each finding link evidence to a cause and an action?
• Will the lessons reach the owner's next brief and my office's standards?