Research methods

In week four Leclerc brings in a guest: Dr. Selin Arslan, who taught the theory course in second year and now runs the school's research methods seminar. Arslan begins with a warning. 'Architects are used to deciding things,' she says. 'Researchers are used to finding things out. In a thesis you must do both, and you must not let the first contaminate the second. If you have already decided what the residents want, do not pretend to ask them.' The lesson is about choosing methods that suit the question, using them carefully, and doing it ethically when people are involved.

A map of methods

Arslan draws a matrix of the methods architects use in research, sorted by the kind of evidence they produce. Quantitative methods measure and count: surveys with closed questions, measurements of temperature, light or sound, energy data, counts of people using a space, and analysis of large datasets. Qualitative methods describe and interpret: interviews, focus groups, observation, walking interviews in which the researcher and a participant walk a route together, and ethnography, in which the researcher spends long periods in a community. Archival and historical methods read documents, drawings, photographs and maps. Simulation and modelling predict how a building or a place will behave. And design research uses drawings, models and prototypes as instruments of inquiry, the subject of a later lesson.

The four theses need different mixes. Ingrid's needs almost every row: simulation of how long buildings stay warm, a survey of households, interviews with residents and the people who run emergency shelters, night-time walking audits of the routes to the buildings, and design research to test the network. Callum's needs measurement, in the form of small bending tests of aspen specimens in the school's workshop, archival and published research on the strength of poplar species, interviews with mill operators and foresters, and life-cycle carbon analysis. Wren's needs archival research on past floods, analysis of river level records, interviews with the people who clean up riverside parks after floods, and design research. Joon-ho's needs a survey of buildings rather than people, archival drawings from the city's permit records, measurement of daylight in a real floor plate, and interviews with owners and city staff.

Arslan's main advice is to triangulate: to answer the same part of the question with two or three methods whose weaknesses differ. A survey can say how many residents would go to a warming room but not why; interviews can say why but not how many; a night walk can show whether the route is frightening, which neither may reveal. When three methods agree, a finding is strong. When they disagree, the disagreement is itself a finding.

Surveys and how many to ask

Ingrid plans a household survey in the district. It will ask whether anyone in the household relies on powered medical equipment, whether they have somewhere to go in an outage, how far they could walk in deep cold, and which buildings in the district they know and trust. Arslan reminds her that a survey is only as good as its sample: who answers decides what it says. A survey left on a café counter reaches people who go to cafés. She suggests a mix of door-to-door visits with a volunteer from the community organization, an online version, and paper copies at the hub, the school and the church, with the questions in the district's main languages.

How many completed surveys does Ingrid need? It depends on how precise she wants the answers to be. For a yes-or-no question, the margin of error at a given confidence depends mainly on the number of answers, not the size of the district, though a small population helps slightly.

# Survey sample size for a district of 6400 households (95% confidence)
import math
N = 6400
z = 1.96
p = 0.5                      # most cautious assumption about the answers
for e in (0.10, 0.07, 0.05):
    n0 = z * z * p * (1 - p) / (e * e)
    n = n0 / (1 + (n0 - 1) / N)   # finite population correction
    print(f"margin of error {e:.0%}: about {math.ceil(n)} completed surveys")
rate = 0.25                   # expected response rate for a door-to-door and online survey
print(f"at a {rate:.0%} response rate, about {math.ceil(math.ceil(n) / rate)} households to approach")

margin of error 10%: about 95 completed surveys
margin of error 7%: about 191 completed surveys
margin of error 5%: about 363 completed surveys
at a 25% response rate, about 1452 households to approach

About 95 answers give a margin of error of about 10 points either way; 363 would give 5 points but require approaching about 1,450 households, which is beyond what Ingrid and one volunteer can do in a term. She settles on a target of about 200 completed surveys, a margin of about 7 points, and decides to use interviews to explore the questions where she needs depth rather than precision.

Interviews, coding and agreement

Interviews are the method most architects use and fewest use well. Arslan's rules are simple. Prepare a guide of open questions, but follow the person rather than the guide. Ask about experience rather than opinion: 'tell me about the last time the power went out' rather than 'would you use a warming room?' Do not lead. Record with permission, transcribe, and read the transcripts several times before deciding what they say. Then code them: tag passages with short labels for the ideas they contain, refine the codes as you go, and count and compare.

Ingrid conducts twelve interviews in the first round and codes them with four codes: warmth, trust, access and cost. To check that her codes are not just her own impressions, she asks Callum to code the same twenty excerpts independently, and calculates how often they agree beyond what chance alone would produce. The usual statistic is Cohen's kappa, which is 0 for chance agreement and 1 for perfect agreement; values around 0.6 to 0.8 are usually taken as substantial agreement.

# Two coders tag the same 20 interview excerpts; how far do they agree beyond chance?
codes = ["warmth", "trust", "access", "cost"]
a = "warmth trust trust access warmth cost trust access warmth warmth trust cost access trust warmth access trust warmth cost trust".split()
b = "warmth trust access access warmth cost trust access trust warmth trust cost access trust warmth warmth trust warmth access trust".split()
n = len(a)
po = sum(x == y for x, y in zip(a, b)) / n
pe = sum((a.count(c) / n) * (b.count(c) / n) for c in codes)
kappa = (po - pe) / (1 - pe)
print(f"observed agreement {po:.2f}, chance agreement {pe:.2f}, kappa {kappa:.2f}")
for c in codes:
    print(f"{c:7s} coder A {a.count(c):2d}  coder B {b.count(c):2d}")
print("disagreements:", [(i + 1, x, y) for i, (x, y) in enumerate(zip(a, b)) if x != y])

observed agreement 0.80, chance agreement 0.28, kappa 0.72
warmth  coder A  6  coder B  6
trust   coder A  7  coder B  7
access  coder A  4  coder B  5
cost    coder A  3  coder B  2
disagreements: [(3, 'trust', 'access'), (9, 'warmth', 'trust'), (16, 'access', 'warmth'), (19, 'cost', 'access')]

The kappa of 0.72 is substantial. More useful than the number are the four disagreements, which Ingrid and Callum discuss. Two involve passages about whether a resident would bring her grandchildren to a shelter: Ingrid coded them as trust, Callum as access. They agree that the code 'access' is doing two jobs, physical access and permission to bring family and pets, and split it into two codes. The coding has done its work: it has made her look harder at what people said.

Research with people: ethics

Any research that involves people, their words, their data or their communities needs ethics review before it starts. Arslan explains the system that applies here. In Canada, research at institutions funded by the three federal research councils follows the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans, known as TCPS 2, whose current edition is from 2022. It rests on three core principles, respect for persons, concern for welfare and justice, and requires review by the institution's research ethics board before research with people begins. Its Chapter 9 sets out how research involving First Nations, Inuit and Métis peoples must engage the communities concerned, respect their governance and share benefits, and many First Nations follow the OCAP principles of ownership, control, access and possession of their own data.

For Ingrid, the practical consequences are several. She must submit her survey, interview guide and consent forms for review, and wait for approval before contacting anyone; the review takes three to six weeks, which is why she wrote ethics into her backward schedule. She must explain to participants what the research is for, what will happen to their words and that they may withdraw. She must protect identities: residents who rely on medical equipment are vulnerable, and a map of their homes would be dangerous if it were lost. And because some of the district's residents are Indigenous, and the community organization that runs the hub's food program is Indigenous-led, she must engage that organization before the survey goes out, not after. Morningstar, her second reader, insists on this. 'Come and ask what they want to know, and whether they want this at all,' she says. 'If they say no, that is an answer too.'

Ingrid meets the organization's director and Margaret Flett at the hub. They agree to help with the survey on two conditions: that the results come back to the community first, in a form residents can use, and that the organization keeps a copy of the data about its own members. Ingrid writes both conditions into her ethics application. The meeting changes one question, too: Flett points out that 'which buildings do you trust?' will be answered differently by people who remember institutions that did harm, and suggests asking instead where people already go when they need help.

Simulation and measurement as methods

Simulation is a research method only if it is treated like one: with stated assumptions, inputs that can be checked, a sense of uncertainty and, where possible, validation against measurement. Ingrid's survivability simulation will use the methods she learned in the simulation course, a resistance-capacitance model of each building and an hourly calculation through a cold outage. Abernathy asks her to calibrate it with measurement: a temperature logger in the school gym through a weekend when the heating is turned down, which the school's facilities manager agrees to allow. Callum's bending tests are measurement in the strict sense, and Szymańska makes him write a test protocol before he cuts a single specimen, stating the specimen size, the moisture content, the loading rate and how many specimens he needs for the results to mean anything.

Exercise 4: a methods plan

• List each part of your question and the methods that will answer it, with what each method cannot tell you.
• Plan to triangulate at least one important finding.
• If people are involved, draft your consent form and identify the review your research needs.
• If a community is involved, meet them before you design the research, and write down what they ask of you.

Self-check

• Have you designed the research to find out, or to confirm what you have already decided?
• Is your sample likely to include the people your thesis is about?