Concept selection

Four concepts survived generation, each with real strengths. Choosing among them is where many design teams go wrong, not because they choose badly but because they choose for reasons they cannot explain. A selection made by enthusiasm, by the loudest voice or by whoever drew the nicest sketch is fragile: the first hard question at a review breaks it. A selection made with explicit criteria, honest scoring and a check of how sensitive the result is to the team's assumptions can be defended, revised when new information arrives, and trusted by the client.

This lesson covers the tools the TDX-150 team used in weeks five and six: screening with a Pugh matrix, scoring with a weighted decision matrix, testing the result with a sensitivity analysis, and recording the decision so it can be revisited.

Developing concepts to a comparable level

Concepts can only be compared fairly if each has been developed to roughly the same depth. A concept sketched in an afternoon will look better than one analyzed for a week, because its problems have not been found yet. Before any scoring, the team gave each concept the same treatment: a dimensioned sketch, a mass estimate from component data, a first-order hydraulic estimate of flow and head, a rough cost, and a list of the three biggest unknowns. Each estimate took about a day per concept.

Those quick estimates changed the picture. Concept C, the generator and electric submersible, needed a generator of about 40 kW to start the submersible motor, and two energy conversions, engine to generator and motor to pump, cost about 12 percent of the fuel energy compared with direct drive. Concept B's hydraulic system lost about 15 percent in its pump, hoses and motor, and the 30 m of hydraulic hose needed to reach the water weighed about 60 kg. Concept D's gasoline trash pumps were light and cheap but could not reach 18 m of head at the flows needed, and storing gasoline in volunteer fire halls raised concerns of its own.

Screening with a Pugh matrix

A Pugh matrix compares each concept with a reference, the datum, criterion by criterion, marking each as better (+), the same (S) or worse (−). The team used the rented 150 mm diesel unit the cooperative had relied on as its datum, because everyone knew it. The matrix is quick, it does not pretend to precision, and it is good at revealing where a concept could be improved by borrowing a feature from another.

Concept A is never worse than the datum and better in three ways, because it was designed from the start for lower mass and noise. Concept B's strengths are priming and debris, its weakness cost. Concept C is worse in four ways and was dropped from further development, though it stayed in the scoring for completeness. The Pugh matrix also prompted a useful hybrid idea: borrow concept B's advantage at depth by offering, as an accessory, a small hydraulic submersible that concept A's engine could drive for the rare jobs deeper than 7.5 m. The team logged the idea for the cooperative to consider but kept it out of the core project.

The weighted decision matrix

A weighted decision matrix scores each concept against each criterion on a numeric scale, multiplies by a weight reflecting the criterion's importance, and sums. The weights came directly from the house of quality's technical importance, normalized to sum to one, which ties the selection to the needs the client expressed rather than to the team's own preferences:

Scores ran from 1 (poor) to 5 (excellent). The team scored independently first, each member on their own sheet, and then compared. Where scores differed by more than one point, the members discussed the evidence and agreed a value. Independent scoring matters: if one person scores aloud, others tend to anchor on that number.

# Weighted decision matrix for four TDX-150 concepts, with a weight-sensitivity sweep
crit = {"flow at 18 m": 0.17, "wet mass": 0.17, "run time": 0.13, "unit cost": 0.13,
        "priming / start": 0.11, "noise": 0.10, "set-up time": 0.10, "debris handling": 0.09}
# scores 1 (poor) to 5 (excellent), agreed by the team after the concept review
S = {
 "A diesel, direct-drive, vacuum-primed": [5, 4, 4, 4, 4, 3, 4, 4],
 "B diesel power pack, hydraulic submersible": [4, 3, 4, 2, 5, 3, 3, 5],
 "C diesel generator, electric submersible": [4, 2, 3, 2, 5, 3, 2, 4],
 "D two gasoline trash pumps": [2, 5, 2, 5, 3, 2, 3, 3],
}
w = list(crit.values()); assert abs(sum(w) - 1) < 1e-9
def total(scores, weights): return sum(a * b for a, b in zip(scores, weights))
for name, sc in sorted(S.items(), key=lambda kv: -total(kv[1], w)):
    print(f"{name:45s} {total(sc, w):.2f}")

# Sensitivity: raise one weight by 50 %, renormalize the rest, and see whether the winner changes
names = list(crit)
changes = 0
for k in range(len(w)):
    w2 = w[:]; w2[k] *= 1.5
    s = sum(w2); w2 = [x / s for x in w2]
    win = max(S, key=lambda n: total(S[n], w2))
    if not win.startswith("A"):
        changes += 1; print("  winner changes when", names[k], "is raised:", win)
print(f"Winner unchanged in {len(w) - changes} of {len(w)} single-weight changes")

# Break-even: how heavy must 'debris handling' be before B overtakes A?
for f in [x / 10 for x in range(10, 61)]:
    w2 = w[:]; w2[7] *= f; s = sum(w2); w2 = [x / s for x in w2]
    if total(S["B diesel power pack, hydraulic submersible"], w2) > total(S["A diesel, direct-drive, vacuum-primed"], w2):
        print(f"B overtakes A when the debris weight is multiplied by {f:.1f} ({w2[7]:.0%} of the total)"); break
else:
    print("B never overtakes A by raising the debris weight alone")

# What if the cooperative needed lifts deeper than 7.5 m (deep basements)? Add that criterion.
deep = {"A": 1, "B": 5, "C": 5, "D": 1}
for wd in (0.10, 0.20, 0.30):
    w3 = [x * (1 - wd) for x in w] + [wd]
    res = {n[0]: total(S[n] + [deep[n[0]]], w3) for n in S}
    best = max(res, key=res.get)
    print(f"Deep-lift weight {wd:.0%}: " + ", ".join(f"{k} {v:.2f}" for k, v in res.items()) + f"  -> {best}")

A diesel, direct-drive, vacuum-primed         4.07
B diesel power pack, hydraulic submersible    3.57
D two gasoline trash pumps                    3.20
C diesel generator, electric submersible      3.08
Winner unchanged in 8 of 8 single-weight changes
B never overtakes A by raising the debris weight alone
Deep-lift weight 10%: A 3.76, B 3.71, C 3.27, D 2.98  -> A
Deep-lift weight 20%: A 3.46, B 3.86, C 3.46, D 2.76  -> B
Deep-lift weight 30%: A 3.15, B 4.00, C 3.66, D 2.54  -> B

Concept A scores 4.07, B 3.57, D 3.20 and C 3.08. A half-point lead on a five-point scale is meaningful but not overwhelming, so the next question is how robust the ranking is.

Sensitivity analysis

Weights and scores are judgments. A good selection checks whether reasonable changes in those judgments would change the answer. The program raises each weight in turn by 50 percent, renormalizes the others, and recomputes the winner. Concept A wins in all eight cases. It also searches for the weight on debris handling, concept B's strongest criterion, at which B would overtake A, and finds none: even a very heavy emphasis on debris cannot overcome A's advantages in mass, cost and set-up.

Then the team asked a different question. What if a criterion was missing? Dale had mentioned that a few member towns had deep basements and culverts where water sits more than 7.5 m below any place a skid can stand. That need was not in the requirements, because the incident log showed it rarely. The program adds a deep-lift criterion and gives it increasing weight. At 10 percent, A still wins narrowly; at 20 percent, B wins clearly. The figure shows the crossover.

That result changed the conversation with the client. The selection of concept A is correct only if deep lifts are rare enough that they deserve less than about 11 percent of the weight. Émilie took the plot to Dale and the board, who confirmed that deep jobs were a few percent of the work and could be handled by the accessory submersible or by a rental. The board's answer went in the decision log. If the cooperative later expanded into a district with many deep culverts, the log would show exactly which assumption had changed.

Beyond the matrix

A decision matrix is a tool for thinking, not a machine for deciding. Three cautions apply. First, scores on a 1 to 5 scale are ordinal judgments, and treating them as precise numbers can give a false sense of rigor; differences of a tenth of a point mean nothing. Second, criteria can overlap: if mass and transport ease are both criteria, mass is counted twice. The team avoided this by taking criteria from the house of quality characteristics, which were defined to be distinct. Third, a single fatal flaw can be hidden by good scores elsewhere. A concept that cannot meet a must-have requirement should be screened out before scoring, not allowed to compensate with high marks on other criteria.

Some teams use the analytic hierarchy process instead of direct weights, comparing criteria in pairs and deriving weights from a consistency-checked matrix. It is useful when there are many criteria and stakeholders disagree, because pairwise comparisons are easier to make than absolute weights. The TDX-150 team did not need it, since the house of quality already gave defensible weights.

Recording the decision

The decision record for concept selection fits on one page. It states the decision (develop concept A, with a deep-lift submersible as a possible accessory), the options considered, the criteria and weights with their source, the scores, the sensitivity results, the assumptions on which the decision depends (deep lifts below about 10 percent of the work; Tidewater's wet end available), who decided and when. Farah, reading it at the preliminary design review, asked only one question: why the welded steel skid rather than aluminum. Hannah answered that material selection was still open and would be decided at embodiment, which was true and is the subject of lesson 7.

Selection is not the end of the other concepts. Their best features often migrate into the chosen one. Concept B's lesson that priming is a weak point pushed the team to put more effort into the priming system; concept D's lesson that crews value light, familiar equipment pushed the team to simplify the controls so a volunteer who uses a trash pump at home would recognize them.