Lean experimentation and the minimum viable product

Interviews told Grace and Marta what owners and brands say and, in a few cases, what they would commit to. The next step is to watch what they actually do when offered something real. Dr. Marchetti opens the fourth session with a story from her own company. Before writing any scheduling software, she and her co-founders spent a month taking orders from three trucking firms by phone and building the schedules by hand in a spreadsheet each night. The customers thought they were using an early version of a product. The founders learned which features mattered, which did not, and that one of their three planned customer types would never pay. The month cost them almost nothing except sleep.

The minimum viable product

Eric Ries defined the minimum viable product, or MVP, as the version of a new product that allows a team to collect the maximum amount of validated learning about customers with the least effort. The definition is often misread. An MVP is not a cheaper or buggier version of the final product, and it is not necessarily a product at all. It is an experiment designed to test a specific hypothesis, and the right MVP is the cheapest thing that will produce credible evidence about that hypothesis. For some ventures it is a landing page; for others, a service delivered by hand; for a few, especially in hardware or medicine, a working prototype is unavoidable.

Ries distinguished two kinds of leap-of-faith assumptions. The value hypothesis concerns whether customers find the product valuable once they use it: do they pay, come back and recommend it? The growth hypothesis concerns how new customers will find the product: through referrals, paid advertising, partners or sales. Both must be true for a venture to work, and founders often test the value hypothesis while assuming the growth hypothesis, only to find later that customers love the product but cost too much to acquire. Second Ascent's value hypothesis is that owners and brands will pay enough for fast, skilled repairs to cover the cost of providing them; its growth hypothesis is that customers can be reached cheaply through clubs, shops, Cedarline referrals and word of mouth.

Build, measure, learn

Ries described the work of a startup as a loop: build something, measure how customers respond, learn whether the hypothesis holds, and decide whether to persevere or pivot. Although the loop is drawn in that order, it is planned in reverse. Founders first decide what they need to learn, then what they must measure to learn it, and only then what they need to build. A team that starts by building tends to build too much, measure what is easy rather than what matters, and learn little. The speed of the loop is the main advantage a startup has over an established company, which usually has more resources but cannot change direction quickly.

Kinds of MVP

Several kinds of MVP are common. A smoke test describes a product that does not yet exist, usually on a landing page, and measures how many visitors take a costly action such as signing up, booking or paying a deposit. Drew Houston of Dropbox famously posted a short video demonstrating file synchronization before the product was ready, and the waiting list grew by tens of thousands of people almost overnight, evidence that the problem was widely felt. A concierge MVP delivers the service by hand to a few customers, with the founders doing everything personally; it tests the value hypothesis in depth and teaches the founders how the service should work. A Wizard of Oz MVP looks automated to the customer while people do the work behind the scenes. Nick Swinmurn, the founder of Zappos, began by photographing shoes in local stores, listing them online and buying them at full price when an order came in, which tested whether people would buy shoes online without building inventory or a warehouse.

Second Ascent's experiments

Grace and Marta designed three experiments to run over eight weeks. The first was a smoke test. They built a one-page website describing multi-brand repair with a ten-dollar deposit to book a drop-off slot, and drove traffic through two outdoor club newsletters, a post on Cedarline's social media accounts and a small search advertising budget. The second was a concierge MVP: Saturday repair days in Cedarline's workshop, where Marta and one paid technician repaired whatever came in and recorded the time and materials for every job. The third was the pair of paid test batches for the two brands that had agreed during interviews.

Each experiment had a written hypothesis and a threshold for success, agreed before it started. For the smoke test, the hypothesis was that at least 2 percent of visitors from targeted channels would pay a deposit. For the concierge MVP, it was that average labor per repair would be under one hour and that at least half of customers would rate the service as excellent and say they would use it again. For the brand batches, it was that the founders could complete twenty items at a cost below the fee each brand had agreed to pay, with a turnaround of under ten working days. Writing the thresholds down in advance, Dr. Marchetti says, is what turns an activity into an experiment.

Measuring the right things

Ries warned against vanity metrics, numbers that look good but do not inform decisions, such as total page views, registered users or social media followers. They usually rise over time regardless of whether the product is working, and they cannot be tied to a specific change. Actionable metrics link cause and effect: the conversion rate of a specific channel, the share of customers who return within a defined period, or the cost of acquiring a paying customer. A useful test of a metric is whether a change in it would change what the founders do. If not, it is probably a vanity metric.

Cohort analysis is the standard way to make metrics actionable. Instead of tracking a running total of customers, founders group customers by the week or month they first bought and follow each group over time. If later cohorts return at higher rates than earlier ones, the product is improving; if they return at lower rates, something is wrong, even if the total customer count is rising. Second Ascent's experiments were too short for much cohort analysis, but the founders set up their records so that every customer's first visit, channel and later visits could be tracked from the start.

Results

The smoke test drew 4,180 visitors in six weeks, of whom 131 paid a deposit, a conversion rate of 3.1 percent, comfortably above the 2 percent threshold. Conversion varied widely by channel: 6.4 percent from the outdoor club newsletters, 3.8 percent from Cedarline's accounts and only 1.2 percent from search advertising, where the cost per deposit was about 41 dollars. The Saturday repair days handled 212 items in eight weeks. Average labor was 0.92 hours per repair, but the distribution was wide: zipper replacements averaged 0.6 hours, while delamination repairs on waterproof jackets took more than two. Customer ratings were high, with 71 percent rating the service as excellent, and 64 percent of first-time customers brought a second item within the eight weeks.

The brand batches were more sobering. Marta completed the forty items in an average of eight working days, inside the target, but the cost per item was higher than expected because the brands sent more complex repairs than their interview descriptions suggested. For one brand the cost was about 38 dollars per item against an agreed fee of 45; for the other, about 49 dollars against a fee of 42. Both brands were pleased with the work. Grace concluded that brand contracts needed tiered pricing by repair type rather than a single fee, and that the venture needed better data on repair mix before signing any long contract.

Testing price

Price is one of the most important hypotheses to test with behavior rather than statements, and one of the easiest to test online. During the last three weeks of the smoke test, Grace and Marta showed half of visitors a standard zipper repair price of 49 dollars and the other half 59 dollars, assigning visitors at random. The question was not which price produced more bookings, since a lower price almost always does, but which produced more contribution per visitor once the cost of the repair was subtracted.

# Second Ascent price test: two prices shown at random to landing-page visitors
from math import sqrt
from statistics import NormalDist

def two_prop_z(k1, n1, k2, n2):
    p1, p2 = k1 / n1, k2 / n2
    p = (k1 + k2) / (n1 + n2)
    se = sqrt(p * (1 - p) * (1 / n1 + 1 / n2))
    z = (p1 - p2) / se
    return p1, p2, z, 2 * (1 - NormalDist().cdf(abs(z)))

cost = 0.6 * 28 + 6.0 + 2.5          # zipper job: technician hours x wage + parts + payment and packaging
arms = {49: (52, 1240), 59: (43, 1262)}   # price: (bookings, visitors)
(k1, n1), (k2, n2) = arms[49], arms[59]
p1, p2, z, pval = two_prop_z(k1, n1, k2, n2)
print(f"Booking rate at 49: {p1:.2%}; at 59: {p2:.2%}; z = {z:.2f}, p = {pval:.2f}")
for price, (k, n) in arms.items():
    print(f"Price {price}: contribution per booking {price - cost:.2f}, per 1,000 visitors {(price - cost) * k / n * 1000:,.0f}")

# How many visitors per arm to detect a drop from 4.2% to 3.4% with 80% power at 5% significance?
pa, pb = 0.042, 0.034
za, zb = NormalDist().inv_cdf(0.975), NormalDist().inv_cdf(0.80)
pbar = (pa + pb) / 2
n = ((za * sqrt(2 * pbar * (1 - pbar)) + zb * sqrt(pa * (1 - pa) + pb * (1 - pb))) / (pa - pb)) ** 2
print(f"Visitors needed per arm: {n:,.0f}")

Booking rate at 49: 4.19%; at 59: 3.41%; z = 1.03, p = 0.30
Price 49: contribution per booking 23.70, per 1,000 visitors 994
Price 59: contribution per booking 33.70, per 1,000 visitors 1,148
Visitors needed per arm: 8,965

The booking rate was lower at 59 dollars, as expected, but the difference was not statistically significant: with about 1,250 visitors in each arm, a gap of less than a percentage point could easily arise by chance. Contribution per visitor, however, was higher at 59 dollars, because each booking earned about ten dollars more. The sample-size calculation shows why the founders should not over-interpret the booking rates: to detect reliably a drop from 4.2 to 3.4 percent, they would need about 9,000 visitors per arm, roughly four times the traffic of the whole smoke test. Dr. Marchetti says this is typical. Early ventures rarely have enough traffic for precise tests, so they should test large differences, look for direction rather than precision, and combine online tests with what they learn by hand.

Persevere or pivot

At the end of each loop, the founders decide whether to persevere with the current strategy, pivot, or stop. Ries recommended regular pivot-or-persevere meetings, scheduled in advance, at which the team reviews the evidence against the thresholds it set. The danger he highlighted is the land of the living dead: a venture that is not failing clearly enough to stop and not succeeding clearly enough to raise money, so it drifts for years. Dr. Marchetti adds a second danger: escalating commitment, the tendency to invest more in a failing course of action because so much has already been invested. Agreeing in advance what evidence would justify a pivot protects founders from both.

For Second Ascent the evidence supported persevering on the core idea with three adjustments. Consumer demand was real and could be reached cheaply through clubs and Cedarline, but not through search advertising at current prices, so the growth plan would rely on partners and referrals. Labor per repair was close to the target on average but too variable, so pricing would be tiered by repair type, and the workshop would track every job type separately. Brand contracts were promising but needed tiered pricing and data on repair mix before any long commitment. Grace says the experiments cost about 11,000 dollars, a little over budget, and saved them from at least two expensive mistakes.

Experiments and ethics

Experiments with real customers raise ethical questions. A smoke test that takes deposits for a service that may never exist must refund them promptly and explain honestly what happened. A price test should not mislead customers about what they are getting, and in some jurisdictions consumer protection law limits how prices can be presented. Customer data collected during experiments must be handled under privacy law, which the cohort studied in MBA-512. Dr. Marchetti's rule is simple: if a customer would feel deceived on learning how the experiment worked, redesign it. In Second Ascent's case every deposit could be used against a repair on a Saturday repair day or refunded on request, and the landing page said that the service was in a pilot.