Technology investment: business cases, agile delivery and product management

Dr. Marchetti opens the tenth session with two famous failures. In October 2013 the US federal government launched HealthCare.gov, the website through which Americans in many states would buy health insurance under the Affordable Care Act. On its first day only a handful of people managed to enroll, and for weeks the site was slow, crashed or lost applications. A small team of engineers brought in from technology companies rebuilt key parts within a few months, using modern practices such as frequent releases and close monitoring. Reviews found that the project had been split among many contractors with no one in charge of the whole, had been tested end to end only days before launch, and had followed a plan written years earlier despite changing requirements. The failure was not mainly a software failure; it was a failure of how the investment was governed.

Large technology projects fail more often and more spectacularly than most investments. Bent Flyvbjerg and Alexander Budzier, studying more than a thousand IT projects, reported in a 2011 Harvard Business Review article that the average cost overrun was modest but that the distribution had a fat tail: about one project in six had a cost overrun of around 200 percent and a schedule overrun of almost 70 percent. Dr. Marchetti's point is that technology investment needs a discipline suited to that risk. This lesson covers how to build a business case that is honest about benefits and uncertainty, how to value flexibility, how to choose between building and buying, how modern delivery methods reduce risk and how funding software as products rather than projects changes what organizations get.

Building an honest business case

A business case sets out what an investment will cost, what it will deliver and why it is worth doing. For technology, the cost side is usually easier. It includes software licenses or subscriptions, implementation services, internal staff time, hardware or cloud costs, data migration, integration with other systems, training, and the cost of running and supporting the system for its life. The last items are often underestimated; the cohort's cloud comparison in lesson 8 showed that running costs over five years can exceed the initial investment.

Benefits are harder. John Ward and Elizabeth Daniel, in their work on benefits management, argue that business cases should identify each benefit, the changes in processes and behavior needed to achieve it, the person accountable for delivering it, and how it will be measured. Benefits can be financial, such as lower cost or higher margin; quantifiable but not easily priced, such as faster delivery or fewer errors; or observable only through judgment, such as better decisions. A sound case is explicit about which benefits are which and does not convert soft benefits into precise financial figures that nobody will check. Most important, it names a business owner for the benefits, not the technology team, because benefits come from changing how the business works.

A business case must also be honest about uncertainty. Optimism bias, the tendency to underestimate costs and overestimate benefits, is well documented in large projects. Flyvbjerg recommends reference class forecasting: comparing a project with the actual outcomes of similar past projects and adjusting the estimate accordingly. A firm planning a new order management system should ask how similar implementations at comparable retailers turned out, not only what the vendor's plan says.

Valuing flexibility: staged investment

Many technology investments are uncertain in a particular way: the firm does not know in advance whether the idea will work, but it can find out cheaply before committing fully. In such cases a staged investment, a pilot followed by a rollout only if the pilot succeeds, is worth more than its discounted cash flows suggest, because it includes the option to stop. The cohort met real options in the corporate finance course. Dr. Marchetti applies the idea to a personalization engine, software that would tailor Cedarline's website, emails and offers to each customer.

import math
# 1. Commit now or stage it? A personalization engine for Cedarline (millions of dollars)
rate = 0.10
def pv(flow, start, years):
    return sum(flow / (1 + rate) ** t for t in range(start, start + years))
p_success = 0.5                       # chance the pilot shows the model lifts sales enough
good, bad = 2.6, 0.4                  # annual net benefit for five years in each case
full_cost = 6.0
commit_now = p_success * pv(good, 1, 5) + (1 - p_success) * pv(bad, 1, 5) - full_cost
pilot_cost, rollout_cost = 0.8, 5.4   # pilot in year 0; rollout in year 1 only if the pilot succeeds
staged = (-pilot_cost + p_success * (pv(good, 2, 5) - rollout_cost / (1 + rate)))
print(f"commit now: expected NPV {commit_now:5.2f}")
print(f"pilot first: expected NPV {staged:5.2f}  (value of the option to stop: {staged - commit_now:4.2f})")

# 2. When will the team finish? Forecast from past sprint velocity (story points per two-week sprint)
velocity = [21, 26, 18, 24, 27, 23]
remaining = 190
ordered = sorted(velocity)
low, mid, high = ordered[1], sum(velocity) / len(velocity), ordered[-2]
for label, v in (("cautious", low), ("average", mid), ("optimistic", high)):
    sprints = math.ceil(remaining / v)
    print(f"{label:10s} {v:5.1f} points a sprint -> {sprints} sprints ({2 * sprints} weeks)")

commit now: expected NPV -0.31
pilot first: expected NPV  1.23  (value of the option to stop: 1.54)
cautious    21.0 points a sprint -> 10 sprints (20 weeks)
average     23.2 points a sprint -> 9 sprints (18 weeks)
optimistic  26.0 points a sprint -> 8 sprints (16 weeks)

Committing the full 6 million now has an expected net present value of minus 0.3 million: if the engine works it earns 2.6 million a year, but there is an even chance it adds only 0.4 million, and the average does not cover the cost. A pilot costing 0.8 million, followed by a 5.4 million rollout only if the pilot succeeds, has an expected value of about 1.2 million, even though the benefits arrive a year later. The difference, about 1.5 million, is the value of being able to stop. The general lesson is that when the outcome is uncertain and a cheap test can resolve much of the uncertainty, the test is often the most valuable part of the investment. It is also why the roadmap in lesson 12 includes the personalization engine only as a pilot.

Staging works only if the firm actually stops when the pilot fails. Pilots that are declared successes regardless of results, or that are never allowed to end because a team has been built around them, destroy the option's value. A good pilot defines in advance what result would justify the rollout, measures it with a comparison group and has a decision date.

Build, buy or partner

For any capability, a firm can build its own software, buy a packaged product, increasingly as software as a service, or partner with a firm that provides the capability as a service. The general rule, which lesson 2 introduced, is to buy or rent commodities and build only where the capability is part of the firm's advantage and no product fits. Buying is faster and cheaper to start, and the vendor spreads development costs across many customers. But packaged software imposes its own processes, and firms that customize it heavily to match their existing ways of working end up with expensive, fragile systems that are hard to upgrade. A useful discipline is to adopt the package's standard processes wherever the firm has no good reason to differ, and to customize only where the difference matters to customers.

For Cedarline, the cohort sorts the roadmap's capabilities. Order management, point of sale, warehouse management, email marketing and the core data platform are commodities and should be bought or rented. The Care app and the product-life record are close to the strategy and should be designed by Cedarline, built on standard cloud services and perhaps with a development partner, with Cedarline owning the data and the product decisions. The resale marketplace could start on a specialist resale platform that several apparel brands use, which would be faster, with an option to bring it in-house if it becomes large.

Vendor selection and contracts deserve care. Dr. Marchetti's checklist includes references from customers of similar size and complexity, a proof of concept using the firm's own data and processes rather than a scripted demonstration, the vendor's financial health and product roadmap, contract terms on data ownership, export, security, service levels and price increases, and an exit plan describing how the firm would move to another vendor.

From waterfall to agile

Traditional software projects followed a waterfall approach: gather all requirements, design the whole system, build it, test it and release it, with each phase finishing before the next begins. The approach suits work whose requirements are known and stable, but most business software is not like that. Users cannot specify in advance everything they will need, requirements change during long projects, and problems are discovered only when the pieces are put together, often too late to fix cheaply. HealthCare.gov and Phoenix both showed the pattern.

The Manifesto for Agile Software Development, written by seventeen software developers in 2001, set out a different philosophy: individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. Agile methods such as Scrum organize work in short cycles, often two weeks, called sprints. A small cross-functional team plans a set of work from a prioritized list, builds and tests it, shows it to users and adjusts the plan based on what it learns. Value arrives early and often, and problems surface quickly.

Agile is sometimes misunderstood as having no plan or no deadlines. Teams do forecast, but from evidence rather than wishful thinking. The second part of the code forecasts when Cedarline's Care app team will finish the 190 points of work remaining for its first release, using the team's velocity, the amount of work completed in each of its past six sprints. At its average pace of about 23 points a sprint the team needs nine sprints, or 18 weeks; at a cautious pace, ten sprints; at an optimistic pace, eight. Giving a range rather than a single date is more honest, and the forecast improves each sprint as the team learns more. Managers should resist the temptation to use velocity as a performance target, because teams asked to raise velocity simply estimate their work more generously.

Agile practices were designed for small teams, and scaling them to large programs requires coordination: shared priorities, regular integration of different teams' work, and architecture that lets teams work independently. Frameworks for scaled agile exist, and firms vary in how much of their structure they adopt. Dr. Marchetti's advice is to keep teams small and stable, give each a clear area of the product to own, and integrate and release frequently so that problems between teams appear early.

Product management and modern delivery

A further shift moves from funding projects to funding products. In the project model, the firm approves a budget for a defined scope, assembles a team, delivers and disbands the team, after which the system is maintained by a separate support group. In the product model, a stable team owns a product, such as the Care app or the order management capability, for its whole life, with a product manager responsible for its value and a budget that funds the team year to year. The team is judged on outcomes, such as repair turnaround times or app retention, rather than on delivering a fixed list of features. The model suits digital capabilities that need continual improvement, and it avoids the knowledge loss that occurs when project teams disperse.

Research by Nicole Forsgren, Jez Humble and Gene Kim, summarized in their 2018 book Accelerate, identified four measures of software delivery performance that predicted organizational performance: how often a team deploys changes, how long a change takes from commitment to production, what share of changes cause failures, and how quickly service is restored after a failure. High performers deploy often in small batches, with automated testing and deployment, and are both faster and more stable than low performers. The finding contradicts the intuition that speed and stability must be traded off: small, frequent changes are easier to test and easier to reverse.

Technical debt is a final concept managers should know. Ward Cunningham coined the term in 1992 to describe the future cost of taking shortcuts in software: quick fixes, missing tests and outdated components that make later changes slower and riskier, like interest on a loan. Some debt is a sensible trade-off to meet a deadline, but it must be repaid. Organizations that never allocate time to repaying it find that every change takes longer, until their systems become too risky to touch. Many high-performing teams reserve a fixed share of capacity, often a fifth or so, for reducing technical debt.

Managing the portfolio

Firms invest in many technology initiatives at once and must choose among them. A common approach classifies spending into running the business, which keeps existing systems operating; growing the business, which improves current capabilities; and transforming the business, which builds new ones. Most firms spend the large majority of their technology budget on running the business, and freeing money for growth and transformation often requires simplifying the legacy estate. A portfolio view also balances risk: F. Warren McFarlan argued in a 1981 article that the risk of IT projects depends on their size, the firm's experience with the technology and how well defined the project is, and that a portfolio should not be loaded with large, unfamiliar, poorly defined projects at the same time.