Service level management II: measuring, reporting and reviewing

An agreement is only the start. Service level management earns its value in the months that follow: collecting the right data, reporting it honestly, talking with customers about what the numbers mean and acting when performance or needs change. This lesson covers the sources of service level information, the watermelon effect, how to build a service level report that customers trust, how service credits work, how to run a service review and how agreements evolve. ITIL-140 lesson 11 covered percentile reporting for support targets and experience level agreements; this lesson builds on that foundation and widens it to the whole service.

Sources of service level information

ITIL 4 describes several sources of information that service level management draws on. Customer engagement provides first-hand insight: initial conversations, regular service reviews and informal listening. Customer feedback comes through surveys, both event-based ones sent after a ticket or transaction and periodic ones that ask about the service overall. Key business-related metrics come from the business itself, such as orders processed, shipments booked or patients seen. Operational metrics come from the provider's tools: availability, response times, incident counts, change failures. Each source tells part of the story; none tells all of it.

The watermelon effect

A watermelon SLA is green on the outside and red on the inside. Every reported target is met, but customers are unhappy. The term describes one of the most common failures in service level management, and its causes are usually easy to spot once you look for them. The targets may measure components rather than outcomes; they may average away bad periods; they may exclude the incidents that hurt most; they may cover only the provider's part of a chain that includes suppliers; or the customer's needs may have moved on since the targets were set.

Consider Larchmont. In the first quarter the LTS report shows every target green: portal availability above 99.9 %, priority 2 incidents resolved within target 93 % of the time and request fulfillment within target 91 % of the time. Yet the dispatch managers complain loudly, and the quarterly survey shows satisfaction with the portal falling from 78 % to 62 %. When the service level manager digs into the data, three things emerge. Availability is measured by a ping to the web server, which stayed up while searches timed out. The slowest searches clustered between 07:00 and 09:00, when dispatchers plan the day, but the monthly average hid them. And the incidents that caused the most frustration were logged as priority 3 because only one team reported them at a time.

# The watermelon effect: monthly averages can hide the hours that matter
import random, statistics
random.seed(141)
# Simulated search response times (seconds) for 22 business days, 05:00-23:00
rows = []
for day in range(22):
    for hour in range(5, 23):
        base = 1.1
        if 7 <= hour < 9:           # morning planning peak
            base = 3.4
        for _ in range(40):          # 40 searches sampled per hour
            rows.append((hour, random.lognormvariate(0, 0.45) * base))

all_t = [t for _, t in rows]
peak = [t for h, t in rows if 7 <= h < 9]
target = 2.0
within = lambda xs: sum(1 for t in xs if t <= target) / len(xs) * 100
print(f"searches measured:          {len(all_t):,}")
print(f"monthly mean response:      {statistics.mean(all_t):.2f} s")
print(f"monthly median response:    {statistics.median(all_t):.2f} s")
print(f"within 2 s (whole month):   {within(all_t):.1f} %")
print(f"within 2 s (07:00-09:00):   {within(peak):.1f} %")
print(f"share of searches in peak:  {len(peak) / len(all_t) * 100:.1f} %")

searches measured:          15,840
monthly mean response:      1.50 s
monthly median response:    1.18 s
within 2 s (whole month):   81.9 %
within 2 s (07:00-09:00):   12.7 %
share of searches in peak:  11.1 %

The whole-month figure looks acceptable, but in the two hours that matter most to dispatchers, most searches miss the two-second target. Because the peak is only about a ninth of the searches, it barely moves the average. A report that showed the morning window separately, or that used a business-weighted indicator, would have made the problem visible months earlier.

Avoiding watermelons

Several practices help prevent watermelon reporting. Measure from the user's point of view, using synthetic transactions that perform real tasks and real user monitoring that records actual page loads. Report distributions and critical windows, not just averages. Include customer satisfaction and business metrics alongside operational ones, so a gap between them is visible. Review incidents by business impact, not just by recorded priority. Cover the whole chain, including supplier components. And above all, talk to customers regularly: a five-minute conversation can reveal what months of green reports conceal.

Building a trustworthy service level report

A good service level report is short, honest and useful. It starts with a plain-language summary of how the service performed and what customers experienced. It shows performance against each agreed target, with trends over several periods rather than a single snapshot. It explains breaches and near misses, including causes and actions. It includes customer satisfaction and relevant business metrics. It lists improvement actions and their status. And it avoids the temptation to bury bad news in footnotes or exclusions.

Exclusions deserve special care. Most SLAs exclude some events from the calculation, such as planned maintenance in agreed windows or outages caused by the customer. Exclusions are legitimate when they were agreed in advance and applied consistently. They become a problem when the provider stretches them, for example by labeling an emergency fix as planned maintenance. A trustworthy report shows both the figure with exclusions and the raw figure, so customers can see what was excluded.

Service credits and other consequences

External SLAs often include service credits: a reduction in charges when service levels are missed. Credits are usually tiered, for example 5 % of the monthly fee if availability falls below 99.9 %, 10 % below 99.5 % and 25 % below 99 %. Credits give the customer some compensation and give the provider an incentive, but they rarely cover the customer's real losses. A customer that loses a day of freight bookings will not be made whole by a 10 % discount on hosting fees. For that reason, credits should be seen as a signal and incentive, not as insurance.

Service credits can also distort behavior. A provider may focus on avoiding the credit threshold rather than on delivering a good service, or may argue over exclusions to avoid paying. Some organizations prefer earn-back arrangements, where a provider can recover credits by sustained good performance, or service improvement plans with agreed actions, which focus attention on fixing the cause.

# Tiered service credits from monthly availability (illustrative contract terms)
tiers = [(99.9, 0.00), (99.5, 0.05), (99.0, 0.10), (0.0, 0.25)]   # (lower bound %, credit share)
monthly_fee = 18_000.00
months = {"Jan": 99.95, "Feb": 99.87, "Mar": 99.41, "Apr": 99.92, "May": 98.70, "Jun": 99.90}

def credit_share(avail):
    for bound, share in tiers:
        if avail >= bound:
            return share
    return tiers[-1][1]

total = 0.0
for m, a in months.items():
    c = credit_share(a) * monthly_fee
    total += c
    print(f"{m}: availability {a:6.2f} %  credit {credit_share(a) * 100:4.0f} %  = {c:9,.2f}")
print(f"credits for half-year: {total:,.2f} of {monthly_fee * len(months):,.2f} in fees")

Jan: availability  99.95 %  credit    0 %  =      0.00
Feb: availability  99.87 %  credit    5 %  =    900.00
Mar: availability  99.41 %  credit   10 %  =  1,800.00
Apr: availability  99.92 %  credit    0 %  =      0.00
May: availability  98.70 %  credit   25 %  =  4,500.00
Jun: availability  99.90 %  credit    0 %  =      0.00
credits for half-year: 7,200.00 of 108,000.00 in fees

Note that June, at exactly 99.90 %, earns no credit because the tier is defined as availability at or above 99.9 %. Precise wording about boundaries, rounding and measurement periods prevents disputes. A good SLA states whether availability is rounded before or after comparison with the target, which time zone defines the month and whose monitoring data is authoritative.

Running a service review

A service review is a regular meeting between the provider and the customer to discuss service performance, issues and changing needs. Reviews might be monthly for critical services and quarterly for others. A good review is a conversation, not a presentation. The provider shares the report in advance so time in the meeting goes on discussion. The agenda typically covers performance against targets, the customer's experience in their own words, significant incidents and problems, upcoming changes and business events, improvement actions and any need to change the agreement.

Reviews also build relationships. Customers who feel heard are more forgiving of the occasional failure and more willing to share early warnings about business changes, such as a new distribution center or a seasonal peak. Lesson 8 on relationship management returns to this.

Service improvement plans

When a service consistently misses targets or customer satisfaction is falling, a service improvement plan (SIP) sets out agreed actions to restore performance. A SIP names the problem, the evidence, the root causes, the actions with owners and dates, and the measures that will show whether the plan worked. SIPs are entered in the continual improvement register and reviewed at each service review until they are closed. For Larchmont, the portal SIP included adding synthetic search transactions to monitoring, reporting the 07:00 to 09:00 window separately, tuning the search index and adding a second application server.

Reviewing and revising agreements

Agreements age. Business processes change, new services appear, user expectations rise and technology makes new levels possible. Each SLA should state when it will be reviewed, usually at least annually, and what triggers an earlier review, such as a significant business change or repeated breaches. During review, the service level manager checks whether each target still reflects business needs, whether it is still measured properly, whether OLAs and underpinning contracts still support it and whether the customer still values it. Targets that nobody cares about should be removed, because every target costs effort to measure and report.

Working with other practices

Service level management depends on many other practices. Monitoring and event management and measurement and reporting supply the data. Incident and problem management explain breaches. Change enablement and release management affect availability and need to respect agreed windows. Supplier management ensures underpinning contracts support SLAs. Relationship management keeps the wider relationship healthy. Continual improvement turns review findings into action. Service level managers often act as integrators, bringing information from all these practices together in a form customers can use.

Common metrics for the practice itself

The service level management practice can measure its own performance. Useful measures include the share of services with current, reviewed agreements; the share of targets measured end to end; the number of breaches and near misses and the share explained with actions; customer satisfaction with service reviews; the gap between operational metrics and satisfaction, which signals watermelons; and the time taken to respond to customer requests for changes to agreements.