Automation and DevOps

Larchmont's baseline showed 45 production changes a month and a lead time of 74 days from approved idea to production. The board wants changes deployed on demand with a lead time under ten days. That cannot happen by asking the change advisory board to meet more often. It needs a different way of moving work from idea to customer, built on automation and on the culture and practices known as DevOps.

This lesson explains what DevOps is and is not, the main technical practices (continuous integration, continuous delivery, infrastructure as code and automated testing), how to measure delivery performance, how service management controls such as change enablement work inside automated pipelines, and how to choose what to automate in service operations as well as in software delivery. Lesson 5 continues with platforms, observability and AIOps.

What DevOps is

DevOps is a set of cultural values, practices and tools that bring software development and IT operations together so that organizations can deliver changes quickly, safely and reliably. The term emerged around 2009 from practitioners frustrated by the wall between developers, who were rewarded for change, and operations staff, who were rewarded for stability. DevOps argues that, done well, speed and stability reinforce each other: small, frequent, automated changes are easier to test, easier to understand and easier to undo than large, rare ones.

DevOps is often summarized with the acronym CALMS: culture (shared responsibility and trust between teams), automation (of builds, tests, deployments and infrastructure), lean (small batches, flow and waste reduction), measurement (data about delivery and operations) and sharing (of knowledge, tools and lessons). Another common summary is the 'three ways': optimize flow from development to operations to the customer, create fast feedback from operations back to development, and build a culture of continual learning and experimentation.

DevOps is not a job title, a tool or a team that sits between development and operations, although organizations often use those labels. Buying a pipeline product or renaming the operations team does not change outcomes if handoffs, queues and blame remain. ITIL and DevOps are compatible: ITIL describes what an organization needs to manage services well, and DevOps describes a way of doing much of that work quickly and safely.

The delivery pipeline

A deployment pipeline is the automated path a change takes from a developer's commit to production. Each stage builds confidence: the code compiles, unit tests pass, security scans find no serious flaws, integration and performance tests pass in a production-like environment, and the change is released to users, often progressively. If any stage fails, the pipeline stops and the team gets fast feedback.

The pipeline also generates evidence. Every run records who changed what, which tests ran, which checks passed and when the change reached production. This evidence is valuable for auditors and for incident investigation, and it is far more complete than a change form filled in by hand.

Measuring delivery performance

The DORA research program, started by the DevOps Research and Assessment group and continued at Google, has surveyed tens of thousands of technology professionals since 2014. It popularized four key measures of software delivery performance, often called the DORA metrics. Two measure throughput and two measure stability, and the research found that high performers do well on all four at once.

Later DORA reports refined the names and added measures, for example describing recovery as failed deployment recovery time and looking at rework, and they emphasize that the measures describe teams and systems, not individuals. Using them to rank or reward individual engineers encourages gaming. Used well, they show where flow is blocked and whether stability improves as speed increases.

# Compute DORA-style measures from a month of deployment records (invented)
from statistics import median
deploys = [
    # (day, lead_time_hours, failed, hours_to_restore_if_failed)
    (1, 30, False, 0), (2, 52, False, 0), (4, 20, True, 3.5), (5, 44, False, 0),
    (8, 26, False, 0), (9, 70, True, 9.0), (11, 18, False, 0), (12, 33, False, 0),
    (15, 22, False, 0), (16, 41, False, 0), (18, 15, True, 1.0), (19, 29, False, 0),
    (22, 24, False, 0), (23, 36, False, 0), (25, 19, False, 0), (26, 27, False, 0),
    (29, 21, False, 0), (30, 25, False, 0),
]
n = len(deploys)
failed = [d for d in deploys if d[2]]
print(f"deployments in 30 days: {n} (about {n / 30 * 7:.1f} per week)")
print(f"median lead time: {median(d[1] for d in deploys):.0f} hours")
print(f"change failure rate: {len(failed) / n * 100:.0f} %")
print(f"median time to restore after failed change: {median(d[3] for d in failed):.1f} hours")

deployments in 30 days: 18 (about 4.2 per week)
median lead time: 26 hours
change failure rate: 17 %
median time to restore after failed change: 3.5 hours

These figures describe Larchmont Connect's team three months into its new pipeline: about four deployments a week, a median lead time of a little over a day, a change failure rate of 17 % and a median recovery of 3.5 hours. Throughput has improved sharply; stability has not yet, which tells the team to invest in automated testing and progressive release before pushing frequency higher.

Change enablement in an automated world

ITIL's change enablement practice exists to maximize the number of successful changes by ensuring risks are properly assessed, changes are authorized and the change schedule is managed. Nothing in that purpose requires a weekly meeting. ITIL 4 already encouraged decentralizing change authority and treating low-risk, well-understood changes as standard changes that are pre-authorized.

In a DevOps pipeline, much of change control becomes automated. Risk is assessed by rules and data: the size of the change, the components touched, test results, the team's recent failure rate and the time of day. Low-risk changes that pass all checks are authorized automatically, with peer review of code providing a human check. Higher-risk changes, such as database schema changes during month-end, are routed for additional review. Change advisory boards, where they remain, focus on genuinely risky or cross-team changes and on reviewing patterns rather than individual forms.

Regulated US firms must show that automated controls are effective. Public companies' internal control over financial reporting, assessed under Sarbanes-Oxley section 404, covers change management for systems that affect financial data, and auditors look for segregation of duties, authorization, testing and traceability. Banking regulators and frameworks such as the FFIEC IT examination guidance raise similar points. Automated pipelines can meet these expectations if they enforce peer review, prevent people from approving their own changes, protect pipeline configuration and keep tamper-resistant records.

Policy as code and guardrails

Policy as code expresses rules such as 'all storage must be encrypted' or 'no public network access to databases' in a machine-readable form that tools check automatically, in the pipeline before deployment and continuously in production. It turns governance from a document people are asked to read into a guardrail that prevents violations or flags them immediately. Guardrails let teams move fast within safe boundaries instead of waiting for approvals.

Good guardrails are few, clear and explained. If a check blocks a deployment, the message should say what rule was broken, why it matters and how to fix it. Exceptions should be possible through a documented, time-limited process with an owner. Larchmont's security team started with nine rules covering encryption, public exposure, secrets in code, approved base images and logging, and added more only when incidents or audits showed a need.

Automation in service operations

DevOps focuses on delivery, but automation applies just as much to service operations: fulfilling requests, handling routine incidents and running repetitive operational tasks. Examples include password resets, access requests with automated approval and provisioning, new-starter setup, restarting a failed service when a known error recurs, scaling capacity when load rises and closing tickets when monitoring shows recovery. Automation reduces toil, the repetitive manual work that scales with load and adds no lasting value, and frees people for work that needs judgment.

Not everything should be automated. Good candidates are frequent, rule-based, stable and well understood, with clear inputs and outcomes. Poor candidates are rare, change often, need judgment or have unclear processes. Automating a broken process simply produces broken results faster, so the first step is often to simplify the process, following the ITIL guiding principle 'optimize and automate', in that order.

Avoiding automation pitfalls

Automation creates its own risks. Scripts written by one person become critical systems that nobody else understands. Automated actions can fail silently or cascade: a self-healing script that restarts a service every minute can hide a serious problem. Automated approvals can be gamed if rules are poorly designed. Treat automation as a product: put it in version control, test it, monitor it, document it, give it an owner and review it.

People are affected too. Staff whose work is automated may fear for their jobs, and losing hands-on work can erode the skills needed when automation fails. Larchmont committed publicly that automation in LTS would redeploy staff, not cut jobs, during the three-year plan, and it trained service desk analysts to build and maintain workflows. Lesson 11 returns to the people side of transformation.