Deployment, serving and monitoring

A model that exists only in a notebook helps no one. The capstone requires a deployed demonstration: a working system that a user or the partner can try, with the model packaged, served behind an interface, monitored and documented well enough for someone else to run. Deployment is also where many hidden problems appear, such as differences between training and serving data, latency that is too high, inputs the model has never seen, and performance that degrades over time. This lesson covers deployment patterns, packaging and serving, interfaces and integration, testing before release, monitoring inputs, outputs and outcomes, detecting drift, and planning for maintenance and handover.

Deployment patterns

How a model is deployed depends on how its predictions are used:

• Batch prediction: the model scores a set of records on a schedule, for example nightly risk scores for all open invoices, and writes results to a database or report. Simple, robust and cheap.
• Online prediction: the model answers individual requests in real time through an API, for example classifying a support ticket as it arrives. Requires low latency and high availability.
• Streaming: the model processes continuous event streams, such as sensor readings or transactions, often with windowed features.
• Edge or on-device: the model runs on phones, cameras or embedded devices, for privacy, offline use or low latency, under tight memory and power constraints.
• Human-in-the-loop tools: the model's outputs appear as suggestions in an application where people review, correct and approve them, which also produces new labelled data.

For a capstone, the simplest pattern that meets the user's needs is usually best: a batch job or a small web application with an API is far easier to build and maintain than a streaming system.

Packaging and serving

Packaging makes a model runnable outside the development environment. Typical steps include saving the trained model and all preprocessing as a single artefact or pipeline; recording the exact library versions; wrapping inference in a function or class with a clear input and output schema; containerizing the application, for example with Docker, so that it runs the same way everywhere; and serving it through a web framework or a model-serving tool that exposes an HTTP API. LLM-based systems package prompts, retrieval indexes, model identifiers and configuration in the same way. Secrets, such as API keys and database passwords, must be kept out of code and repositories and supplied through environment variables or a secrets manager.

Training–serving skew

A common deployment failure is training–serving skew: the data the model receives in production differs from the data it was trained on, because preprocessing was implemented differently, features are computed from different sources or at different times, or inputs arrive in unexpected formats. Preventing skew means reusing exactly the same preprocessing code for training and serving, validating inputs against the expected schema and ranges, logging inputs in production and comparing their distributions with training data, and testing the deployed service with real examples from the validation set to confirm that it reproduces offline predictions.

Testing before release

Before the demonstration is shared, test it like software. Unit tests check preprocessing, feature and post-processing functions. Integration tests send realistic requests to the running service and check responses, including malformed, empty, very large and out-of-scope inputs. Regression tests confirm that the deployed model reproduces the evaluated model's predictions on a fixed set of examples. Load tests measure latency and throughput under expected and peak traffic. Security checks cover authentication, input validation and the handling of personal data. For LLM systems, the evaluation set from Lesson 7 runs against the deployed configuration. A short release checklist, signed off by a team member who did not build the component, catches many problems.

Monitoring

Once deployed, a model must be monitored, because the world changes and models do not adapt by themselves. Monitoring covers several layers:

• System health: uptime, error rates, latency percentiles and resource use.
• Input data: missing values, out-of-range values, schema violations and changes in feature distributions.
• Predictions: the distribution of scores and predicted classes, abstention rates and confidence.
• Outcomes: actual performance, measured when ground truth becomes available, often with a delay, such as fraud confirmed weeks later.
• User feedback: corrections, complaints, ratings and overrides.

Each signal should have a threshold that triggers an alert and a named person or procedure for responding.

Detecting drift

Drift is change over time in the data or in the relationship between inputs and outcomes. Data drift (covariate shift) is a change in the input distribution, such as new suppliers, seasons or customer types. Concept drift is a change in the relationship between inputs and the target, such as fraudsters adapting their tactics, so that the same inputs now mean something different. Data drift can be detected immediately from inputs; concept drift appears only when outcomes are known. A common measure of distribution change for a feature is the population stability index (PSI), which compares the share of observations in each bin of a reference distribution with the current distribution:

where \(r_i\) and \(c_i\) are the reference and current proportions in bin \(i\). A common rule of thumb treats PSI below 0.1 as stable, 0.1 to 0.25 as worth watching, and above 0.25 as a significant shift.

import numpy as np
# Monitoring a deployed model: population stability index for input drift
rng = np.random.default_rng(7)
reference = rng.normal(50, 10, 20000)
def psi(ref, cur, bins=10):
    edges = np.quantile(ref, np.linspace(0, 1, bins + 1)); edges[0], edges[-1] = -np.inf, np.inf
    r = np.histogram(ref, edges)[0] / len(ref); c = np.histogram(cur, edges)[0] / len(cur)
    r, c = np.clip(r, 1e-6, None), np.clip(c, 1e-6, None)
    return np.sum((c - r) * np.log(c / r))
for week, (mu, sd) in enumerate([(50, 10), (51, 10), (53, 11), (58, 13)], 1):
    cur = rng.normal(mu, sd, 3000); v = psi(reference, cur)
    flag = "stable" if v < 0.1 else ("watch" if v < 0.25 else "investigate")
    print(f"week {week}: PSI {v:.3f}  {flag}")
# Latency budget check for the serving path
p50, p95 = np.percentile(rng.gamma(4, 30, 10000), [50, 95])
print(f"simulated latency p50 {p50:.0f} ms, p95 {p95:.0f} ms against a 250 ms target")

week 1: PSI 0.003  stable
week 2: PSI 0.014  stable
week 3: PSI 0.101  watch
week 4: PSI 0.493  investigate
simulated latency p50 111 ms, p95 231 ms against a 250 ms target

The simulated feature is stable for two weeks, shows a small shift in week 3 that crosses the "watch" threshold, and a large shift in week 4 that should trigger investigation. Drift does not automatically mean the model has become worse, but it means its validation results may no longer apply, and outcome performance on recent data should be checked. The latency check shows the 95th percentile within the 250 ms target, with little margin.

Retraining and maintenance

Models need maintenance. Retraining strategies include scheduled retraining at fixed intervals, triggered retraining when drift or performance alerts fire, and continual learning with care to avoid feedback loops, where the model's own decisions shape the data it learns from. Every retrained model should pass the same evaluation and safety checks as the original before replacing it, and deployment should allow rollback to the previous version. Shadow deployment, where a new model runs alongside the current one without affecting users, and canary releases, where it serves a small share of traffic first, reduce the risk of updates.

Handover

At the end of the capstone, the partner may want to continue using the system. A responsible handover includes the code repository with documentation, the model and data artefacts or instructions to recreate them, deployment and operations instructions, the monitoring plan and current dashboards, the model card and risk register, a list of known issues and recommended next steps, and a clear statement of the system's maturity: a capstone prototype is not production software, and the partner should know what would be needed to operate it safely at scale.

Security basics for deployed demos

Even a demonstration system can expose data or be misused if it is reachable from the internet. Basic precautions include requiring authentication for anything beyond public demo data; validating and size-limiting inputs; rate-limiting requests to control cost and abuse; keeping dependencies updated; running services with minimal permissions; avoiding logging of sensitive inputs, or masking them; and shutting down or restricting the demo after the term. For LLM systems, treat retrieved documents and user inputs as untrusted text that may contain instructions, and never let model output trigger consequential actions without validation.

Choosing where to host

Capstone teams typically host demonstrations on program-provided infrastructure, a cloud provider using student credits, or a free tier of an application hosting service. Consider cost, data location requirements in the partner agreement, availability during the review period, and who will own and pay for the system afterwards. Document the hosting setup so that the partner can move it if they continue the project.

User interface design for AI systems

The interface determines how people use a system's outputs. Good AI interfaces show outputs in the context of the user's task; indicate confidence or uncertainty where it helps decisions; show sources or reasons where users need to check outputs; make it easy to correct, override or reject a suggestion; avoid presenting uncertain outputs as definitive; and collect feedback on errors. For capstone demonstrations, a simple, clear interface that reflects how real users would work is more convincing than a polished but unrealistic one. Test the interface with a few prospective users and observe where they hesitate or misunderstand.

Logging for improvement

Production logs, handled with appropriate privacy safeguards, are the raw material for improving the system. Logging inputs, outputs, confidence, latency, user corrections and outcomes allows the team to find new failure modes, build fresh evaluation sets from real usage, and collect labelled data for retraining. Decide in advance what will be logged, how long it will be kept and who may access it, consistent with the data agreement and privacy law.

Deployment work is often underestimated, so plan it early. Many teams find it useful to deploy a trivial version of the service, even with a baseline model, in the first half of the term, so that packaging, hosting, monitoring and the user interface are ready long before the final model is chosen.