AI in practice: from idea to deployed system

Most of the effort in real AI projects goes not into inventing algorithms but into everything around them: choosing the right problem, collecting and cleaning data, deciding how to measure success, integrating the model into a product or process, and keeping it working after launch. Surveys of organizations repeatedly find that many AI projects never reach production or fail to deliver value, usually for practical rather than technical reasons. This lesson walks through the life cycle of an applied AI system, using examples from industry, health care and data center operations, and explains the decisions that separate useful systems from expensive experiments. It also covers the newer pattern of building applications on top of general-purpose AI models.

Start with the problem, not the technology

A good AI project starts with a clear problem and a decision or action that a better prediction would improve. Useful questions to ask:

• What decision will the system support or automate, and who makes it today?
• What does a mistake cost? Are false positives or false negatives worse?
• Is there enough relevant data, and is it legally and ethically usable?
• How will success be measured, in business or human terms, not just model accuracy?
• Would a simpler approach (a rule, a dashboard, a process change) solve most of the problem?

Many promising ideas fail these questions. A model that predicts equipment failure one hour ahead is useless if repairs take a day to arrange; a model with 95% accuracy is useless if 99% of cases are negative and a trivial rule already achieves 99% (Lesson 5).

The AI project life cycle

Applied AI projects follow a recognizable cycle, often iterated many times:

1. Problem framing: define the decision, users, constraints and success metrics.

2. Data collection: gather historical data and, where needed, labels; check consent, ownership and privacy.

3. Data preparation: clean errors, handle missing values, join sources, and split data into training, validation and test sets.

4. Modeling: start with a simple baseline, then try more complex models; tune and compare them on validation data.

5. Evaluation: test on held-out data that resembles real use, including subgroups and edge cases; involve domain experts.

6. Deployment: integrate the model into software and workflows, with user interfaces, alerts and fallbacks.

7. Monitoring and maintenance: track performance, data drift and incidents, and retrain or retire the model as needed.

Data: the largest share of the work

Practitioners often say that most of their time goes into data. Typical problems include missing values, inconsistent formats, duplicate records, sensor errors, labels that are wrong or inconsistent between annotators, and data that does not represent the people or situations the system will face. Data must also be split carefully: if information about the test examples leaks into training (for example, records of the same patient in both sets, or features that are only known after the outcome), the model will look much better in testing than in reality. This data leakage is one of the most common causes of disappointing deployments.

Evaluation that reflects reality

Good evaluation goes beyond a single accuracy number. Teams compare models against a simple baseline (such as predicting the most common class or using last year's value), report metrics that match the costs of errors (precision, recall, false alarm rate), check performance for different groups and conditions, and test on data from a later time period than the training data, because the future often differs from the past. For high-stakes uses, prospective trials (testing the system on new cases as they occur, alongside existing practice) give the strongest evidence. In medicine, regulators require such evidence before approving AI software as a medical device.

Deployment and people

A model only creates value when it changes what people or systems do. Deployment decisions include whether the model acts automatically or advises a person, how its output is presented, what happens when it is uncertain or unavailable, and how users can give feedback or override it. Human factors matter: people may over-trust a system that is usually right (automation bias) or ignore one that raises too many false alarms (alarm fatigue). Explaining the basis of a prediction, showing confidence, and designing clear escalation paths help users rely on the system appropriately.

Monitoring, drift and maintenance

Unlike ordinary software, AI systems can degrade without any code changing, because the world changes. Data drift occurs when the inputs change (a new sensor model, a new customer group, a pandemic changing behavior); concept drift occurs when the relationship between inputs and outcomes changes (fraudsters adopting new tactics). Production systems therefore monitor input distributions, prediction distributions and, when outcomes become known, actual accuracy, and they schedule retraining. They also log decisions for audit and investigation. The engineering discipline of running machine learning in production is called MLOps (AI-420).

Building on foundation models

Since 2023, many AI applications are built not by training models from scratch but by using a general-purpose foundation model, such as a large language model, through an API or as an open-weight model run locally. Developers write instructions (prompts), supply relevant documents through retrieval-augmented generation, connect the model to tools and databases, and sometimes fine-tune it on examples (Lesson 9 and AI-410). This greatly lowers the cost of building prototypes, but the same life cycle still applies: define success, build evaluation sets of realistic inputs and expected outputs, test for errors and harmful outputs, monitor in production, and manage cost, since every request consumes paid computing.

Case studies

• Medical imaging: AI tools that flag possible findings on scans have been cleared by regulators for many uses; the most successful deployments support radiologists, prioritizing urgent cases rather than replacing them, and were validated on data from the hospitals that use them.
• Data center operations: machine learning predicts cooling needs, detects failing equipment, forecasts capacity and optimizes energy use; operators typically keep humans in the loop for control actions on critical infrastructure.
• Customer service: assistants built on language models answer routine questions and draft responses for human agents; studies have reported productivity gains, especially for less experienced staff, alongside risks of incorrect answers that must be managed.
• Software development: AI coding assistants are now widely used to write and review code, with organizations adding tests and review processes to catch errors and security issues.

Privacy and data rights

AI projects often use personal information, and privacy law applies whether or not the law mentions AI. Teams should identify the legal basis for using each dataset, collect only what is needed, protect it with access controls and encryption, remove or mask identifying details where possible, and set retention limits. Using data collected for one purpose to train a model for another may require new consent.

In the United States, there is no single federal privacy law; sector laws such as HIPAA (health) and the Fair Credit Reporting Act (credit) apply, along with comprehensive privacy laws in a growing number of states, such as California's Consumer Privacy Act, some of which give rights related to automated decision-making. The Federal Trade Commission has acted against companies that used data deceptively to train AI models, in some cases requiring them to delete the models.

Privacy-enhancing techniques can help: training on de-identified or synthetic data, federated learning (training across devices or organizations without pooling raw data), and differential privacy (adding carefully calibrated noise so that no individual's data can be inferred from the model).

Governance in organizations

Organizations deploying AI increasingly adopt governance processes: an inventory of AI systems, risk assessments before deployment, documentation of data and models (often in "model cards" and "datasheets"), privacy reviews, testing for bias, human oversight for consequential decisions, incident reporting, and periodic audits. Frameworks such as the NIST AI Risk Management Framework and the international standard ISO/IEC 42001 for AI management systems give structure to this work and are used in both Canada and the United States, alongside the laws discussed in Lesson 11.