Rules of Machine Learning
Start with the deployment pipeline
Read Google's practical ML engineering guide, focusing on the first pipeline and launching and iterating.
Start with: Rules of Machine Learning
RedLectern topic
AI deployment
Move models and applications into operation · architecture · release controls · reliability.
For: Developers · operators
After this page you can: Plan an AI application's deployment and rollback.
Page reviewed
Part of: Artificial Intelligence
About AI deployment
AI deployment is the work of moving a model and the application around it into an operating environment. The model may serve predictions through an online service, run as part of a batch process, or be embedded in an edge or mobile application. Production work also includes the surrounding pipeline, application integration, validation, and monitoring; deploying a model artifact alone does not describe the whole system.
A useful plan connects the intended outcome to a system design, checks each release before promotion, and decides how traffic can move between versions. Common approaches include canary, shadow, blue/green, and A/B releases. Each has different implications for what users see during the rollout, what evidence can be collected, and how a rollback is triggered. This page walks from architecture through release checks and recovery, so developers and operators can make those choices before a change reaches users.
Start here
MLOps: Continuous delivery and automation pipelines in machine learning
Map the production system
Use the MLOps architecture guide to trace code, validation, deployment, and monitoring through a pipeline.
Start with: MLOps: Continuous delivery and automation pipelines in machine learning
Continuous deployment
Plan promotion and rollback
Compare rollout strategies and make rollback part of the release plan.
Start with: Continuous deployment
Learn the foundations
1.Design the system around the modelStart by deciding where inference belongs in the application: an online service, a batch prediction process, or an embedded model on an edge or mobile device.
Start by deciding where inference belongs in the application: an online service, a batch prediction process, or an embedded model on an edge or mobile device. Those patterns have different operational boundaries, so map the application, model, data inputs, and prediction outputs as one system. A production pipeline may include source control, build and test services, deployment services, a model registry, metadata storage, and orchestration. Keep the training and serving path understandable enough to check compatibility and trace which model version is serving.
- Rules of Machine Learning
Its first-pipeline guidance discusses application integration and whether to apply a model live or precompute results offline.
Try it
For a model that scores incoming support tickets, choose online, batch, or embedded serving. Name one application component and one operational artifact you would include in the deployment plan.
Online serving fits a score needed as each ticket arrives. The application needs a component that sends ticket features and handles the returned score; the deployment plan can track the model in a registry.
2.Set release checks and promotion gatesA release pipeline can build and test changes, package artifacts, deploy them to a target environment, and then serve a validated model.
A release pipeline can build and test changes, package artifacts, deploy them to a target environment, and then serve a validated model. Define checks before promotion: test application and pipeline behavior, validate the model against an agreed baseline, and confirm that the serving environment supports the model's required dependencies and resources. Keep the checks tied to the behavior and quality you expect from the system. A staged validation process can run tests against validation data before the candidate reaches production, with a runbook and metrics for promotion.
- Continuous deployment
Its promotion checklist calls for staged validation, metrics, and a runbook.
Try it
A candidate model passes an offline quality check, but its serving package has not been checked in the target environment. Should it be promoted? State one check and one artifact to add before promotion.
Do not promote until compatibility with the target serving environment is checked. Add a recorded validation result and a release runbook with the promotion metrics.
3.Choose a controlled rolloutChoose a rollout method based on what you need to learn and how much production traffic should reach the candidate at first.
Choose a rollout method based on what you need to learn and how much production traffic should reach the candidate at first. In a canary, a small share of traffic reaches the new model before the rollout expands; monitor it while it is serving. In a shadow deployment, the candidate receives traffic but does not influence user decisions, so its predictions can be compared with the current system. Blue/green keeps old and new environments available while traffic shifts between them. An A/B test compares outcomes across groups; identify the product measure that matters, since model metrics may only be proxies for it.
- Continuous deployment
It describes how canary, shadow, and blue/green releases route traffic between model versions.
Try it
You need to compare a candidate model's outputs with the live model without letting the candidate affect decisions. Choose a rollout strategy and explain what happens to each model's output.
Use shadow deployment: send traffic to both models, but let only the existing model's output affect decisions. Compare the candidate's predictions with the existing model before manually promoting it.
4.Make recovery part of the releasePlan recovery before promotion: identify the condition that pauses or reverses a rollout, the person or automation that acts, and the version or fallback to us…
Plan recovery before promotion: identify the condition that pauses or reverses a rollout, the person or automation that acts, and the version or fallback to use. A rollback returns to a previous deployment version; a fallback replaces the model with a heuristic. Keep both the earlier version and the steps needed to restore it available for the period your release process requires. A canary or blue/green release can make traffic movement reversible, but the team still needs monitored signals and a runbook that explains when and how to act.
- Deployment
It explains how keeping the old system available in blue/green deployment enables a return to it.
Try it
A new model's monitored behavior deviates from expectations during a canary rollout. Give one rollback action and one item the runbook should specify.
Stop sending traffic to the candidate and route it back to the previous model version. The runbook should identify the rollback trigger and the steps or owner for switching traffic.
Explore subtopics
Latest
No recent updates are available.
Books
- Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications
Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications
Chip Huyen · 2022
Use it to plan an ML platform and a lifecycle that carries models through evaluation, deployment, and updates.
- Reliable Machine Learning: Applying SRE Principles to ML in Production
Reliable Machine Learning: Applying SRE Principles to ML in Production
Cathy Chen, Niall Richard Murphy, Kranti Parisa, D. Sculley, and Todd Underwood · 2021
Use its SRE perspective to think through production troubleshooting and the operational work needed to run ML systems reliably.