Deploy ML models in production with auto-scaling

The production deployment gap: why most ML models never leave the notebook

A trained model in a notebook and a model answering live requests are different engineering problems. The second is a deployment environment with availability, latency and scaling requirements, and it is where most machine learning models in production stall.

What it takes to serve a model at scale (cluster, ingress, autoscaling, monitoring)

Model selection and training are upstream problems. The classic path to serving models in production is a separate process, a process that requires a cluster, an ingress controller, a scaling policy, a monitoring stack and a release pipeline. Each is a decision needing an owner. A data scientist who has finished model development now needs cluster networking, load balancing rules and a metrics backend before one user sees a prediction.

The checklist: a container runtime, ingress rules, scaling policies, secrets and data management, log shipping, release pipelines, plus a tool chain to maintain each. Every item is essential, requires technical ownership, and none of these tools improve the model.

For a small team that is weeks of work before any business value appears, and it does not stop at launch. The result is familiar: model development succeeds, the deployment stalls, and the artifact waits for infrastructure nobody has time to build.

Why DIY VM + Flask + nginx breaks under real traffic

The shortcut is a VM running a Flask app behind nginx. It works in testing and fails in production. One server is one point of failure. No autoscaling means a load spike queues requests until latency is unacceptable. No rolling release means every new version means downtime.

It also hides a cost problem: the instance runs continuously whether the endpoint serves a thousand requests a day or none, and someone still patches the OS and renews certificates. Efficient to set up, expensive to keep.

The hidden cost of idle GPU instances on hyperscaler endpoints

Managed inference endpoints on the large hyperscalers solve availability but keep the compute instance running around the clock. A GPU endpoint sized for a daytime peak is still billed at 3 AM with nothing to serve, and idle time is most of the day.

Utilization is the metric that matters. An endpoint serving a few thousand requests a day on a GPU sized for peak may use a small fraction of what it pays for, every day, for months. Cloud GPU instances make sense when load is constant, but a per-hour instance is the wrong unit for demand that arrives in bursts.

Per-replica, per-minute billing instead of always-on instances

Billing is per replica, per minute. A deployment holding one replica overnight and eight at midday costs the sum of minutes each replica ran, not eight instances for 24 hours. Autoscaling billing is computed on the minimum replica count you define, spike capacity only while it runs. Forecasting is simple: your floor is the minimum multiplied by time, everything above tracks demand, and the software bill keeps the shape of the load curve.

Indicative rates: CPU only from €0.04 per core per hour for classic models such as scikit-learn or XGBoost; GPU L4 from €0.91 per replica per hour for 7B inference, vision and classification; GPU A100 around €1.52 to €1.85 for 7B to 30B; GPU H100 PCIe from €3.10 for larger models and long context windows. Confirm current rates on the pricing page.

Scale to zero when traffic drops, scale up on real spikes

Scale to zero is generally available: a deployment automatically drops to no replicas when nothing is calling it, and the idle cost becomes zero. That is the key economic difference between a scalable managed endpoint and a fixed instance you manage yourself. It has an honest trade-off. Returning from zero means a cold start while the container boots and loads weights, so the first request after idle is slow. For a batch job or internal tool that is acceptable. For a user-facing application with a latency budget, keep a minimum of one replica and accept the floor cost.

Static scaling vs autoscaling (CPU/RAM) vs custom-metric autoscaling

Three scaling strategies, and the choice follows your traffic shape.

Readiness probes and zero-downtime rolling upgrades

A readiness probe is an HTTP path the platform polls before sending traffic to a replica. It is how you ensure that the model is ready: point it at your /health route and the endpoint routes no request until the model is loaded. Without it, a replica takes a request while weights load and returns an error your client sees as a failed request.

Rolling upgrades use the same check. Push a new version and apply the update, and new replicas pass readiness before old ones retire. Users see no failed request, and a version failing its probe never receives traffic. Each version is addressable, so day-to-day releases and an emergency rollback share one mechanism.

Live monitoring of GPU, CPU and network per replica

The dashboard exposes a live metric set per replica: a GPU metric, a memory metric and a network metric, each scoped to one replica rather than averaged, alongside application logs in the Control Panel.

That set answers whether the deployment is healthy, whether the scaling metric is triggering, and where latency sits. It does not answer whether predictions are still correct. For that you need an application metric you define, accuracy on a labelled sample being the usual choice. Teams maintaining models in production usually share one dashboard across data science and platform engineering, so both see the same numbers. This monitoring feature is standard on every app.

Step 1: Package the model in a Docker image (or use a pre-built one)

Containerisation is a prerequisite, not an optional extra. The container image can come from Docker Hub, from Managed Private Registry, or from GitHub Packages. Three requirements matter: create the workspace directory in the Dockerfile, target linux/amd64, and serve on port 8080 unless you declare another explicitly, as vLLM does on 8000.

Teams without container expertise can start from the OVHcloud catalogue of pre-built images with PyTorch, TensorFlow, HuggingFace or FastAI already installed. That reduces the work; it does not remove the requirement. Deploying machine learning this way is an efficient default for most applications, and deploying ML models straight from a catalogue image is faster still, and the experience of maintaining your own base layer is rarely worth it early on.

Step 2: Configure resources, scaling and access via Control Panel, API or ovhai CLI

Create the deployment from the Control Panel, the API, or the ovhai CLI. Choose your compute resources, set the scaling strategy, point the readiness probe at your health path and port, then set how the endpoint is reached.

On access, one rule: a public endpoint is for testing only. Production deployments use restricted access, with AI Platform user credentials or a token, and access policies you maintain yourself. Every endpoint also sits behind native DDoS protection, and each endpoint keeps its own credentials.

ovhai app run --gpu 1 --default-http-port 8080 \
  --probe-path /health \
  --unsecure-http false \
  my-registry/my-model-server:v2

Step 3: Choose the right scaling strategy for your traffic pattern

Match the policy to the workload. Steady internal scoring: static, one or two replicas, an efficient default for internal applications and low-volume tools. A customer-facing API with daily peaks: autoscaling on a utilization metric, minimum one replica to avoid cold starts. Occasional batch scoring: autoscaling with scale to zero. Batch work is where that decision is easiest, because nobody watches a progress bar. Each batch run is a cost you priced already.

Set the maximum deliberately. It caps throughput and bill at the same time, and the ceiling is 10 replicas per app.

Step 4: Monitor inference with the dashboard and the MKS reference architecture

Start with the built-in dashboard for resource metrics and logs. For latency percentiles, throughput or model-level accuracy, add an external stack: Managed Kubernetes Service running Prometheus and Grafana, scraping your endpoint. OVHcloud publishes this as a reference architecture.

Two things to continuously monitor: infrastructure signals that tell you whether scaling works, and model performance in production, which tells you whether the model still does its job. Performance monitoring on the second is what catches a silent regression, and performance metrics from a training dataset will not warn you about it. Drift shows up in the second long before the first. Export a week of online metrics to a file before you optimize a threshold, ensuring the change follows observed load. Set an alert on each to ensure a regression surfaces from your own insights rather than from a customer, ensure the threshold is based on a real baseline, and ensure it has an owner.

Step 5: Push a new version with a zero-downtime rolling upgrade

Build and push the new version, then run ovhai app update or apply the change from the Control Panel. The platform starts new replicas, waits for readiness, shifts traffic, and retires the old. No downtime, no maintenance window.

The Dockerfile is code: keep it in version control and the base updated, since a stale base layer is a common cause of a failed rollout. Keep tags versioned rather than reusing latest. Version control on each version is what makes a rollback a one-command operation instead of an investigation.

Why CPU/RAM is a poor proxy for LLM load

Deep learning inference is a computational workload with a queue, not a CPU-bound loop, so the standard practice of scaling on hardware misfires. An LLM server batches concurrent requests on the GPU. CPU utilization stays low while the queue grows, so that threshold triggers late or never. The signal you care about is how many requests are in flight, and a hardware metric cannot see it.

Scaling on vllm:num_requests_running with Prometheus and Grafana on MKS

vLLM exposes vllm:num_requests_running on its metrics endpoint. Scrape it with Prometheus and autoscale on that value instead of a hardware proxy. Replicas arrive when real concurrency rises and leave when the queue drains, keeping latency inside budget without over-provisioning.

Generally available and documented as a reference architecture. It needs Prometheus and Grafana alongside your deployment, so treat it as the advanced approach, not the default.

Setup time, idle cost and operational expertise

Self-managing inference means provisioning the cluster, node pools, an ingress controller and a horizontal autoscaler, typically one to two weeks, plus DevOps and MLOps expertise to keep it running. You own the scaling policies, deployment pipelines, manifest files and monitoring strategies, and you continue to own them after launch. Best practices in the field are documented and the tools mature, but in this field a documented practice is not a robust running system. The GPU node usually runs continuously, including at zero traffic.

AI Deploy needs one image and one command. Setup is under an hour, MLOps expertise alone is enough, and scale to zero removes idle cost entirely.

Prices and figures are indicative. Confirm live rates on the OVHcloud pricing page before publishing.

No CLOUD Act exposure on inference data

Training data is not the only sensitive input. Every request to a production endpoint carries user or business data, every request is logged somewhere, every request is an input you did not choose, and endpoint traffic is continuous rather than a one-off transfer. If the endpoint runs on a US-headquartered provider, that data sits under US jurisdiction regardless of the region hosting it.

OVHcloud is European with no US parent company and no structural CLOUD Act exposure. Inference data stays in EU jurisdiction under EU law.

GDPR, HDS and ISO 27701 for regulated workloads

OVHcloud holds ISO 27001, ISO 27017, ISO 27018 and ISO 27701, is SOC 2 certified, and holds HDS certification for health data in France, relevant for medical inference. Processing inside the EU simplifies a Data Protection Impact Assessment, and the data privacy commitments you made to your own users become simpler to evidence. Support can answer a privacy question without escalating.

One boundary to state plainly: AI Deploy supplies sovereign serving infrastructure. Model-level EU AI Act obligations, model cards, audit logs and bias detection, remain your responsibility as the system developer. Sovereign infrastructure supports compliance; it does not deliver it.