Terraform at Scale: What Changes When You're Managing ML Infrastructure

ML infra has different properties

Standard application infrastructure tends to be long-lived and relatively stable. You provision a service, attach autoscaling, and it runs with predictable patterns. ML infrastructure behaves differently. It is inherently elastic, frequently ephemeral, and often driven by experimentation rather than deterministic workloads.

GPU-backed workloads, for example, introduce a fundamentally different cost and scheduling model. Unlike CPU-based services, GPUs are scarce, expensive, and often subject to regional quota constraints. Keeping them warm ensures low latency but creates significant idle cost. Spinning them up on demand reduces cost but introduces provisioning latency that directly impacts user workflows, especially for model serving or batch inference jobs.

This creates a tension that Terraform alone does not abstract away. Infrastructure definitions must account for bursty provisioning patterns, pre-emptible capacity, and heterogeneous node pools. In practice, this means your IaC is no longer just declarative infrastructure—it becomes a control plane for resource orchestration strategies.

A typical Kubernetes node pool definition for GPU workloads might look like this:

resource "google_container_node_pool" "gpu_pool" {
  name       = "gpu-pool"
  cluster    = google_container_cluster.primary.name
  node_count = 0
 
  autoscaling {
    min_node_count = 0
    max_node_count = 10
  }
 
  node_config {
    machine_type = "n1-standard-8"
 
    guest_accelerator {
      type  = "nvidia-tesla-t4"
      count = 1
    }
 
    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform"
    ]
  }
}

The key detail here is that the pool can scale to zero, allowing cost optimisation while still enabling rapid scale-up when jobs are scheduled. However, this introduces cold-start latency that must be accounted for at the application layer.

The module design that worked for us

At scale, Terraform module design becomes less about reusability in the abstract and more about encoding operational intent. In ML environments, the variability of workloads means modules must strike a balance between standardisation and flexibility.

The approach that proved effective was to build modules with strongly opinionated defaults while still allowing controlled overrides. Most environments—training clusters, inference services, experiment sandboxes—share common requirements: networking, IAM roles, logging, and baseline monitoring. Encoding these into modules reduces cognitive load and prevents configuration drift.

At the same time, ML workloads frequently require deviations. A researcher might need a different instance type, more ephemeral storage, or a custom autoscaling policy. Rather than forcing a fork, modules exposed override variables that allowed deviation without breaking the underlying contract.

module "ml_environment" {
  source = "./modules/ml_env"
 
  name            = "experiment-123"
  region          = "europe-west1"
  machine_type    = var.machine_type != "" ? var.machine_type : "n1-standard-4"
  enable_gpu      = var.enable_gpu
  gpu_type        = var.gpu_type
  max_node_count  = var.max_node_count != 0 ? var.max_node_count : 5
}

This pattern ensures that the majority of environments can be provisioned with minimal configuration, while still supporting edge cases without introducing module sprawl.

State management also required a deliberate design. ML environments operate across different lifecycles—ephemeral experiments, semi-stable staging environments, and long-lived production systems. Collapsing these into a single Terraform state introduces unacceptable risk. A misconfigured destroy operation in an experiment environment should never have the blast radius to affect production.

Separating state per lifecycle, with distinct backends, isolates failure domains:

terraform {
  backend "gcs" {
    bucket  = "ml-terraform-state"
    prefix  = "experiments/experiment-123"
  }
}

This isolation is not just a safety mechanism; it also improves team velocity by allowing parallel changes across environments without state lock contention.

Another non-obvious constraint in ML infrastructure is GPU quota management. Cloud providers treat GPUs as constrained resources, and quota exhaustion can silently block deployments. Instead of treating quotas as an external concern, we encoded them directly into Terraform-adjacent workflows so that capacity planning became visible and enforceable.

resource "google_project_quota" "gpu_quota" {
  project = var.project_id
  metric  = "compute.googleapis.com/gpus_all_regions"
  limit   = 16
}

While not all providers allow direct quota manipulation via Terraform, representing expected quotas in code—and validating against them in CI—prevents a class of runtime failures that are otherwise difficult to debug.

The drift problem

Infrastructure drift is a known issue in any IaC system, but it becomes significantly more pronounced in ML environments. The reason is behavioural rather than technical. Data scientists and ML engineers frequently make ad hoc changes during experiments—resizing disks, attaching GPUs, modifying runtime parameters—because iteration speed matters more than process in early-stage exploration.

These changes are often not codified back into Terraform, which leads to divergence between declared state and actual infrastructure. Over time, this drift accumulates and manifests as failed applies, inconsistent environments, or, in worst cases, production incidents.

The only reliable mitigation is to treat drift detection as a continuous process rather than a reactive one. Running terraform plan manually is insufficient; it must be automated and enforced.

A typical CI-based drift detection job might look like this:

name: Drift Detection
 
on:
  schedule:
    - cron: "0 * * * *"  # hourly
 
jobs:
  terraform-plan:
    runs-on: ubuntu-latest
 
    steps:
      - name: Checkout
        uses: actions/checkout@v3
 
      - name: Terraform Init
        run: terraform init
 
      - name: Terraform Plan
        run: terraform plan -detailed-exitcode

Using -detailed-exitcode allows the pipeline to distinguish between no changes, drift detected, and execution errors. This can then be wired into alerting systems so that teams are notified as soon as divergence occurs.

In practice, this shifts drift from being an operational surprise to a managed signal. Teams can decide whether to reconcile the infrastructure back to code or accept the change and update the Terraform configuration accordingly.

The underlying shift

What changes when managing ML infrastructure with Terraform is not just the scale—it is the nature of the system being managed. Infrastructure becomes more transient, workloads become less predictable, and human interaction with the system becomes less disciplined by necessity.

Terraform remains a powerful abstraction, but it must be augmented with patterns that acknowledge these realities: lifecycle isolation, flexible module design, quota awareness, and continuous drift detection.

Without these, IaC in ML environments degrades quickly into a source of friction rather than a source of control.