AI jobs are waiting while GPUs sit idle. Kubernetes has a new plan.

A Kubernetes scheduler places pods on nodes that already exist. An autoscaler adds nodes when more capacity is needed. With GPUs, those decisions can fall out of sync, so a workload may remain pending even while GPUs elsewhere sit idle.

Capacity reservations, device topology, and GPU sharing make the mismatch harder. The scheduler may not recognize capacity that an autoscaler could provision, or the autoscaler may add a node that does not fit the workload’s placement needs. This creates delay and wasted accelerator time.

The problem is especially serious for inference. Bursty serving and agentic workloads cannot wait long for a cold node. AWS authors describe the issue as a lack of agreement between scheduling and provisioning loops. Proposed fixes include DRA-aware autoscaling, workload-aware scheduling, and provisioning requests that connect placement needs with capacity creation.

The proposed mechanisms aim to make GPU scheduling and capacity provisioning act as one coordinated process. Today, the scheduler chooses among existing nodes, while the autoscaler creates new ones. AI workloads expose the gap because GPUs are scarce, specialized, and often needed in particular configurations.

Dynamic resource allocation, or DRA, gives autoscalers a better view of accelerator requirements. Workload-aware scheduling adds knowledge of each job’s needs. Provisioning requests provide a companion mechanism for connecting those needs to capacity that can be created. Together, they reduce the chance that pods remain pending while suitable GPUs are unavailable to the scheduler.

A public SIG-Autoscaling proposal goes further. Its APIs would let the scheduler see provisionable capacity, express placement preferences across it, and explicitly request it. The article expects autoscalers to become more scheduler-aware partners, improving responsiveness for bursty inference and agentic workloads.

Kubernetes is a platform for running and coordinating containerized workloads across a cluster of machines. It groups application processes into pods and places those pods on available nodes. This matters for AI because training and inference jobs can require many machines, specialized GPUs, and automated handling of changing demand.

The article describes enterprises using Kubernetes for large-scale AI and machine-learning jobs through HPE Private Cloud AI. Kubernetes supplies troubleshooting commands and manages the infrastructure where long-running workloads run. Operators can also use automation to collect pod-restart details and send diagnostics to Slack.

Kubernetes is not automatically perfect for AI. GPU scheduling, autoscaling, reservations, topology, and sharing introduce extra complexity. The ecosystem is therefore adding DRA-aware autoscalers, workload-aware scheduling, provisioning requests, and distributed inference tools such as llm-d to make production AI operations more reliable.

Dynamic resource allocation, or DRA, helps Kubernetes handle specialized resources such as GPUs more precisely than simple capacity counting. It can represent which accelerator characteristics a workload needs. That gives scheduling and autoscaling better information when deciding whether existing or future capacity will fit.

Provisioning requests add the missing link to capacity creation. If a workload needs a particular GPU arrangement, the scheduler can express that requirement instead of merely reporting that a pod is pending. The autoscaler can then act on the request and provision compatible capacity, while placement preferences guide where the workload should run.

These mechanisms are proposed solutions, not a completed universal fix. The article describes DRA-aware autoscalers, workload-aware scheduling, and provisioning requests as approaches already taking shape. A SIG-Autoscaling proposal would deepen coordination by letting schedulers see and explicitly request provisionable capacity.

IBM Research reported a major efficiency gain using llm-d for an agent factory exploring quantum error-correction hardware. The demonstration used open models and achieved inference costs 11.5 times lower than an all-proprietary configuration. That makes the result notable for workloads that call models repeatedly and unpredictably.

The key mechanism was distributed inference tuned for agentic traffic. llm-d handled bursty requests across a cloud-native stack and pushed cache reuse to about 98% in IBM’s test. Reusing cached information can reduce repeated inference work, helping the system serve more requests without simply adding equivalent new capacity.

The result is one demonstration, not a guarantee for every deployment. Still, llm-d’s v1.0 release signals a move from experimental serving toward infrastructure intended for production. The project has also seen use at Tesla, AWS, Oracle, Google, and beyond, according to the article.

Adding always-on GPU servers is not the only way to handle sudden inference demand. Operators can distribute inference across a cloud-native stack, scale capacity with workload demand, and reuse results or intermediate state where possible. These approaches aim to absorb bursts without paying continuously for the largest fleet.

The article’s clearest example is llm-d. The CNCF sandbox project reached v1.0 with a focus on sustaining performance in production. In an IBM Research demonstration, it handled an agent factory’s bursty agentic traffic and pushed cache reuse to about 98%. Open models also produced much lower reported inference costs than an all-proprietary setup.

Kubernetes coordination is another alternative to brute-force capacity. DRA-aware autoscalers, workload-aware scheduling, and provisioning requests can help create or place GPU capacity when needed. The article presents these mechanisms as developing solutions, rather than claiming they eliminate every scaling challenge.

A pod is Kubernetes’s basic unit for running one or more related containers. A node is a machine in the cluster that can host pods. A GPU is specialized accelerator hardware often used to process AI models efficiently. Inference is the act of using a trained model to produce an output from an input request.

When a request arrives, an inference application runs inside a pod. Kubernetes’s scheduler places that pod on a node with suitable resources, including an available GPU if the model needs one. The application loads or accesses the model and uses the GPU to calculate the response. The autoscaler may add a node when existing capacity cannot meet demand.

The article highlights why this flow can fail under pressure. The scheduler sees existing nodes, while the autoscaler provisions new ones. If they disagree about GPU capacity, the inference pod can remain pending while accelerators sit idle. llm-d aims to improve distributed inference for these workloads.