Production AI workloads often sustain only 10% to 50% GPU utilization, even under load. The mismatch is structural: much of the orchestration layer inherited assumptions from uniform, stateless web services, while AI jobs are heterogeneous, stateful, and multi-stage.
Suman Debnath, Director of Developer Relations at Crusoe Energy Systems and an IEEE Senior Member, has spent years teaching developers how distributed AI pipelines behave under real load. His operating principle is simple: measure the movement of the whole workload, not the activity of its most expensive component. To understand why AI infrastructure produces so much idle time, and what teams should build instead, we spoke with Debnath about the state of GPU utilization and what lies ahead.
The Slowest Stage Owns the Run
“On one video run, the GPU graph looked busy in short bursts, but the wall clock was barely improving,” Debnath recalls. “The slow part was upstream, so tuning the model would have polished the wrong stage.” That is the bottleneck. AI workloads rarely consist of one clean kernel running at full speed. Video has to be decoded, frames prepared, batches assembled, models loaded, inference completed, and results written somewhere useful.
A 2026 analysis of production clusters found average GPU utilization of just 5%; at that level, organizations held roughly 20 times as much GPU capacity as their workloads consumed. Debnath demonstrated the alternative with a mixed CPU and GPU video pipeline that processed between 40,000 and 45,000 raw videos, representing roughly 3,000 to 3,500 hours of content, in under five hours. The result came from sizing independent actor pools to each stage, allowing CPU and GPU workers to run concurrently, and streaming data directly between stages instead of writing intermediate files. No single stage dictated the shape of the entire cluster.
Utilization Is an End-to-End Number
Once the slowest stage is visible, the measurement problem becomes harder to ignore. A GPU can report high activity during inference bursts while the job spends most of its life waiting for preprocessing, object transfers, model initialization, or downstream writes. The dashboard can still lie. One recent distributed training design cut training time by as much as 17 times and raised GPU utilization to 94% by filling idle cycles with useful inference work. The result came from measuring and scheduling the complete job.
Debnath has pushed that distinction through a distributed training blog series that reached more than 300,000 professional impressions, with one post drawing roughly 800 to 900 reactions. The series taught distributed training through Ray and PyTorch or FSDP, translating the subject into material that practitioners could apply when scaling work across multiple machines. “A percentage on one device is not an efficiency metric,” he says. “The useful number is end-to-end throughput per dollar and per hour.”
The series turned a specialist concern into a practical engineering question. Teams could compare whether a faster model step actually shortened the run, or whether it simply moved the queue to another stage. That is a more demanding standard, but it is also harder to game.
Concurrent Work Beats Sequential Speed
Changing the measurement changes the design. Sequential pipelines make CPUs wait during GPU work, then make GPUs wait while the next batch is prepared. Intermediate files create another pause. A 2025 distributed-training system achieved up to a 1.28 times speedup over a ZeRO-3 baseline and up to a 7.01 times throughput increase under limited GPU resources by coordinating execution and memory decisions. The gains came from treating communication timing and memory pressure as connected parts of the training run. Sequential execution is costly.
Debnath brought the same systems lesson to a Ray and DeepSpeed “AI at Scale” presentation at the PyTorch Conference. The presentation focused on distributed training at scale, giving practitioners a concrete setting for examining how execution choices affect the use of GPU workers. His role as a judge for the Creatorscorner AI Agent Hackathon reinforces the same point from another angle, because evaluating agent systems means looking beyond whether a model answers once and asking whether the complete workflow remains reliable across tools, state, and repeated steps.
Efficient Pipelines Have to Become a Developer Habit
Better scheduling does not spread by itself. The decisions that determine GPU utilization are often made before a job reaches production. A team may tie preprocessing and inference to the same scaling policy, or write every intermediate result to storage because that is how an earlier service was built. Those choices create idle time later.
Debnath addressed that knowledge gap through education around Ray, the open-source distributed-computing framework used for AI data processing, training, and inference. During his tenure, course signups rose from roughly 200 to 250 per quarter to about 600 to 650, nearly tripling. The increase does not prove that every learner improved GPU utilization. It does show substantial demand for practical guidance on distributed execution, the layer where teams learn to identify pipeline bottlenecks rather than treating accelerator activity as an isolated problem. “Most waste is decided before the job reaches production,” Debnath says. “Teams have to learn that a busy GPU is not the same as an efficient pipeline.”
That lesson changes the design review. Teams ask which stage is starving the next one, whether data can stream directly between stages, and which worker pool should scale first. They stop treating additional accelerators as the default response to slow throughput. Those operational questions become harder as workloads grow more stateful, irregular, and dependent on multiple compute stages.
The Next Constraint Is Workload Shape
The utilization problem will become more severe as agentic AI spreads. Agents are stateful, tool-using, and irregular. One request may trigger retrieval, code execution, model inference, validation, and a second model call, with each step demanding a different resource profile. Debnath saw that complexity move into mainstream developer practice while leading public education for the Strands Agents SDK. Within four months, the SDK recorded roughly 1 million to 1.2 million downloads and between 3,000 and 3,500 public repository stars; his tutorial series reached about 70,000 to 75,000 views. Adoption moved quickly.
The deployment curve is steep. Actively deployed AI agents are projected to exceed 1 billion worldwide by 2029, roughly 40 times the 2025 level. In one major enterprise software category, agentic AI adoption is projected to rise from 5% in 2025 to 60% by 2030. Debnath’s technical review work for O’Reilly’s forthcoming AI Systems Performance Engineering book fits this next phase because performance engineering is becoming a capacity discipline, not merely a latency exercise. “Enterprises cannot keep treating idle accelerators as a procurement problem,” he says. “The organizations that pull ahead will measure useful output across the whole pipeline, then design every stage so expensive compute is never waiting without a reason.”
