Production AI often looks flexible from the outside. A user uploads an image, a recommendation system ranks new content, or an AI feature receives a request that differs slightly from the last one. Inside the inference stack, that variability can be expensive. The AI inference market is expected to grow from $106.15 billion in 2025 to $254.98 billion by 2030, making inference efficiency a central engineering and cost problem rather than a background infrastructure issue.
That is the kind of systems problem behind Aneesh Gupta’s work. A Machine Learning Engineer with experience across ML systems, compilers, model evaluation, and large-scale production infrastructure, Gupta has worked on dynamic shape handling for ML model inference, a capability that allows input shapes to change at runtime for the same trained model. His work sits at the intersection of model deployment, compiler behavior, runtime design, and accelerator performance.
Why do static input shapes become a problem in production ML systems?
Static shapes are convenient during development because they make the system easier to reason about. You know the size of the input, you know the path the compiler will take, and you can optimize around that assumption.
Production does not stay that clean. A model may receive different image dimensions, batch sizes, sequence lengths, or feature layouts depending on the product surface and user behavior. If the inference system expects everything to arrive in a fixed form, teams end up paying for that rigidity through recompilation, duplicate deployment paths, or conservative fallbacks. The model may be capable, but the serving path becomes the bottleneck.
What is the core idea behind dynamic shapes?
The core idea is to let the same trained model accept changing input shapes during runtime without forcing every shape variation into a separate deployment problem. My work focused on dynamic shapes for ML model inference on custom AI accelerator chips, and the project ran across roughly 12 to 14 months of production infrastructure work.
The goal was not just to make a feature work in isolation. It had to work in the actual deployment path. That meant building new capabilities and helping models integrate with the accelerator platform so teams could run ML workloads more efficiently when inputs varied.
Why does this matter economically?
Inference cost scales with volume, and inefficiency compounds quickly. The AI infrastructure market was valued at $58.78 billion in 2025 and is projected to reach $497.98 billion by 2034. When infrastructure spending is moving at that scale, small inefficiencies in the serving path can become large annual costs.
The efficiency implications can be substantial. When models must handle variable input shapes, the effects extend beyond compiler behavior to accelerator utilization, runtime latency and operating cost. In my view, dynamic shape support should therefore not be treated as a narrow compiler feature. It is a systems capability whose value depends on how effectively the compiler, runtime and hardware work together to preserve performance as workloads change.
Where does the compiler fit into the problem?
The compiler is where a lot of the promise either holds up or breaks down. A model can appear to support variable inputs conceptually, but if the compiler and runtime cannot handle that variability efficiently, the system still behaves as if the real world must be static.
Dynamic shape support requires the compiler, runtime, and hardware execution path to cooperate. The system has to preserve performance while accepting changing input dimensions. If every new shape creates a costly special case, the feature does not solve the production problem. It just moves the complexity somewhere else.
Why is this also a runtime and MLOps issue, not just a compiler issue?
Production teams need a repeatable way to deploy and operate models, not a one-off technical solution. Dynamic shape support may begin at the compiler level, but its practical value depends on whether the runtime can manage changing inputs reliably and whether MLOps workflows can deploy, monitor and update those models without adding fragile, shape-specific processes.
For dynamic shapes to work in production, the capability must fit into standard model pipelines. Teams should not have to maintain separate execution paths for every input configuration. A well-integrated approach reduces engineering overhead, makes performance improvements available across workloads and allows models to adapt to real-world variability without sacrificing operational stability.
How do custom AI accelerator chips change the stakes?
Accelerators are valuable because they can run ML workloads efficiently, but that value depends on utilization. The AI accelerators market is expected to grow from $140.55 billion in 2025 to $518.12 billion by 2031. If a model cannot adapt cleanly to real input variability, expensive accelerator capacity can be underused or forced into inefficient execution paths.
My role was to help models integrate with the accelerator platform and build features that made dynamic shape inference practical. The work had to respect hardware behavior, not just model logic. You are not only asking whether the model can run, but whether it can run efficiently when production inputs stop matching the neat assumptions of development.
What makes dynamic shape handling difficult to productionize?
The hard part is that it touches many layers at once. The model, compiler, runtime, and hardware path all have to agree on how variability is represented and handled. If one layer treats a shape change as routine and another treats it as a special case, the system can lose the performance benefits.
There is also the adoption problem. If model teams have to understand every low-level detail, the work will not scale. The path has to be usable. In my view, the best infrastructure work makes the efficient path the default one. Teams should not have to become shape-handling experts to benefit from dynamic inference.
How does this work compare with ordinary model optimization?
Ordinary model optimization often focuses on a specific model, a specific input pattern, or a specific performance target. Dynamic shape work is more foundational. It changes what kinds of inputs the serving system can handle without multiplying operational complexity.
That matters because product behavior changes. User inputs vary. Workloads shift. Models get reused in new contexts. If the infrastructure is too rigid, every change creates a new engineering burden. Dynamic shape handling reduces that burden by making variability part of the system design rather than an exception the team has to patch around.
What did the project prove inside production infrastructure?
It proved that dynamic shape handling could move from infrastructure feature to production capability. The project was a first of its kind capability inside the organization, and it helped engineering teams deploy ML models and workflows more performantly on custom AI accelerator hardware.
It also reduced the need for shape-specific workarounds, which matters because operational complexity has its own cost. The real proof was that a systems-level feature could make accelerator-backed inference easier for model teams to adopt without forcing every team to rebuild its workflow around input-shape edge cases.
How should engineering teams think about static inference going forward?
Static inference is still useful where inputs are genuinely predictable. The problem is assuming that predictability exists everywhere. The Edge AI Chips market was valued at $3.67 billion in 2025 and is estimated to reach $11.54 billion by 2031, reflecting demand for inference workloads that must respond closer to real-world conditions.
My view is that dynamic shape handling will become part of the infrastructure maturity curve. Teams will not judge systems only by whether a model runs under ideal conditions. They will judge whether the system can absorb changing inputs, keep performance stable, and avoid unnecessary cost.
For Gupta, the hidden cost of static AI is not that models fail dramatically. It is that rigid inference assumptions quietly tax the system every time real-world inputs stop looking like the development environment. Dynamic shapes offer a different path: one trained model, variable inputs, fewer deployment workarounds, and infrastructure that treats variability as expected rather than exceptional.
