Why Finance Teams Are Auditing Where Inference Runs

Why Finance Teams Are Auditing Where Inference Runs — Why is our GPU bill suddenly bigger than our data warehouse bill? CFOs are typing that question into search bars this quarter, and the answer is almost always the same. A workload pitched as a pilot has slipped into production, and nobody told finance where the inference actually runs.

What used to be a rounding-error line on the cloud invoice is now a standalone cost center with its own vocabulary: tokens, context windows, GPU-hours, egress. Finance teams are catching up fast, and they are asking harder questions than engineering has had to answer before.

Why Finance Teams Are Auditing Where Inference Runs

The shift is not subtle. Training was the headline cost for years, but the live, user-facing workload is now the one chewing through the budget. Menlo Ventures found that inference has overtaken training as the primary AI workload, with 74% of startups and 49% of enterprises reporting that inference accounts for most of their compute usage.

That matters for finance because inference does not behave like training. Training is a project with a start, an end, and a check you write once.

Inference is a utility bill. It scales with every signed-in user, every retry, every long context window somebody pasted into a prompt. The meter runs whether anyone is watching or not.

Once the workload stopped being a one-off and started being a recurring cost, the controllers noticed, and so did the auditors. That shift is also why vendors building private AI hardware for on-prem LLM deployments are showing up in finance conversations that used to be purely about cloud spend.

What Is Finance Looking For When They Audit Inference

The audit question is not really “how much are we spending.” It is “what are we getting per dollar, and where is the money leaking.” A few specific things tend to come up first.

• Cost per request. A dollar figure attached to each call to the model, broken out by product and by user tier.

• Utilization. How busy the GPUs actually are during the hours they are billed, not the hours they are reserved.

• Idle and retry spend. Money burned on failed calls, timeouts, and warm capacity nobody is using.

• Egress and context. The surcharges that pile up when data moves between regions and when prompts keep growing in token count.

The FinOps Foundation has codified much of this into tagging schemas that separate training from inference and tie workloads back to a product or team. If your current cost report cannot split those two, you are flying blind.

Where Does Inference Actually Run Today

In most companies that started with an API call to a hosted model, the answer is “somewhere else.” That was fine when traffic was small. It becomes a strategic question once the workload is predictable.

• Hosted API. Someone else runs the model; you pay per token. Fastest to start, hardest to forecast at scale.

• Cloud GPU rental. You run the model on rented hardware in a hyperscaler. More control, still a metered bill.

• On-premise or colocated hardware. You own or lease the GPUs and run them in your own racks or a colo. Highest fixed cost, lowest marginal cost per request.

For teams running steady, high-utilization inference, which is the profile that pushed this onto the CFO’s desk in the first place, the on-premise option has moved from unusual to ordinary. Vendors now ship configurations sized to a team’s actual model and user count, which turns “we should look at this” into a quote finance can model.

When Does On-Prem Actually Pencil Out

There is a utilization threshold below which rented cloud GPUs win and above which owned hardware wins. The exact crossover depends on your model, your traffic shape, and how long you plan to run the workload. The directional answer is well-established: once a GPU is busy most of the time, you are paying a premium to rent it.

The practical test for finance is simple. Pull last quarter’s GPU utilization. If it is low and spiky, stay in the cloud and work on right-sizing. If it is high and flat, model the owned or leased alternative, including power, cooling, and someone to operate it, and compare the three-year total, not the first-year sticker.

What Finance Should Ask Engineering This Quarter

A short list of questions tends to surface most of what matters.

1. What is our cost per thousand requests, by product, this month versus last?

2. What is the average GPU utilization on the instances we are paying for?

3. How much of last month’s inference bill came from retries, timeouts, or runaway context windows?

4. If traffic doubles, does the bill double, or does it do something worse?

5. At what sustained volume would owned or leased hardware be cheaper than what we are paying now?

None of these are gotcha questions. They are the ones a thoughtful controller will ask eventually. Answering them before the audit is a lot less expensive than answering them during one.

For more coverage on this topic, see Techbullion.