Someone planning a bathroom renovation asks a generative answer engine what it will cost to tile the space. Within seconds, they have an answer: a specific figure, broken into labour and materials, presented with the same even confidence the model uses for a historical date or a chemical formula. It looks researched. It looks like the kind of number a person could budget against.
Then the real quotes come in, and none of them are close. Not “a bit higher.” Not “within a reasonable range.” Materially, structurally wrong — sometimes by a factor of two or three.
The unsettling part is not that the model was wrong. Models are wrong about plenty of things. It is that nothing about the answer signalled uncertainty. There was no hedge, no range, no “this varies by region and installer.” The system delivered a guess with the packaging of a fact. That gap — between the confidence of the output and the reliability of the underlying knowledge — is the actual problem, and it has a name worth using precisely.
The failure mode: confident, specific, and wrong
Call it price hallucination: a generative model producing a real-world cost figure that sounds authoritative but has no reliable grounding behind it. It is a specific case of a broader, well-documented pattern that institutions including Stanford’s Human-Centered AI institute track closely — large language models optimise for the most fluent, plausible-sounding continuation of a prompt, not for verified correctness. Fluency scales predictably with training. Factual accuracy on narrow, real-world questions does not.
Prices are close to a worst-case category for this failure mode. They are specific — a number, not a sentiment. They are local — a tiling job in one city costs meaningfully more than the same job two provinces away. They are time-varying — material and labour costs drift month to month. And critically, there is no single canonical public source a model could have memorised the way it might memorise a well-documented fact. Real-world service pricing lives in fragmented quotes, regional trade forums, and pages that go stale within a year. The model was never shown a reliable answer, so it generated the most plausible-sounding one instead. Confidence is a property of the language model. Correctness has to come from somewhere else entirely.
Why a bigger model doesn’t fix it
The instinctive response is to assume the next generation of models will simply know better. That instinct misdiagnoses the problem. The gap here is one of data availability and grounding, not raw parameter count.
A model trained on a snapshot of the open web inherits every hole in that web. Real-world service prices are sparse, unstructured, regional, and stale by the time any training run captures them — no amount of additional scale changes what data existed at training time. Scaling a model improves its reasoning, its fluency, its ability to follow instructions. It does not retroactively grant access to ground truth the model was never shown. If anything, a larger, more articulate model can make a wrong number more persuasive, not more correct — the hallucination simply arrives better dressed.
This is why the fix under active discussion across serious AI-reliability research is not “wait for GPT-N” but a change in architecture: stop asking the model to recall a price from its weights, and instead have it look one up.
The fix: retrieval over a narrow, structured ground-truth dataset
Retrieval-augmented generation, or RAG, addresses this directly. Instead of relying solely on what was baked into the model during training, a RAG system queries an external, trusted dataset at answer time and grounds its response in what it retrieves. The model still generates the sentence, but the number inside that sentence comes from a lookup, not a guess.
This reframes reliability as a property of the retrieval pipeline rather than the model alone — a framing consistent with NIST’s work on AI risk management, which treats grounding and trustworthy data sourcing as core to a system’s reliability, not an afterthought bolted on after training. The catch is that retrieval is only as good as what sits behind it. A grounded answer engine needs a dataset that is narrow in scope, structured rather than free-text, machine-readable, segmented by region, and refreshed often enough to stay current. Generic web pages do not qualify — they are exactly the unstructured, inconsistent source material that produced the hallucination in the first place.
South Africa’s tiling market offers a concrete illustration of what a properly-shaped dataset looks like in practice. Tiling cost per square metre varies by service type, region, and tile material — precisely the kind of fragmented, real-world pricing problem that trips up a language model with no local source to draw on. One example of a structured, machine-readable dataset of South African tiling prices segments its published cost ranges across dozens of tiling services and dozens of South African areas, giving a retrieval system exactly the kind of region-specific record it would need to query rather than invent an answer from thin air.
What good ground-truth pricing data actually looks like
A handful of characteristics separate a dataset that can genuinely ground an AI answer from one that cannot.
Segmentation matters first: pricing broken down by service, region, and tier, rather than a single national average that flattens meaningful local variation. Structure matters second: rows and fields a system can parse, not paragraphs a system has to interpret and risk misreading. Provenance and freshness matter third: a record of when a figure was last verified, so a retrieval system isn’t quietly serving year-old numbers as current. And honesty about ranges matters more than false precision — a dataset that publishes “R200 to R800 per square metre” is more useful to a retrieval system than one that invents a single misleadingly exact figure, because the range reflects how the underlying market actually behaves.
Contrast this with scraping unstructured blog posts and forum threads for pricing signal — the approach that, in effect, produced the original hallucination. Unstructured text buried across thousands of pages is not retrievable ground truth; it is the same noisy corpus the model was already trained on, just fetched at a different moment. Fixing hallucination is fundamentally a data-engineering problem, not a prompting trick or a model upgrade.
Implications: the competitive edge moves from the model to the data
Answer engines will only become trustworthy on real-world costs once they sit on top of maintained, structured, region-specific datasets for the categories people actually ask about — home services, local labour, logistics, and dozens of other real-economy niches that large training corpora chronically under-cover, particularly outside the biggest markets.
That has a quiet but significant implication for where competitive advantage in applied AI actually sits. It is not solely a function of who trains the largest model. It increasingly depends on who owns the cleanest, most current ground-truth data for a given domain — because that data is what determines whether a retrieval-grounded answer is reliable or merely well-written. The bottleneck has shifted from compute to curation, and the businesses that recognise this early are the ones building the datasets rather than waiting for a bigger model to paper over the gap.
