The Stack Overflow Developer Survey for 2025 asked 49,000 developers whether they trust AI tools. Forty-six per cent said they actively distrust the accuracy of AI output. Only three per cent said they highly trust it. In the same year, 84% of those developers were using or planning to use AI tools anyway. We are building on technology most of its own users do not believe.
That distrust is not paranoia. It is experience. And it is not just about hallucinations. It is about the quieter problem: the model seemed fine last month, and now it is not, and nobody noticed until a customer pointed it out.
LangChain’s “State of Agent Engineering” report, published in June 2026, surveyed 1,340 people building with agents. Eighty-nine per cent had observability in place. Only 52.4% ran offline evaluations on a fixed test set. Just 37.3% ran online evaluations. And 29.5% did no evaluation at all. For teams whose agents were already in production, the no-evaluation figure was still 22.8%.
The report is a vendor survey with a self-selected sample, mostly from tech and smaller companies, so treat it as a strong signal rather than a census. Even so, the gap is striking. Almost everyone is watching. About half are testing. Nearly a third are flying blind. And 32% of respondents named output quality as the single biggest obstacle to shipping.
The reason this keeps happening is that model updates are not always announced the way software updates are. OpenAI’s own model page explains that snapshots exist “so that performance and behavior remain consistent.” The implication is that without a pinned snapshot, performance and behavior are not guaranteed to remain consistent. That is exactly what researchers have measured.
A widely cited study by Chen, Zaharia and Zou at Stanford and UC Berkeley tested GPT-4 in March and June of 2023 using the same prompts. The share of code-generation solutions that were directly executable dropped from 52.0% to 10.0%. Accuracy on a prime-versus-composite classification task dropped from 84.0% to 51.1%. The authors’ summary is the memorable line: “when and how these models are updated over time is opaque.”
A separate study presented at CAIN 2024 found the same mechanism in a different guise. When a provider swapped one model for another behind an API endpoint, the accuracy of a toxicity-classification task changed enough to flip which prompt was optimal. The swing was 8.7 percentage points. Prompt engineering from last quarter was no longer valid this quarter.
So the model you tested in January may not be the model answering your customers in March, even if the API name is identical. And failure mode is not always dramatic. Often it is a slow drift in tone, formatting, compliance with instructions, or edge-case behaviour that your monitoring dashboards do not capture because your monitoring only checks latency and cost.
This is why evaluation is harder than observability. Observability tells you that a request happened. Evaluation tells you whether the answer was right. And “right” is not the same as “plausible.”
LLM-as-judge is the obvious shortcut. It is also where a lot of teams go wrong. The original MT-Bench paper from NeurIPS 2023 showed GPT-4 as a judge agreed with humans over 80% of the time. That sounds like a passing grade until you read the later work. Wang et al. showed that simply swapping the order of candidate answers made Vicuna-13B “beat” ChatGPT in 66 of 80 questions when ChatGPT was the judge. Ye et al., published at ICLR 2025, catalogued twelve separate categories of judge bias. And Dorner, Nastl and Hardt, also ICLR 2025, proved that even 85% human agreement can still produce a completely wrong model ranking at the frontier.
The point is not that LLM judges are useless. The point is that a judge is another model with its own drift. If you are not benchmarking the judge against human labels on a representative sample, you are measuring something, but it is not quality.
The benchmark problem is worse than the judge problem. Public leaderboards are useful and heavily gamed. “The Leaderboard Illusion,” a 2025 preprint by Singh, Kapoor, Smith and Hooker, found that Meta privately tested 27 Llama-4 variants before release and could retract scores. They also found that Google and OpenAI each received roughly a fifth of the Arena data used to train and tune their models, while 83 open-weight models combined received about 29%. A leaderboard becomes less useful when the people being ranked also help write the test.
Apple’s GSM1k dataset, published at NeurIPS 2024, made the contamination point concrete. It was constructed to be distribution-matched with the widely used GSM8k benchmark but held out from training. Mainstream models dropped by up to 8% on the fresh set. The score that looked solid on the public benchmark did not hold up on a near-identical, unseen one.
Then there is non-determinism. Two recent papers tested whether setting temperature to zero makes outputs reproducible. It does not. Atil et al. found that even with deterministic settings, accuracy varied up to 15% across repeated runs, and the gap between the best and worst run reached 70% on some tasks. Ouyang et al., published in ACM TOSEM, found that the share of coding tasks where the exact test output could not be reproduced ranged from 47.56% on HumanEval to 75.76% on CodeContests. You cannot regression-test a feature if the same input gives different outputs.
What is surprising is not that any of this exists. It is that so many teams accept it. We would not ship a database migration without a rollback plan. We would not refactor a checkout flow without A/B testing. But a model swap that changes the behaviour of every AI-generated response in the product often ships with nothing more than a spot-check of five prompts.
The fix is boring. Pin snapshots. Build a golden dataset of real user queries with human-rated correct answers. Run it before every model change. Compare outputs across multiple models on the same inputs, not just the new one against itself. Keep a human reviewer in the loop for the subset of outputs that matter. And treat the evaluation pipeline as production infrastructure, not a research notebook.
That is the practice MuseSpark AI is built around: run the same prompt against multiple models, compare the outputs, and choose the one that actually passes your test. The model with the best benchmark score is not always the one that best follows your instructions. The only way to know is to measure.
The cost of not measuring is real. Air Canada lost a small-claims case in 2024 because its chatbot invented a bereavement-fare policy that did not exist. The tribunal’s language was blunt: “Air Canada did not take reasonable care to ensure its chatbot was accurate.” That was an accuracy failure on day one. A drift failure three months later looks identical to the customer. The difference is only on the engineering side.
If you do not know whether your AI is getting better or worse, it probably is getting worse. The only question is whether you will find out from your dashboard or from your customers.
