I remember about six months ago being in a board meeting with my MBA alma mater, and our school Dean told a funny story about using ChatGPT. She prompted the AI to create a detailed profile of her academic and professional career using publicly available data. ChatGPT produced some accurate facts alongside several interesting inventions about her background, citing schools she never attended and public employment positions she never held. Before explaining which claims were false, she showed us ChatGPT’s original response. Because I didn’t know much about her background, I assumed the response was accurate. My first thought was that the AI must have mixed her name with someone else, but her name is very distinctive, making a simple identity mix-up unlikely, and she is the only one in the upper echelons of university academic administration. Also, accurate information about her was readily available online, so this did not appear to be a simple case of garbage in, garbage out. Clearly, the AI hallucinated. Even as an AI specialist, I was fooled because the response seemed so authoritative. I can only imagine how easily the average person, who may not be familiar with the intricacies of AI technology, can be fooled.
What matters in cases like the Dean’s search using generative AI models is that blaming the AI model alone is incomplete when it comes to hallucinations. Ultimately, responsibility lies in how the system is designed and in the engineering teams behind it. But before getting into the challenge of dealing with hallucinations in a system, one must challenge assumptions. Specifically, whether hallucinations are a problem in the first place or an inherent property of AI systems. This changes how we approach system design. If we are to go with the conventional discourse that AI is an extension of human intelligence, then is human intelligence not prone to reliability issues or what I personally call “involuntary hallucinations”? Human beings and their accompanying intelligence strive for perfection but have yet to achieve it. For example, when an attorney consults with a client, both parties may work from incomplete information, assumptions, and bad memories. Both parties most likely are not conscious of “involuntary hallucination” unless they choose to deliberately lie.
With this in mind, let us return to the challenge of engineering AI systems. It is important to balance the system’s functions with the expected output. Is the function of an AI system to respond with 100% accurate answers? If so, the system stops being AI and becomes more like a traditional, data-driven, deterministic system. The point is that hallucinations are inherent to AI and can be managed by focusing on components like system prompts, retrieval, guardrails, workflows, monitoring, and human escalation. In other words, the goal isn’t to deliver models directly to the user, but to build well-designed components around them. By tweaking these components, we can change system behavior and run regression tests to catch behavioral failures. This must happen before deployment to identify behavioral failures and should include real-world edge cases, high-risk questions, and checks that the surrounding safeguards respond correctly. Before production use, the system needs more than a successful demonstration. It must be evaluated across its target use cases, edge cases, high-risk scenarios, and known failure conditions. During production monitoring, if a hallucination is discovered, it should leave behind a regression test so that the same failure is not reintroduced in future updates. Although automated monitoring has limitations, it remains critical for determining appropriate system behavior and driving engineering work. Another critical monitoring step is ensuring hallucinations are minimized or do not cause safety issues, especially in regulated industries such as healthcare and legal services. It is human review. Think of checks and balances. Intelligent systems need independent review by qualified people who can challenge their outputs and recognize errors that automated monitoring may miss. Hallucinations can be subtle, and domain experts are often better positioned to spot inaccuracies, context issues, or flawed conclusions.
One misconception about AI system design is that the system must answer every question or prompt. In well-designed AI systems, it is often better to ask clarifying questions when the available context doesn’t meet an established confidence threshold or poses a critical risk. It’s like jumping to conclusions without understanding the full context. This is a common human trait, but we can do better by following traditional design principles, like asking clarifying questions, which help the AI system converge on a more probable response, build more confidence, and minimize hallucinations.
We accept that hallucinations in AI systems may never disappear completely, but we can control how often they occur in user interactions, how much authority they are given, and what happens when the system detects uncertainty or unexpected behavior. That is why hallucination risk is ultimately an engineering and accountability challenge.
