When the Answer Changes Daily: How Lohith Reddy Kalluru Helped Rethink Enterprise AI Testing

Kalluru and his colleagues at Hewlett Packard Enterprise worked out a new way to assess how AI systems deal with changing data.  Their work addresses a practical question: how do you test an answer that may be different tomorrow?

This is what makes testing for AI platforms tricky. If you ask an AI a question today it may return a correct answer. Then if you ask it next week it may return a different correct answer. Maybe it is in a different state or data has changed since you last asked. So when testing your AI application those answers you are expecting to return may also change.

Which leads us to the dilemma for engineers who have to test these AI applications. If your test fails, it’s either because the answer is wrong or it could have changed.

Lohith Reddy Kalluru, a cloud developer III at Hewlett Packard Enterprise, faced this issue while working with his teammates to build the enterprise based AI assistant. He played an important role in engineering an automatic evaluation pipeline to keep pace with the dynamic nature of enterprise data and also implement validations effectively.

When the expected answer becomes the problem

Many conventional regression tests use a set of predefined expected results to compare against the application output; for an assistant answering questions about live enterprise systems, these may become out of date quickly.

For example, the number of deactivated devices could be used as a question. At the time the test was designed, the correct answer was twelve, but when the test is taken some time later, the stored answer is still twelve, but the inventory has been updated and several devices have become activated. This would be reflected in the assistant’s answer; when it says fourteen, the data will be correct, but the test will fail.

Another problem comes with language, more than one answer may express the same correct information but just in different words so the validation needs to take into account the change of facts but also the change of how those facts are expressed. One such issue was ensuring programmers could be supplied with something robust upon which they could base their own judgments about whether or not a result was actually incorrect, otherwise they would follow leads that were in fact a fault with the tests.

Computing the answer at evaluation time

The framework addresses the stale expected answers by maintaining executable reference queries that will be executed. These can be SQL queries that access relational databases or pandas code that processes the API responses.

 During the evaluation phase, the reference queries will be executed to get the expected result for the current execution. For example, if the question asked is “how many of each type of device did the customer have last month”, the code in common will be executed to find the appropriate counts of the devices. The count depends on the available data and does not need to be manually written if the underlying value changes.

The language model will generate queries from the question and data structure; it is then the job of the domain expert to verify the filters, joins and aggregations do what was intended. This approach reduces maintenance due to shifting data values. Kalluru’s addition to the evaluation pipeline was helpful in getting the framework to speak to the real-world task of verifying the AI agents. It provided engineers with more solid ground to analyze failures and impact of changes before it hits production.

Making evaluation useful to engineers

The framework combines rule-based checks to verify answers along with the use of a model to detect factuality, semantic correctness and hallucinations. Some questions will allow for the answer to be directly compared to a calculated value. If a question is meant to be answered descriptively we need to check if the answer makes sense and if there is supporting evidence for it.

The judging model receives reference data, and human review provides an additional check on its scoring.. As mentioned before each question will have different criteria when scoring an answer. A number has to be factually correct but if you’re looking for a descriptive answer you should allow for variations of the same thing.

It’s important to understand why it failed as well. Tool calls and traces will help identify if you have an incorrect result or something higher up may have caused your api call to fail or your prompt to guide the assistant the wrong way. This makes for an improved test suite during development and release pushes.

Through these engineering efforts, the team were able to improve acceptable responses up to over 90%. And,  lower hallucinations from over 50% to under 5%. While these outcomes reflect the collective work of the engineering team, Kalluru’s contribution on automated generation and evaluation infrastructure allowed for better testing on how the assistant was performing and being more accurate along with judging efficiency.

A practical contribution to enterprise AI reliability

The work’s significance lies in addressing a recurring operational difficulty: an assistant’s answers and the data used to judge them can both change. Keeping the two aligned is essential if evaluation is to guide engineering decisions.

For Lohith Reddy Kalluru, the project offered an opportunity to solve a critical problem in enterprise AI which is  maintaining accuracy and reliability as business data evolves. The pipeline Kalluru helped build embodies that discipline. It treats evaluation not as a gate before release but as an ongoing function, as integral to the system as the assistant itself. That framing changes how the team thinks about quality.

For enterprise AI to be genuinely useful, it has to be trustworthy over time, not just at the moment it is deployed. Kalluru’s contribution to this project was a concrete step toward that standard, grounding reliability in process rather than optimism.

Publication reference:

“Multi-Dimensional Evaluation Framework for Generative AI Applications with Dynamic Data Sources,” 2026 IEEE International Conference on AI and Data Analytics. DOI: 10.1109/ICAD69378.2026.11608392.