An AI model asked to recommend antivirus software will answer immediately, with total confidence, whether or not the answer is current. Rob Smith, a former Gartner analyst who founded Lionfish Tech Advisors, has tested these tools against markets he knows well enough to be blunt about the risk in that confidence: without a human check on the underlying data, a buyer has no reliable way to know whether the recommendation reflects this month’s market or a version of it that is already six months stale.
That is the practical problem Periscope was built to solve, not by replacing AI but by pairing it with expert review that catches errors before a buyer sees them. Here is how AI-only recommendations go wrong in IT purchasing, what Lionfish’s verification actually checks for, and why pairing the two is the harder but more defensible approach.
KEY TAKEAWAYS
▪ The dangerous AI failure is a real vendor recommended with outdated pricing, a discontinued feature, or a changed owner.
▪ An annual report can be accurate on release and silently wrong for months when a vendor is acquired, breached, or discontinued.
▪ Incentivized reviews inflate scores; Smith trusts negative reviews more than glowing ones.
▪ A market often has more than a hundred vendors; generic AI surfaces only the ten to twenty most visible names.
▪ Periscope pairs AI speed with analyst verification, so buyers act on current, human-checked facts.
The specific ways AI gets IT purchasing wrong
The failure mode Smith describes most is not a model inventing a vendor that does not exist. It is subtler and, in some ways, more dangerous, because it looks correct on the surface. A model trained on public vendor data will confidently recommend a product using outdated pricing, a discontinued feature set, or a company that has quietly changed ownership since the training data was collected. Smith points to the habit of renaming products as an example of how fast public information about even a major vendor drifts out of sync with reality. The underlying product may be unchanged, but an answer tied to a name or feature set that was public at training time can mislead a buyer evaluating current options.
Traditional research is not automatically safer either; it is simply slow to update rather than never verified. An annual report can be accurate on publication day and silently wrong for months afterward, when a featured vendor is acquired, breached, or discontinued, with no mechanism to flag the change until the next cycle. That is the exact failure Lionfish’s daily-refresh model is built to prevent.
Why reviews make the problem worse
Reviews compound the problem rather than fixing it. Many AI systems weight customer reviews as a scoring signal, and Smith is candid that a meaningful share of positive reviews are incentivized, with vendors offering small gift cards for a favorable rating. His rule of thumb, only half joking: the reviews worth trusting are the negative ones, since people rarely go out of their way to praise a vendor they are satisfied with, but they will reliably complain about one that let them down.
What does human verification actually catch?
It catches the errors that look correct. At Lionfish, every data point that reaches the recommendation engine behind Periscope has already been checked by one of the firm’s roughly thirty former Gartner analysts, the same caliber of expert who once had to clear one of the toughest hiring processes in the industry before advising a client. It is not a spot check; it is the standing process behind the Aquarium platform, refreshed at least daily and more often in fast-moving categories like agentic AI.
The practical effect shows up when Periscope tells a buyer their current vendor is a poor fit, or recommends a smaller, regionally specialized company they would not have found through a mainstream search. Brad LaPorte points to this as one of the clearest advantages: a market often has well over a hundred vendors, and a generic AI answer tends to surface only the ten or twenty most visible names, missing the regional or specialized players that might be the better fit for a specific buyer’s industry, language, or compliance requirements.
Why the pairing matters more than either piece alone
Smith does not frame this as AI versus human expertise. He frames it as AI without expertise being an incomplete tool for high-stakes decisions. The AI layer is what makes Periscope fast: a report that once required a scheduled call now takes under a minute. The human layer is what makes it trustworthy enough to act on: an analyst has already confirmed the facts are current before the AI assembles them into a recommendation.
For a real purchase, that combination addresses two risks usually treated as one. Speed alone, from an unverified tool, risks a confidently wrong answer. Accuracy alone, from a traditional call, risks a multi-week wait that does not match a procurement timeline. Pairing the two is a harder operational problem than either piece by itself, which is part of why it took a team with this specific background to build it. For a decision with real budget attached, the combination is what holds up to scrutiny.
FREQUENTLY ASKED QUESTIONS
How does AI recommend the wrong software?
Usually not by inventing a vendor, but by recommending a real one with outdated pricing, a discontinued feature, or an owner that changed after the training data was collected, all of which look correct on the surface.
What does Lionfish verification catch that AI misses?
It catches stale or marketing-shaped facts and surfaces regional or specialized vendors a generic AI answer skips, since a market can have well over a hundred vendors and AI tends to name only the most visible ones.
Why pair AI with humans instead of choosing one?
AI alone is fast but can be confidently wrong; a traditional analyst call is accurate but slow. Pairing them gives buyers speed and verified accuracy on the same recommendation.
THE BOTTOM LINE
AI will always answer instantly, and it will sometimes answer wrong while sounding exactly as confident either way. The fix is not to unplug it. It is to put an expert between the data and the recommendation, so the speed is real and the answer is current. For a buying decision with real money attached, that pairing is the part that survives scrutiny.
See what human verification catches at lionfishtechadvisors.com, or run a recommendation at periscope.fish.
SOURCES & REFERENCES
- Lionfish Tech Advisors https://lionfishtechadvisors.com
- Periscope https://periscope.fish
