The problem
Ask a capable language model a business question — market size, competitor pricing, regulatory status, whether a supplier is creditworthy — and it will almost always answer. The response reads fluently, uses the right vocabulary, and rarely hedges. That fluency is not a signal of accuracy. It is a signal that the model was trained to produce plausible continuations of text, and a plausible-sounding answer is exactly what plausible-continuation training produces, whether or not the underlying facts hold up.
The practical problem is that a fluent answer and a supported conclusion look identical from the outside. Nothing in the output format distinguishes "I found this in three independent primary sources" from "this is the statistically likely thing to say next." A reader without domain expertise — often the exact person relying on the tool to compensate for not having that expertise — has no way to tell them apart. The interface gives no friction, no visible seam, at the point where trust is actually being extended.
What the evidence suggests
The pattern shows up wherever a system is asked to answer beyond what its training or retrieved context actually supports. It is well documented that language models will provide specific-sounding figures, cite sources that do not say what the answer claims, or state a regulatory position confidently when the honest state of affairs is "unclear" or "depends on jurisdiction." The underlying cause is structural, not a bug that better prompting fully removes: these systems are optimized for likely-sounding text, and confident phrasing is more likely-sounding than hedged phrasing, independent of whether the content is correct.
Retrieval-augmented approaches reduce but do not eliminate the gap, because retrieving a source is not the same as verifying that the source actually supports the specific claim being made, or that the claim is not contradicted elsewhere. A system can cite a real document and still misstate what that document says. The presence of a citation makes an answer look more trustworthy without making it more likely to be true — arguably a worse failure mode than an unsupported claim, because it disarms the reader's remaining skepticism.
What remains uncertain
How much this actually changes decision quality in practice, as opposed to being a theoretical concern, depends heavily on the domain, the base rate of correct answers a system would give anyway, and how the output is actually used downstream — read and second-guessed by an expert, or acted on directly. There is no general answer to "how often does this matter," and any claim of a precise number without naming the domain and the decision it applies to should be treated with real skepticism.
It is also unresolved how well any of this generalizes across model providers and versions — behavior here shifts with model updates in ways that are not always documented, so a specific failure mode observed today may already look different in six months, in either direction.
Practical implications
For a business, the cost of acting on a plausible-but-wrong answer scales with the size of the decision and how far downstream the error propagates before anyone checks it — a wrong number in an internal slide deck is recoverable; a wrong number that goes into a pricing decision, a supplier contract, or a market-entry commitment is not always recoverable, and is often not caught until the outcome itself is bad.
This is the practical case for evidence-first design: a system that separates what the evidence directly supports from what is a reasonable inference from what is not known, and makes that separation visible rather than folding everything into one uniformly confident paragraph. It is also the case for building in the ability to say "insufficient evidence to answer" as a first-class output, not a failure state to be avoided at all costs. A tool that never abstains is a tool that will eventually be confidently wrong about something that matters.