This week I watched a marketing manager and an account manager build a basic prototype with AI. There was nothing unusual in that. Prototypes are useful, and AI has made it possible for more people to turn an idea into something visible.
What concerned me was not that they were using the technology. It was the confidence placed in what sat beneath it.
Claude was being given incomplete files, partial knowledge and information that was not clear enough to support the claims being made. This was described as training the AI. In reality, the system was being asked to find a convincing path through material that the people using it had not yet understood themselves.
I heard people say, “We feed the data to AI,” as if the act of feeding it somehow settled the question of whether the data was fit to use. The data in front of them was among the poorest and messiest I have seen in 20 years. It did not become more reliable because a model could turn it into a fluent answer.
The language was polished. The foundation was not.
A prototype is not understanding
A prototype can be valuable because it makes an idea easier to discuss. It can reveal a possible workflow, expose assumptions and help a team decide whether a problem is worth pursuing. It is a way to learn.
It is not proof that the problem has been understood.
AI can produce something that looks coherent long before the underlying work is coherent. A screen can respond. A summary can sound assured. A process can appear to move cleanly from one step to the next. The rough edges disappear from view, even when they are still present in the source material.
That creates a dangerous shift in attention. People begin discussing the prototype as if it were evidence, rather than a question made visible. They debate features and presentation while the basic matters remain unsettled: where the information came from, what is missing, which definitions conflict and who is responsible for deciding what is true.
The speed of the output can make these questions feel slow. They are not slow. They are the work.
This is not a problem limited to people without technical backgrounds. Technical skill can hide the same weakness behind more sophisticated language. The issue is not the role someone holds. It is the decision to treat a plausible result as a reliable one without examining how it was produced.
The input is part of the product
There is something oddly comforting about the phrase “feed the data”. It makes the process sound mechanical. Put information in, receive intelligence out.
But data is not a neutral ingredient. It carries the history of how an organisation has worked. It contains duplicate records, missing context, inconsistent naming, old decisions and exceptions that became habits. It reflects what people chose to capture, what they ignored and what they were never able to see.
When those conditions are not understood, AI does not remove them. It absorbs them.
An incomplete file does not announce every omission. A confused category does not explain why its meaning changed. Two conflicting documents do not identify which one should be trusted. The model can still produce an answer because producing an answer is what it has been asked to do.
That answer may be useful. It may also be a smooth restatement of the confusion already present.
The responsibility therefore begins before the prompt. Someone has to understand the source, test its quality and make uncertainty visible. Someone has to know which claims can be supported and which remain guesses. Someone has to decide what evidence would be good enough for the decision being made.
This is not a demand for perfect data. Perfect data rarely exists. It is a demand for honesty about the data we have, the data we do not have and the consequences of confusing the two.
Thinking is still part of the process
The larger trend worries me more than any single prototype. AI is increasingly discussed as if using it were a substitute for understanding the work around it. People talk about models, agents and automation while paying less attention to the quality of the judgement being automated.
There is, of course, an older tool available before opening a prompt. It is called thinking. Its interface is inconvenient, it has no demo mode, and it sometimes requires a conversation with another person.
Thinking does not mean doing everything manually. It means pausing long enough to define the problem, examine the material and recognise what would make an answer trustworthy. It means knowing what the tool is being asked to carry on our behalf.
AI can help with that work. It can compare documents, surface contradictions, organise questions and test possible interpretations. It can widen the field of view. Used well, it can make judgement sharper because it gives people more room to examine what matters.
But it cannot take responsibility for a judgement that nobody has made. It cannot repair missing knowledge that nobody has acknowledged. It cannot tell an organisation the truth when the organisation has not decided what evidence it is willing to face.
The risk is not that AI will make people less intelligent. The risk is that its fluency will make the absence of thinking harder to notice.
The habit worth protecting is not the habit of doing everything by hand. It is the habit of remaining present enough to understand the material, challenge the answer and know the difference between something that works in a demonstration and something that can be trusted.
AI can extend our thinking. It should not become the place where thinking went missing.