AI Isn't What You Think It Is

If you've read our previous investigation, you may have been left with one simple question.

How could an artificial intelligence system become so confidently wrong?

It's a fair question. But before we can answer it, we need to understand what AI actually is. Because the AI most people imagine is not the AI they are using.

For decades, artificial intelligence has been part of our culture. Films portrayed it as a conscious machine. Television gave it a personality. Science fiction imagined computers that could think, reason and understand the world much like we do. Magazines described the coming age of intelligent machines. Even the name Artificial Intelligence encourages us to think of it as a digital mind.

Those ideas have quietly shaped our expectations.

Then we open ChatGPT, Gemini, Claude or another AI assistant. It greets us politely. It remembers parts of our conversation. It apologises when it makes mistakes. It explains itself. It sounds intelligent. It sounds friendly. It sounds confident.

Without realising it, our brains begin treating it like another intelligent person. After all, this is exactly how knowledgeable people communicate. They answer questions. They explain themselves. They admit mistakes. They apologise.

Our subconscious has spent a lifetime learning that these are signs of understanding. So we naturally extend the same assumptions to AI.

The problem is not that AI talks like a human. The problem is that we often assume it thinks like one too.

It doesn't.

The first surprise is that AI doesn't begin by asking:

“Is this true?”

Instead, it begins with a very different question:

“What is the most likely response to this prompt?”

Those two questions sound similar. They are not. Truth and likelihood are different things.

That single distinction explains almost everything that happened in our investigation. Most of the time truth and likelihood overlap. Sometimes they don't.

When they don't, AI can produce an answer that sounds entirely believable while having little or no evidence behind it. That isn't because it has decided to deceive you. It's because it has done exactly what it was designed to do.

Generate the most likely continuation of the conversation.

Imagine I asked you to complete this sentence.

Fish and ______

Most people would answer: chips.

Not because it's the only correct answer. You could have said salad, rice, vegetables or peas. They're all perfectly reasonable. But “chips” is the response you've encountered most often throughout your life.

Your brain isn't searching a database. It's recognising a familiar pattern.

Modern AI does something remarkably similar. Except it doesn't stop at one word. It builds an entire conversation by repeatedly predicting what is most likely to come next.

That's why AI can write essays, explain difficult subjects, generate computer code, write poetry, translate languages, summarise books and draft business plans.

It has become extraordinarily good at recognising and reproducing the patterns of human language.

But understanding language and understanding reality are not always the same thing.

There is another important distinction.

AI doesn't automatically know when it is right.

In other words, AI can become extremely good at producing an answer without possessing any independent awareness that the answer is correct or incorrect. It generates an answer.

Knowing the answer and generating an answer are two different things.

The answer may be correct. It may be partially correct. Or it may simply be the most likely continuation based on everything it has learned before.

As humans, we are easily persuaded by confidence. If someone speaks fluently, uses technical language and presents information clearly, we instinctively assume they know what they are talking about.

AI has learned those same patterns of communication. It knows what confident writing looks like. It knows what expert explanations sound like. That does not necessarily mean the underlying information has been verified.

Now think back to the investigation. The AI sounded knowledgeable. It sounded certain. It sounded as though it had inspected the website.

But now you know something you didn't know before.

Sounding knowledgeable and possessing verified knowledge are not the same thing.

Everything that happened next began with that distinction.