By This post on LinkedIn by Britney Muller is, IMHO, the most important short piece on AI you’ll read this year. It’s so critical to understand what she’s saying here that I’ve focused this week’s 5-Minute Whiteboard on explaining and demonstrating exactly the problem she’s so succinctly captured.
If you’ve ever asked ChatGPT, Claude, Perplexity, Gemini, or Deepseek why it gave you a particular answer or how it selected the responses it gave, remind yourself:
- It’s All One System: LLMs use their probability token systems to answer everything–including questions about how it works or why it came up with an answer.
- Retrieval (RAG): The model can list the sources it found in its temporary memory (e.g., “I found this on Wikipedia”) or those it scrapes from Google (yes, ChatGPT scrapes Google using SerpAPI)
- System Instructions: The model might vaguely reference its instructions (“I am told to be helpful”), but it often hallucinates specific rules to satisfy you.
- Foundational Logic (The Black Box): Models cannot access their own neural pathways. They cannot say: “I chose this brand because weight 4.5 in layer 12 fired” (even though that’s the truth). LLMs do not have access to their own architectural logic for use in prompt responses.
Give this a watch (and if someone you know has been guilty of making bad assumptions about LLM answers, send ’em here, too). 👇
Transcript:
Britney Muller is one of my favorite AI experts in the world (in fact, you should take her Actionable AI for Marketers Course, January 5-30, where she’ll shed light on how these systems actually work). She has a very realistic and scientific understanding of the space and she presents it so well. I think this in fact is the most important AI post I have read yet this year. The reason is because it dispels a notion too many people have a problem with.
Why is this so critical? It’s because we have all seen so many people say, hey, I asked ChatGPT, Claude, Perplexity, Gemini, or Google’s AI mode why it gave me a particular answer. The reasoning that it gave me back told me more things about how it works, and so now I understand the sources that it uses and all this stuff.
But it’s not true. This is false.
You get this weird answer. You think to yourself, why did you say that?
LLMs are not a SQL database that contain a set of information that you can query against. They’re not logging anything internal about like how their systems work. They’re not truth engines. They are statistical lotteries and predictors of next tokens or next words.
Probability machines, exactly what Brittany calls them here. It works like this, right? Essentially, when you say what’s the capital of France, the LLM breaks it down into numbers, into tokens. And then it says, hey, the most likely token based on our model of all the rest of the words around capital of France is Paris.
And so that’s the answer that we’re gonna return almost all of the time. With some scrubbing and some basically editorial input into the model, they’re able to return that a hundred percent of the time. And that’s why you get these replies that seem probably right, especially on obvious stuff almost all the time. The sky is in the LLMs right word model returns blue because blue is the thing that most of the time should be in there.
But sometimes it might say clear or it might say somewhat cloudy because that appears in there. You ask about the sky in Seattle, for example, we at SparkToro, Amanda and I like to call LLM spicy auto complete because that’s kind of what they are and I can prove it to you. So last month we asked people to volunteer, put their information to the survey and help us basically take exactly the same question and ask LLMs over and over and over again, like a hundred times per. And we had more than, six hundred people filled out the form.
We broke them into groups of two hundred each. And then we asked enough so that we got a hundred, responses for exactly the same input. So they all put in the same prompt. I’ll give you an example.
We asked them to input the prompt. What are the best chef’s knives under three hundred dollars for a home cook?
And my answers, the Wusthoff classic eight inch and the made in eight inch chef’s knife. Those were not the answers other people got. In fact, almost no two people got the same list of brands, which of course has implications for whether you should track your brand in there. But when we broke this down, what we looked at is the brand, the number of mentions in total, the times it was ranked number one, ranked number two, ranked number three and other ranks. You can see like it’s all over the place. It’s almost as though ChatGPT is just a spicy auto complete. It’s running a statistical lottery of the next token it should show.
And then when you ask it, how did you determine that these were the best knives?
It gives an answer that is a lie.
It’s not because it’s intending to lie, it’s because it doesn’t know the truth. It doesn’t know anything. The only thing it can do when you ask it, how did you determine these? Is give you the same thing I gave you in the initial response. The statistical lottery of probable next words, next tokens.
But why when a hundred other people ask ChatGPT the same thing, did it show a bunch of different brands? And then it goes off about how best is a subjective word, which of course is a non answer in itself. That’s because the model just didn’t have a predictive thing to say, Oh, well that’s because ChatGPT works this way. Because it doesn’t tell you how it works. It doesn’t even know how it works. It only knows next tokens.
This is the only answer that’s the truth. The most likely token is the thing that’s gonna appear. And yes, granted people are going to jump into their replies and they’re going say, oh, retrieval augmented generation means that it’s not just most likely tokens from the original model, but also ones that are coming from, I grant you that. It’s also crawling the web.
It’s also doing Google searches and that kind of thing. Then returning results from that and using the text from those. Sure. But it’s still fundamentally a probability spicy auto complete machine.
And that’s why this is the most important AI post of the year. Because if you don’t understand this and you don’t understand how AI works, you are going to really fall apart when you start applying it to your professional career or your life decisions.