Office Hours: Can You Actually Track AI Visibility?

Hi, that guy. So as you might know, we recently did some very interesting research, with a company called Gumshoe, who’s also based here in Seattle. They’re in the AI tracking world, and I was skeptical that AI tracking was even a possibility. Because as as you probably know, if you ask an AI, to give you a response to to virtually anything, It will give you a different one every single time, and this is particularly applicable when you’re asking for a list of brands or recommendations. So what how can you possibly track something that gives you a different answer every single time? That’s that’s the rabbit hole we wanted to go down. My worry was that marketers are getting fleeced because there is a lot of hype around AI. I’ll I’ll show you in a sec. And I I was just worried, truly, truly worried that people are spending tens of thousands, hundreds of thousands of dollars of budget on tracking their AI presence in a world where it does not make sense or matter and instead instead of investing in marketing things that do matter. And I still believe there’s some of that happening, but maybe a little bit less than my original hypothesis was. So let’s go kick off. I bet you’ve seen research like this. I’m not gonna mention the company behind this because I don’t believe their data. I think it’s sketchy. I I left a LinkedIn comment saying like, hey. This doesn’t this doesn’t look real to me. I think you I think you’ve not made this up, but squinted and made things motivated thinking wise. I bet you’ve seen research like this. I love this one. It’s only gonna be four more years until ChatGPT overtakes Google. Yeah. Sure. That sounds that sounds right. And then they say data source similar lab, but you see those dots? Those are all projected. Also, the little drop in Google, total lie. I don’t know why they included that. Or even this one, which I think this one might be true. This is from Ahrefs showing that visitors who have the refer string of coming from an AI tool have four to twenty three times the conversion rate. And I believe this because AI tools send one tenth of one percent of the traffic that search engines do per search. And so it is only the one in one thousand person who is, like, absolutely directly bought in. And oftentimes, you have to say the AI tool, okay. Thanks for the recommendation. Can you give me a website? And then when it gives you the website, you can say, please turn that into a link for me. And then it will give you the actual link to go. And so I I do sort of believe this, but it’s it’s very, again, a little motivated thinking there. So I wanted to dig into how big AI search is. Really, granted this is desktop, but as you might be well aware, seventy five percent of ChatGPT’s activity is desktop, whereas sixty six percent of Google’s search activity is mobile. So we are way under counting Google in this for sure, and we’re almost certainly over counting AI tools. And still, when you look at the share of visits, this is visits, right, not not searches, but visits, all AI tools combined to two point nine percent of visits, and search engines, thirty four percent. So, you know, look, there’s lots of things happening on the web, but AI is still relatively small when you think about it globally from a visitation perspective. We also did this with searches. So this was recent research that we did with Datos, our clickstream data provider. Again, desk desktop, so it’s gonna have an AI bias in their favor. And what we looked at is the number of prompts and number of searches. Now we we treated searches and prompts the same. So if you had a long prompt session where you asked tons of prompts or you had a long search session where you click through tons of results and went to tons of pages, those are treated equally. The answer here is AI tools, they have some searches, but they they don’t have as many searches as social networks do or as many prompts as there are searches happening on on Instagram or Reddit or Facebook or LinkedIn, all those places. And yet there’s no hype around that. There’s no hype around social search. There’s very little hype around ecommerce search, commerce platform search, and traditional search engines still dominate. Google, by the way, is, like, seventy three, seventy four percent of that eighty percent. So we we we looked at this trend over time to to try and see what was going on here, and you can see Google’s comments really skews this. So I I broke it down. Now I’ve made the the y axis start at seventy percent. I don’t wanna fool you with visuals like lots of people do with charts, but I’m just taking let’s take this top seventy percent here and just look at that, and you can see how, you know, Amazon, Bing, YouTube are growing their share. Google is shrinking their share very slightly, and ChatGPT share is also shrinking because ChatGPT, as you probably know, stopped growing middle of last year. Right? They started plateauing, and then we saw a slight decline in usage. That’s probably continuing as people are switching to Claude and Google AI mode and other things. I I have some skepticism about this whole field. So, we looked as well at the percent of visitors who performed a search, and I just wanna point out that when someone shows you a chart like this one, right, so this this, like, here’s how many people visited, you should make sure you counter with a ninety seven percent of people who go to Google perform a search. Fifty seven percent of people who go to ChatGPT perform a search. Twenty two percent who go to YouTube perform a search. Right? So, like, the the intent there and you might say, wait a minute, Rand. That doesn’t make any sense. How is it possible? I’ll I’ll show you how it’s possible. I always get this question. Wait. Let me grab my browser. Yep. That one. Oh, la. That’ll make it easy. In the in the chat, Nicoletta asks, when you say perform a search on ChatGPT, do you mean any prompt? Yes. Any prompt. Right? Any any prompt doesn’t right. So if I perform a if I go to my best men’s dress shoes, which I’ve been researching, you probably tell, and then I say at the bottom here, hey. You know what? I want to share this with someone, and I copy the link and I send it to somebody, which my mother-in-law does all the time on WhatsApp. She’s like, oh my god. It’s alive. Look. I figured out how to make the computer alive. And then, right, she sends it to me. That’s her accent, by the way. Perfect perfect representation. So, and she and then that’s how you get tons of people visiting ChatGPT without ever she’s not she has an Italian accent, sort of, but she sounds more like Ariana Huffington. I don’t know why. Like, she sounds like no other Italian. That’s why you see that fifty seven percent number. Okay. Okay. So a lot so Also fifty seven percent, a lot of people are sending their chats to other people. Like, hey. Check this out. Is that mostly it? And then I guess other people, they might be looking for something on the OpenAI interface or ChatGPT interface? That’s possible. Yeah. I would guess that’s a probably a smaller percent. I I do think, Amanda, I do think there’s a large percent of people who go to ChatGPT because they’ve heard about it on the news or from a friend, and they stare at it and they go, what am I supposed to do? And then they leave. And you can see in the visitation data that there’s tons of, like, one page visits to ChatGPT who don’t come back, you know, that month again. So That that would be my guess. Right? That there’s lots of that. There’s lots of sharing, all that kind of stuff. Okay. I don’t know if you saw this one, Amanda. I I guess you would like I I thought you might like this. This was an NBC poll. You know, they do these every week or every month. Right? And then they they publish the results. They’ve been doing these for, god, decades, right, since I was a kid. And so they ask how popular certain people are. Right? So, like, they they asked this poll of how popular is Paul Pope John Paul the second in nineteen ninety eight, you know, prior prior to Seanan O’Connor going on SNL and revealing his activities. And you can see that artificial intelligence, AI, minus twenty, right, when you take the somewhat negative, very negative, subtract them from the somewhat positive and very positive, That is that is lower than Trump or ICE or either political party in the US. AI is extremely, extremely unpopular. And I am shocked. I think I think this is the most shocking number. Only one percent of Americans said they were unfamiliar. What one percent of Americans are unfamiliar with AI. Like, that’s that’s Surprising. Way less than are unfamiliar with Iran, the country with ninety million people in it. That’s crazy. Alright. So more people know what AI is than Iran? Yep. Yeah. Okay. Well Yep. For America, that sounds about right. It’s weird. We’re in that geography. AI’s impact on search, I don’t wanna underplay it. I just I’m just trying to show you that the AI tools are quite popular with a very select subset. It’s it’s the people who are on this webinar in your LinkedIn feed, in your professional world. All of us are sort of obsessed with AI because the executive class of the US is obsessed with AI, and they’re spending tons of money on it. But that is not where AI’s true influence is being felt by most people. Most of the influence and impact in in my view and from all of our research is Google. It is because eighteen percent of Google’s search results now show an AI overview. That is that is a fifth of five trillion searches a year. That is impacting almost everyone on the planet because almost everyone uses Google. Almost every one of us performs, you know, an average of ninety to a hundred searches in a month. And and there you go. Right? Twenty of those searches, eighteen of those searches contain an AI overview, and you can see by industry. If you’re in science or health or pets or, you know, people results, right, how old is Paul Rudd, like, those are they are showing way more frequently. And, you know, maybe if you’re in shopping or real estate, less so. So AI, huge impact. And that brings us to our research on AI consistency. We care about this stuff. Fifty percent because our execs are obsessed and fifty percent because of Google’s AI overviews, and maybe one percent, two percent because of ChatGPT, right, and and Claude and Perplexity and DeepSeek and all the other models. Alright. Let’s dig into how these work. Alright. If you were to go to ChatGPT and ask for a list of brand recommendations, what are what are the top chef’s knives? A hundred times. How many different answers will you get? That was the that was the core of the question that we were trying to answer so that we could figure out whether you can track AI visibility. And, yes, Robert is correct. The AI is not correct, but Robert Karnes, knife genius, knows what’s up. We solicited six hundred volunteers. Probably some of you on this call, we truly appreciate your and their labor because I sent email after email in q four to people saying like, hey. Sorry. I need you to run another, you know, bunch of prompts and, like, will you try this and will you try that? And so we we collected all these through Google Forms. We asked people to essentially run the same prompts over and over and then submit the answers that they got back, just copy and paste them in here. And then we did well, Christy Christy Morrison, my chief of staff, did an incredible job of normalizing this data, which meant making sure that, you know, if it said the Victorinox Pro, which sometimes it would, we knew that that meant, oh, that means the Victorinox Fibrox Pro, and let’s, you know, make sure that those two answers are the same. So normalizing all this data. But you can see the the variety of responses. Absolutely astounding. Right? So this is essentially when you ask for the top chef’s knives, here’s all the brands that were mentioned. Here’s when they were the first brand to appear, the second, the third, other ranks that they appeared in, the total number of mentions which we sorted by just wild wild amounts of variety. And after creating and distributing and then and then analyzing thousands of these responses, we sent five different surveys across three months. And I’m gonna try and succinctly summarize that for you to show you what’s up. So this is the number of unique brand mentions. Right? So this would be like the Victorinox or or let’s let’s use, you know yeah. We’ll we’ll use chef’s knives. That’s fine. So the Victorinox versus the Mac versus the Wusthof. Right? That’s that would be three different brands that were mentioned. But if if there were seven models mentioned from Wusthof, we we still normalize that as a a Wusthof mention brand mention. So we’re not counting individual models. We’re counting brands here. And you can see that it varies dramatically based on what you ask. Right? So in LA Volvo dealerships, turns out in the greater Los Angeles area, there’s only eleven Volvo dealerships or eleven dealerships where you can buy a Volvo. Only nine of them were ever mentioned by any AI, which I thought was interesting. There’s, like, two invisible LA Volvo dealerships to AI, in the in this, which, you know, I mentioned to Darren Shaw that might be an interesting case to look at. Like, why are they invisible to why can Google Maps see them and AI can’t? And, this and then the the pink dot is the average number of mentions. Because remember, you know, unlike this is one of my problems with an AI tracking tool, like a ProFound or even Gumshoe or other ones, right, is that when they prompt AI, they say, what are the five top Seattle real estate companies? What are the five best, you know, US politics accounts to follow on TikTok? They they have a number that they assign to get a specific number of responses, but nobody else prompts AI that way. When we looked at how people really prompt AI from the from the open source data that we have well, from from Datos, nobody uses a number. Nobody says, give me top five, top ten. They just ask. So you can see those response number of responses bouncing between two and twelve, you know, just all over the place. And chat GBT, Claude, and Google AI, which we used AI overviews unless the overview didn’t show, in which case we went to AI mode, then you you can see what those look like. So, like, for example, nutrition accounts, way more diversity of accounts mentioned. Sci fi novels, incredible amount of diversity, especially by Google. Google gave huge lists, massive lists, more than more than ten on average, and more than two hundred different sci fi novels were mentioned. Whereas LA Volvo dealerships, right, the the numbers were quite small. Alright. There’s the there’s the numbers for you. How consistent were these? So we we calculated based on all these, you know, data points and all these different ones that we did. We calculated the odds that the same list of brands would show up. So, like, if it’s the Wusthof, the Mac, and the Victorinox, what are the odds that that will show up in two responses out out of a hundred? Well, them odds are low. This this it it’s not gonna happen. Right? Like, Google a m I AI mode, less than one percent shot. Right? So one, you would have to ask Google’s AI mode a hundred and twenty four times to get the same two brands or the the same list of answers twice. I I I want you to take this graph and that stat to all your meetings where people are like, we need to track our AI rankings. Oh, boy. Oh, boy. I I I know this seems crazy, but if you try it yourself, you will quickly see this. Go go ask an AI right now for, I don’t know, Amanda, what are you shopping for? A new microphone? A new microphone. Yeah. Actually is a good example. Yeah. I bet if you go ask ChatGPT for, hey. What are the best microphones for webinars and podcasts? And you ask that ten times, every single one of the answers you get will be different. It’ll be different in the brands that are mentioned. It’ll be different in the order that’s mentioned. The visualization will be different. It’s just different. And by the way, if you want the same brands in the same order, fourteen hundred and twenty nine times before you are statistically likely to get to. Crazy. This is why you have to be careful trusting AI visibility metrics. I’m not say I’m gonna show you some ways to do this, and and that there is a number that I think is reasonable. But, man, if you are just running some searches for example, a lot of agencies tell me that people will email them. They’ll send them like a chatty p t, you know, screenshot or or a link to the responses and they’ll be like, hey, you don’t show up in AI. And and sometimes this convinces a team that they don’t show up in AI. I’m like, dude, just ask again. You’ll show up. You you can you can run it until you don’t you get or don’t get what you want. The AI response consistency, we we put this through the different prompts that we put in, and you can see that there are some where it’s more and less likely. So the LA Volvo dealerships example. Right? Like, that prompt consistency is granted still very low, but way way higher than, you know, home studio microphones, which was which was one of our one of our questions or sci fi novels, right, where it’s vanishingly unlikely. Claude, we did the same thing. Google AI mode, we did the same thing. I this is one of the examples that really stands out to me, friends. And I think this is a little bit less about how you use it in your marketing career and a little bit more about how you use it in your personal and and sort of research life Which is that lots of people trust Google’s AI for medical advice. Right? Like, we ask we ask Google’s AI mode or we ask ChatGPT, and we’re like, hey. You know, my friend was just diagnosed with cancer. Where where should they get care? Where are the best hospitals? And you think you’re getting a great answer, but what you’re getting is a is a somewhat randomized list. If you ask again, you’re gonna get a different list of hospitals in a different order. I I’m not sure that’s the right thing the the right thing to be asking. Right? It just makes me very worried. I get nervous. I mean, just to take the kind of smaller stakes example of the home microphone, I mean, that’s going to vary a lot based on which websites you would go to for that opinion. Right? Because people want different things in microphones. If you’re looking for a home microphone, you might be prioritizing noise reduction, like external noise reduction. Or maybe you’re prioritizing the sound of your voice in the microphone. Or maybe you’re thinking about podcasting versus creating music. Like, these all have very, very different use cases. So that’s already gonna be different when you look at different websites. But then add to that the, I don’t know, the the the lottery machine of LLMs, it’s gonna be even more variable and unpredictable. And the one thing about going to a website is you know who’s behind it. They’ll they’ll say why they have their opinion or they won’t, and then you you judge appropriately. They have criteria that they list. The LLM has no criteria. Right? It doesn’t say, like, oh, we weighted, you know, survivability of cancer patients versus comfort of care versus number of doctors to nurses ratio. They they don’t tell you what their criteria are. There are no criteria. The criteria is, did you appear in the training data, you know, this many times and in the citation alongside you know, words alongside other words? It’s it it just bugs me. Right? It feels like to be honest, Amanda, to me, AI responses on a lot of the stuff feel like Internet one point o, and Google feels like two point o. Right? Like, if AI had come out first in the nineties, I think we’d all be like, oh my god. It’s so much better now that I just type one or two words and get, like, a list of more trustworthy, you know, responses that I can go investigate. Like, it feels it feels like AI to me feels like ask Jeeves in nineteen ninety nine. Like, it really it really does. I know I’m dressed like him, but, you know, that’s exactly what thought. Sorry. I don’t wanna hold this up, but I do wanna make another point here about intent. So intent when people are searching on these tools. Because I’m also remembering, you know, after you had your study, which we’re gonna link to I’ll link to right now, but we’re also you know, it’s been out. We’ll also link to it in our follow-up email. But since your research, I I was then thinking about, like, what if I ask other mom friends a similar question? So this was my my very, very small scale experiment of, like, okay. These people are pretty similar to me demographically. Right? We have similar sort of values. What if I ask them to look up local basketball leagues? Now these are friends near me, but also friends across the country. So and I wanted to see all of their prompts. I have a blog post on this too. And so what was interesting to me was just the the very different ways in which we framed it. Some friends really prioritized, you know, locality, like, low location. Like like, I want the closest, you know, league to me. Other peep someone else was like, I really want one that’s gonna nurture my son. Right? And so people can still get the same answer. Right? These are people who are near each other. Let’s say one person really values, you know, closeness. The other person values the the coaching, the quality of the basketball league. Right? They might go to the same league, but they’re both still going to have different intents, different values. Right? They’re still gonna be like, oh, I chose this league because it’s next door to my orthodontist. Someone else is gonna say, oh, no. This is the best one because they really focus on the, you know, coachability and interpersonal interactions with the children. Right? But they could still be the same league, and maybe that league doesn’t have better coaching. It’s just that ChatGPT said that it did. I I copied and pasted your your research in there. Yeah. And you can find that on the SparkToro blog as well. Yeah. But just all that to say, like, people have very different intent. People describe things very differently. And in certain settings, they might get the same answer from chat GPT, but they’re still going to interpret that differently. I, I have some research around that coming up. Oh. Coming. Right? So we asked about how often the most mentioned brands are included in prompt responses. Like, hey. What how how much is their concentration around the sort of, like, top three? And so this this is showing in in these, you know, blue for Google AI, orange for Claude, green for ChatGPT, like that. Is there concentration around a few brands? The answer is kind of. Most of the time, some there is some concentration around those top brands. For example, Smart Sites was the the brand that appeared most frequently in the dataset. I think it was, like, eighty five times in ninety five Google AI responses for recommended digital marketing consultants in ecommerce. Quite impressive. Right? I that that’s a big field. I’m I’m surprised to see them so mentioned that they were the most mentioned of anyone, and they appear, you know, whatever, eight out of ten times. The least mentioned was the men’s fashion Instagram account, Adam Gallagher, who was who was the most popular account appear in there but appeared in less than half the responses. So depending on, you know, which types of questions you’re asking, the inconsistency can be extremely high, or or it can be much more consistent and contained around the top few. And so we we looked at that consistency rate and average rank, which you can if you wanna dig into the math, you can. We we did pair consistency calculations. Thanks to the folks at Kumshu who have a PhD math student who helped me put this together because I am I am a stats amateur. And and the only thing that was very important to me is I said, have to explain it to me in a way that I can explain it to other people. And so this is this is essentially the how how often are you likely to get two answers, you know, any any two pairs of brands in the answer and the and then the rank difference between those. It varies. Right? The answer is it varies. And you can see that it’s not consistent in the variation either. Like, if you don’t if you don’t have all the numbers for your specific space or the space you’re tracking for a client or or, you know, around a brand or a competitor, you will not know how consistent or inconsistent, how much concentration of the brands there are at the top or or not. Like, seeing this data by running a few, you know, chat GBT prompts or a few Google AI mode prompts will not get you there. You need to do serious at scale data. There is more Oh, yeah. Oh, sorry. We’re curious in the chat. Like, Robert and Twyla mentioned this. How like, why why cloud computing was always so highly correlated in that chart. Why is cloud computing highly correlated? This one. Yeah. Okay. Right. So that that so it’s not correlation. Pair consistency calculation. Pair consistency rate is the chance the odds that two that a pair of answers will appear again and again. And so as you might imagine, in cloud computing, if you ask the AI, hey. What are good cloud computing providers for startups? The Google Cloud and Amazon’s AWS are gonna get mentioned almost every time. So that is why you see that very, very high or relatively high pair consistency across all three models. Right? Cloud computing for ChatGPT, TrachVP for Cod. Like, they almost always gave those. Right? Does that does that make sense? Yes. Okay. I see. I see. It. Alright. Cool. The one of the important things that we wanted to point out here is that the consistency happens more across the sectors than across the tools. Right? So when you looked at cloud computing here, saw similarly consistent versus, you know, sci fi novels similarly inconsistent. You you can see that in the when we break it down by the prompt as well. Right? That essentially, like, these these bars tend to be similar when they’re grouped by there’s the cloud computing bars. Right? That really stands out. Or the nutrition accounts, that really stands out. Right? So that consistency happens more in the sector than it does in the in the model or tool used. And then you can see visibility of top and and second most likely, as we talked about previously, City of Hope Hospital, great example here of of a cancer care hospital that came up sixty nine out of seventy one answers. Like, it was almost always mentioned by ChatGPT, but it was the top answer in only a third of those. So which one matters more? Does that mean City of Hope is a better hospital for cancer care because it was mentioned so often or because it’s not the number one one? Look. I don’t have a perfect answer for you here, but what I’m going to say what I believe after doing this research is that ranking is a bunch of ********. You should throw out the idea of ranking. If if an AI puts a thing first or second or third or fifth or twelfth, you should consider them all the same. There’s there’s no not enough consistency for me to believe that AI tools, when they put a brand or recommendation or a location or product in those results that there is nothing meant by putting one first and one second and one third. Those are random. That’s that’s just what I’m saying. Okay. We did this for Claude and ChatGPT and and Google AI. Across all three of them, the average visibility, so this is like the percent of times that the top three mentioned brands came up, had quite a bit of variability. You can see the standard deviation’s high in all of these, like near twenty percent. But ChatGPT, it’s sixty four percent. Claude, seventy three. Google AI, sixty eight. So kinda similar. Like, kinda similar. They tend to have the same patterns around this. And that yeah. I think that’s that’s quite interesting too. You can get a sense that, like, AIs have a consistent amount of inconsistency. Alright. Visibility percentages across numerous prompts run dozens of times actually does seem to be a reasonable metric. I I was not convinced of this. Like, when we started this experiment, if you look at sort of my hypothesis for it, I thought that was not true. That is true. But rankings rankings, bunch of baloney. Right? The the order that things appear in just not meaningful, not useful. I am upset with the AI tools that try and, you know, make that a thing. I think that’s I think they’re misleading you and everyone else in the industry when they do that. So here’s an example. This is Gumshoe, the the the company that ran this with us. By the way, if you’re wondering why would an AI visibility tool provider work with you on this RAND, what if the results had been much, much worse for them? I also asked that question. I’m gonna tell you the answer. It’s because one of the cofounders, Patrick O’Donnell, who’s just a lovely guy, plays Dungeons and Dragons with me. And and so I basically managed to convince him. I’m like, oh, Patrick, come on. Let’s do this research together. He was like, well, okay. You know, I’ve been I’ve been feeding him great meals and, you know, keeping his characters alive for years, so I think I think he owes me. Turns out this research actually made me quite a bit more of a fan of gumshoe. I’m not gonna lie. Patrick knows this, but I was a little skeptical of him starting that company and I was a little skeptical of like this this whole process. But I I’m I’m converted. I think it is useful if you if you really care about your visibility in AI. I think it is useful to know the visibility percent. So this is essentially saying, if you were to ask ChatGPT is that what we’re looking at? No. I think this is Google AI. If you were to ask Google’s AI, what are easy to use mark audience research tools for solo marketers? And, you know, and then a bunch of prompts around that and you were to ask those hundreds of times, you would find that SparkToro is mentioned in sixteen approximately sixty nine percent of those responses. There’s no ranking here because ranking, we’ve already determined. And then you can see that that Audience I’m not totally sure if I’m pronouncing their name right, but Audience and SimilarWeb and BrandWatch and curated by Quantcast, which I’ve actually never used, and FullStory, those brands are also mentioned sometimes. And I appreciate that, like, they’re showing me, hey. Here’s who else gets mentioned along with this, and here’s how frequently. Tracking that over time might be useful for us. Right? It might be interesting to see, like, who else is popping on the radar. I’m always trying to include other audience research providers in my, you know, presentations because I hate being self promotional. So Yeah. And just to clear, like, in the case for easy to use audience research tools for solo marketers, I mean, BrandWatch may not want to show up in that. Right? Because they’re not really a tool for solo marketers. Yeah. So that’s also just to say, like, it’s not like we’re throwing shade at them. It’s just they probably don’t wanna be there. Right. Right. Yeah. I mean Yeah. Definitely, my my understanding of Quantcast’s pricing is that it would be extraordinarily unfriendly to a solo marketer. Yeah. My my understanding as well. Yeah. Alright. There’s one more problem. I know we I know we have last twenty five minutes here, but there’s one more problem, big problem, and it goes to, Amanda, what you were talking about when you, asked your mom friends about, you know, prompting. Do any two people ever use the same language in their AI prompts? Like, remember, for all this research, every single slide I’ve shown you, we asked people to ask the exact same question to chat GBT precisely framed. Right? Like, the the the phrasing was the same every time copied and pasted. And that’s the inconsistency. The level of inconsistency is astounding for exactly the same question. Like, if Google rankings were that inconsistent, you couldn’t track them. There’d be there’d be no point. Right? Do two people ask use the same language when they’re asked? So what we did is I asked several hundred people. I think I I wanna say, like, a hundred and fifty ish filled this out. Our panelists to provide the text that they would use for a simple prompt. Right? So imagine you’re seeking to find the best headphone options for a gift to a family member who blah blah blah blah. How would you prompt at the AI tool? And then you can see I I didn’t copy in the hundred and forty some odd, forty seven whatever responses, but I wanna buy a set of headphones. I am buying a gift for a frequent traveler. What are the best headphones for someone who travels a lot? People ask this question in a million different ways. There was there were no two that were similar. So what we did with these, because they’re they’re long strings of text. So what we did is we calculated a metric called semantic similarity. And semantic similarity, again, I was like, you need to explain it to me so I can explain it to other people. Semantic similarity is a numeric metric that tells you how similar the not the exact words, but the underlying meaning of those words is. Right? So for for example, if you said what’s a what’s a good example of this? If if you said shoes and footwear, semantic similarity one. Right? Like, that’s the same thing. They mean the same thing. It doesn’t matter if they were phrased, you know, differently. We’re gonna calculate. So so the number zero point zero five, that means semantic similarity. Both are food, but one’s tacos and one’s banana bread. K. Like, point one, both are dinner entrees. Both do chicken, but from totally different cuisines. Point one eight, both are chicken, lemon, garlic recipes. You would probably recognize that they’re actually similar. And if you’re if you’re over point five, it’s probably a recipe for the same dish with some tweaks in there. So how similar were the answers from these people? Hundred and forty four responses. There’s the semantic similarity graph. The average average pairwise similarities, we’re getting that that pairwise number, zero point zero eight zero nine. I’ll shorthand it for you. Even with the exact same goal, people ask AIs very, very different questions just as Amanda’s small scale research showed too. The similarity of survey takers’ prompts to one another is approximately the similarity of, like, kung pao chicken and peanut butter. I know Casey loves kung pao chicken. He I think he likes peanut butter too. Maybe maybe because they have this zero point zero eight one semantic pairwise similarity. That’s probably He’s in the he’s in the comments. He loves it for the stats. And and so, essentially, what we’re saying is these these prompts, these hundred and forty prompts, not not close. They are not close to each other. The the underlying intent is similar, right, as we know because we we ask them to all have the same intent. But the way they phrase it, completely wildly different. So Gumshoe took all hundred and forty two unique prompts through their system, and here’s what they found. And before this, I will just say that when I saw that data for the first time, I was like, I told you. AI tracking is ********. Like, I I was I was like, yes. You’re taking people’s money. Like, it’s a scam. Then then Gumshu was like, okay. Sure. We’ll run all hundred and forty two prompts. We ran them a bunch of different times. They ran them through the seven what they call the seven major AI model families and then, you know, did it nine hundred and ninety four times. You know what’s wild about this, Amanda? The visibility of the brands in this looks almost exactly like if you were to take one of those prompts, anyone, and run it a hundred times, which has me which which got me thinking, oh, you know what? This actually makes sense because what the AI can do quite well is take vastly semantically different, you know, words and prompts and interpret it, essentially, you know, collect that and interpret it as the underlying intent, and then give the responses to the underlying intent, which which made me realize that this the the process here, right, whether if you come up with synthetic prompts, right, from a from an AI or you use human prompts, if the intent is correct and you ask enough times, you can get visibility in a sector even despite the fact that the way people ask is wildly different and probably no two prompts are ever the same. So, again, I I had a hypothesis invalidated through real data and experience, which which is kinda cool. We did this again, by the way. We did this again because I was like, no. I don’t I don’t believe it. I wanna do it again. We did b to b. Again, the prompts were were wildly unique, and the visit visibility percent numbers are much lower no matter the model. But that’s because this one, right, imagine you’re trying to help a friend’s coffee shop find a good brand design agency. This, you know, the the set of options is I don’t know. There there must be tens of thousands just in the United States alone for brand design agency options versus headphones where there’s, you know, maybe twenty options. I don’t know. Twenty five that that ever came up. And you can see here, wow, the percent numbers are much, much lower. And this also matches how the AI responds to synthetic prompts around this, again, sort of confirming that the process here works out. Okay. Which which brands get mentioned with varying degrees of frequency appears to be stable across multiple runs. So if you run, you know, prompts, like like, time wise. Right? So we ask Gumshu like, hey. Let’s run it, you know, this time, this time, this time, this time. That that looks pretty similar. It doesn’t fluctuate day to day too much, too badly, hour to hour. That that consistency does say the same. And and so okay. We these are the this is my conclusions from this project. I wanna represent that this is relatively small scale. Like, we asked, you know, a few hundred people to participate. We ran a few hundred different times on dozens of prompts, not thousands or millions, but we know I think we we we can say with relative certainly, AIs rarely give the same list of brands or recommendations. They almost never give the list in the same order. You can think of these tools as probability engines, and you and you probably should. Worse still, users almost never craft similar prompts. If you’re counting on, like, your prompt that you came up with to be representative how real people really ask their AI tool things, you’re no. That’s not. Measuring your brand’s presence in AI answers with precision, like like, you know, how many times are we ranking number one, number two, number three, all that. That’s fully fully below me. But you can, with enough prompts, run enough times get this sort of dartboard like pattern, right, that we this kind of thing, and that is pretty reasonable. Pretty reasonable. I am not gonna spell baloney bologna, Marjorie. I’m not that’s rude to the rude to the city of Emilia Romana. Alright. The good news the good news here is if AI generated synthetic prompts and topics are accurate, then you know what? Like, if they if they really represent how user behavior works, what you need what you would essentially need to get a great metric is you need the search terms, right, this and the underlying intent that an audience uses. Like, you need to know a lot of information about an audience. So, you know, Amanda, I was like, hey, Casey. After doing this research, I think, could we at SparkToro tell people what AI prompt topics the audience is likely to run? And Casey was like, yeah. I can do that. In fact, here it is. You could see we launched it last night. What do knitters prompt AIs about? This stuff. And and if the you know, I think a reasonable question, because I have tons of skepticism around this too, a reasonable question is, how do you know? And the answer is, well, we know what websites they visit. We know their demographics, and we know their search keywords. Most importantly, keywords, like the things that they ask Google. And if you feed those into an AI API call and then say, give me the prompt topics, you you get good results. I don’t wanna say they’re fantastic. Maybe you might know your audience even better and know exactly what they’re, you know, curious about, and maybe you would say, oh, I you know what? Best yarns for sweaters should be higher. That’s actually something that more nears I’m not gonna I’m not gonna argue. You might know your audience even better, but this is a an excellent approximation. And we’ve run, you know, dozens and hundreds of tests around this. We can also do this, which is kinda cool. So if you if you click on the see examples, we will show you those examples of what those are. And you could take this and put it in your prompt tracking platform of choice or whatever you vibe coded to save some money. That’s great too. I am just gonna say this this looks this looks pretty good. I don’t know. Like, Marjorie says, you would not trust an AI for patterns. Do these look to you? Well, we’ll talk later, Marjorie. I don’t wanna interrupt a webinar. Ask what purpose. I I found these pretty compelling. I asked about a bunch of topics that I know more about than knitting hobbyists shopping for fiber arts. But okay. Next steps. I don’t wanna talk too much SparkToro stuff. Next next steps. AI tracking tool providers, in my opinion, need to pick up where we left off. We need larger sample sizes. We need people to do more research around this. And also marketers and execs, I would really urge you to stop throwing your money at AI tracking products unless they have great answers to these things and research back findings around this. Like, I I think it’s pretty embarrassing that we are the ones publishing this research. We don’t have a dog in this race. Right? Like, that’s doesn’t we’re we’re not an AI tracking product. We don’t care about this stuff. We I just worry about marketers getting fleeced, and I’m still worried about that. I’m still worried that a lot of these products are not doing this in a scientific way, that they’re not asking the prompts enough times, that they are not getting the topics from the right kinds of places with with real user data. Hopefully, you know, the search keywords, which is which is most indicative of people’s behavior and and interests and topics. And I I’m worried that they’re giving you rankings instead of visibility. So after we published this a few weeks ago, some folks did pick up the Slack, I wanna highlight them because I I love when people do this. So, Gumshu published a methodology piece, which any AI tool tracking tool could copy. Right? Like, here’s how many times you need to run a prompt or a set of prompts in a model in order to get visibility percentages that you can trust with, you know, a high, you know, plus or minus this this much visibility percent. So if you want, you know, plus minus one, you would need to run ten prompts, seven phrasing, seven models. That means a single pass generates four hundred and ninety kind of results. You would need to run that twenty times to get plus or minus one percentage point. Or you could only run it once and you’d have plus or minus five percent, but you should specify that. That’s very important. Mike Saunders from Contender did some excellent research. This is not his only research piece. He’s done some some other good work there. I I started subscribing to him. He looked at how citations correlate with brand mentions in b to b prompt responses. I found I find a lot of that citation research, like here’s how citations correlate to this. Really interesting. But I think it’s super important. Please treat it the same way you treated ranking factor correlations in SEO. Like, this thing correlates with ranking higher or lower, not this definitely influences that. Be really careful. I I with credit to Britney Mueller here, I I filmed this video about essentially the link between citations and responses. And there’s a there’s a lot of people who believe that if you take if you were let’s say you opened up a new cancer care hospital and you then got listed in Health US News and UCSF Health and US News Health that because those are the citations that the AI points out, you would automatically appear in these results on the left. Right? Like, you you would be mentioned here. These citations do not power these results very often, not always. Lily Ray made some good points about how sometimes they do. But most of the time, they are what’s called post hoc rationalization. So that means the AI generates the answer it’s gonna give, the list of brands. And then after generating that, it gives you it goes and finds a list of citations that support. That’s that’s post hoc rationalization. That is not these citations are why these things rank. That is or we’re gonna come up with the prompt with the answer, then we’re gonna find citations to closely match it. And you can you can see if you ask for AI recommendations, by the way my my last point here before we get to q and a, please, dear god, ask ten times before you trust the list. Right? Like, here’s my here’s me asking for the best high quality brands of whole cut leather men’s Oxfords. You can see wildly different results, by the way, that Berluti is an outstanding choice. But, my god, no. Please do not get Beckett Simon or the Tom Fords. I don’t even recommend the Santonis, honestly. Xenia’s hit or miss. Crockett and Jones. Great. Yeah. Fine. Okay. Thank you. I have three links for you to check out. Spark Toro, you can go test our new AI prompts, prompt topics that Casey built. We’d love your feedback on that. Gumshoe, obviously. And for those who don’t know, Amanda has a new website, ZeroClickMarketing dot co, where she has got her new podcast, which is launching tomorrow. No. Now. Now. Already launched. Amazing. Already live. Soon to be announced book, our our zero click marketing book that we’re writing together. And, and Amanda also is doing some consulting work. So, you know, lucky you if you kind.