Krolik Marketing

Do AI answers change every time someone asks?

Yes. In tests by tool companies, only about half the businesses ChatGPT named for a local question were named again a week later. One answer tells you little: use the test below and count how often you appear.

One AI assistant gives three different lists of shops; your shop appears in two of the three answers.

What you can do today

  1. Do not rewrite your website because one answer left you out. A wrong fact, such as old hours or a wrong phone number, is different: fix it at its source. Cheers, a tool company, says "A verified wrong fact can be corrected without waiting for repeated answers."
  2. Write up to three questions a customer would ask, with your town in them. Ask each one in a new chat, three times. Count how many of the nine answers name you: "named in X of 9." More asks give a steadier count.
  3. Don't read too much into a single count. If you were named 3 times in 9, your real rate could be anywhere from about 1 in 8 to 2 in 3.
  4. Be careful with tools that sell an "AI ranking position." SparkToro, a tool company, writes that "any tool that gives a 'ranking position in AI' is full of baloney."
For each count of 0 to 9 named in 9 asks, the range the true rate could fall in; 3 of 9 is 12% to 65%.

What the studies found

Everything here that isn't from OpenAI comes from companies that sell tracking tools. It shows patterns, not causes. None of it covers auto repair, body shops or churches.

Same question, same day: the lists differ. A few days later, about half the businesses named came back. A week later, a bit over half. A second spot in the same town gives different results again.

A famous "less than 1 in 100" figure is about product recommendations, not local businesses.

Why your customer's answer won't match yours

OpenAI says ChatGPT uses an approximate location from your IP address and, "If memory is enabled, ChatGPT may use relevant saved memories when rewriting a search query." Your customer's answer won't match yours.

For why a competitor may be named instead of you, see this page. To check whether any of it sent customers, see how to know. What OpenAI says about ranking is here.

Based on public documentation: Spearleaf; Cheers; BrightLocal; SparkToro; OpenAI help.

All posts