Anatomy of a ChatGPT local answer

Ask ChatGPT for the best HVAC company in Austin and you get a map with a few businesses on it, some paragraphs about them, and lately an ad on top. It arrives as one answer, but it's built in six steps.

At Complete SEO we've spent the last month taking that answer apart. We collected 44,988 answers through a scraper, then captured the raw payload a logged-in browser receives, because the two don't contain the same thing. Here's the answer with each part tagged by the step that built it.

Step 1

It rewrites your question

You type a question, and ChatGPT searches with a query it writes itself. The payload carries that query in a field called map_search_model_queries, and on September 23 we captured three of them next to the questions that produced them.

"Who is the best roofer in Phoenix, AZ?" went out as "best roofers Phoenix AZ reviews roofing contractors". "Who is the best HVAC company in Austin, TX?" became "best HVAC companies Austin TX reviews licensed AC repair". "Who is the best plumber in Raleigh, NC?" became "best plumbers Raleigh NC reviews licensed plumber".

The word "reviews" is in all three, and none of us typed it. "Licensed" shows up in two. ChatGPT decided that "best" meant reviewed and licensed, and searched on those terms. The query your business has to match is the one ChatGPT wrote, and you never see it.

The scraper misses this field. On the same prompts run the same day, a logged-in browser received the rewritten query on 95% of responses, and the DataForSEO scraper returned nothing on 95% of them.

Step 2

It looks up listings and searches the web

The card comes from a places lookup. ChatGPT searches an index of business listings and gets records back, the way a maps app does. That's a different job from reading web pages and summarizing them.

It sends that lookup in the same call as a regular web search. On every capture where we could see the call itself, 100 of 100 from September 8 and 10, ChatGPT sent one request carrying both. Here's one from September 10, exactly as it went out.

The lookup line carries one to four category phrases ChatGPT wrote itself, and some of them add words the question never used. "Top rated" appears in 12 of the 100 lookups, on questions that never said it. The word "review" appears in 70 of the 111 web queries, and none of the 100 questions used it.

The card shows up on 95.4% of the 27,405 local answers in our census. Of 60 prompts we ran through a real browser the same day, 57 had one.

We can't tell you who supplies that index. The business details on those rows track Google's listing data closely, but the provider isn't visible in anything we can capture.

Step 3

Each business comes back as 57 fields

Each business on that card comes back as a record with 57 fields. We checked 173 business records across 16 captures on September 23, and every one had the same 57.

Eight of them are drawn on the card. The same business, pulled through the DataForSEO scraper that much of the industry measures with, has 9.

The other 49 include whether the listing is claimed, its hours, its distance, review highlights, a quote-request link and a ranking score. One of them, provider_logo_url_short, is a small image URL that records where the star rating came from. Of the card rows carrying a Yelp logo, 15 of 16 matched Yelp's own rating and review count exactly. Of the rows without it, none did.

So you can tell where each business's rating came from. Yelp supplies it for more card businesses than any other source, but never a majority, and the exact share depends on which list you count.

Step 4

The records split into three lists

What looks like one card is three separate lists of businesses, and they differ enough that pooling them gives a number that describes none of them.

The scraper returns the map card and the pool behind Expand as one set. So every figure our census published about "the card", and as far as we can tell every figure anyone else has published from the same source, describes the pool behind Expand and not the businesses on screen. On the one prompt where we counted the rendered page by hand, five businesses were visible before expanding, and sorting that capture's payload put five in the map list. That's one prompt, and it's the only direct check we have.

Step 5

It writes the answer from other sources too

Then ChatGPT writes paragraphs about the businesses, and they often name businesses the card never held. On 70.4% of answers with a card, at least one business named in the text isn't on that answer's card.

Directories are a big part of those sources. Across our census, the most common domains behind local answers are angi.com, expertise.com, bbb.org and reddit.com. On answers with a card, directories make up 29.9% of cited sources and business websites 56.5%. Where we could see both what was retrieved and what was cited, directories were 39.8% of the URLs retrieved and 29.9% of those cited, so ChatGPT pulls more directory pages than it uses.

The logged-in payload lets us check that from the other side, because it lists what the search returned separately from what the answer cited. Across 116 captures, about one page in five that ChatGPT retrieved ended up cited. The median answer retrieved 12 distinct pages we can see and cited 2.

Directories vary a lot in how often they get cited. Thumbtack and Expertise.com were each retrieved in 45 answers. Expertise was cited in 24 of them and Thumbtack in 2.

Step 6

An ad can sit on top

On September 8 we captured 63 of these payloads and found no ads in any of them. On September 23, with the same account, the same free tier and the same method, 10 of 16 local answers carried an ad.

The payload records it as one field on your own message, a flag reading ad_delivered. The rest of that message's metadata is identical on both dates, so the September 8 result isn't a gap in what we recorded. The ad itself is never in the payload. Only the flag is.

A few hours later we ran the same 16 questions through the DataForSEO scraper. It reported zero ads on all 16.

If you track AI visibility through the scraper, ChatGPT ads on local answers look like they disappeared in late August and stayed near zero. In a logged-in browser on the same questions, they're on more than half.

Of the 10 ads, six came from lead-generation marketplaces, Thumbtack three times and Angi three times. The other four were businesses and brands, including a Raleigh plumber and an Austin HVAC company. Only one of the 10 advertisers also appeared in the organic card under its own ad.

So the directories that already make up 29.9% of the cited sources on these answers are now also buying the ad slot above them.

Thumbtack bought 3 of the 10 ads. Across the 116 logged-in captures in our three samples, it was retrieved in 45 answers and cited in 2.

What we still can't tell you

Every row on that card carries a ranking_score, and the card's display order doesn't follow it. A popularity_score field sits beside it, empty on every row we've captured. Neither field is documented anywhere.

We can tell you that the card exists, how often it appears, what arrives for each business, which of the three lists a given number describes, and where a star rating came from. We can't tell you what gets a business onto that card, or higher on it. The fields that might answer that are in the payload, and we haven't modeled them yet. That's what we're working on next.