Models
The nine surfaces Recomma asks, and why how it asks them matters.
Recomma samples nine models:
| Model | |
|---|---|
| ChatGPT | Claude |
| Gemini | Perplexity |
| Copilot | Grok |
| DeepSeek | Google AI Mode |
| AI Overviews |
Which are enabled is a per-project choice. Every enabled model is asked every tracked prompt on every cycle, so the model list is one of the two multipliers on what a cycle costs — see sampling.
Model means the surface, not the build
ChatGPT is a model in this sense; gpt-4o-2024-08-06 is the build that
answered on a given day. The first is what you track and filter by, because
it is what your buyer opens. The second changes under it without warning and
is recorded on each chat so a shift in the numbers can be traced to one.
Where both appear — the archive, the CSV export — the surface is model and
the build is model_version.
Models disagree, a lot
The single most common mistake in reading these numbers is treating the all-models figure as the figure. It is an average over models that frequently disagree: different retrieval, different training, different willingness to name brands at all.
A brand can be named in most ChatGPT chats and almost no Gemini ones. The average is real, but it describes no model anyone actually uses.
Check a metric per model before acting on it
The model filter on the Overview exists for this. If a figure looks strange, split it by model first — the answer is usually that one model is behaving differently, not that the brand’s position changed.
Fidelity: interface or API
How a chat was obtained is recorded on it, and there are two ways:
- Interface — the model’s own product, the way a person sees it. Web search runs, the product’s own ranking applies, sources are attached as the product attaches them.
- API — the build behind the product, called directly. A fallback, used when the interface cannot be reached.
These are not the same measurement. ChatGPT the product and the build behind it answer differently, because most of what makes the product’s reply is the retrieval and formatting wrapped around it.
Interface chats are the truer measurement. API chats are marked as such rather than quietly mixed in, so a figure never rests on an unstated change in what was being asked.
Failed chats
A model that does not reply in time produces a failed chat. These are:
- counted and shown on the Overview, so you can see how many of the chats asked for actually came back;
- excluded from the metrics, because there is no text to count;
- retried on the next pass.
A small number is normal. A persistent one against a single model usually means that model is rate-limiting, and the figures for it will be measured over fewer chats — check the sample size when filtered to it.
The count and the banner are different facts
The Overview reports every failure in the window beside the chat count, and explains only the ones still failing. A model that timed out on Monday and has answered since drops off the explanation without changing the count.