LLM results change because answer engines are not deterministic. When you ask the same question two times, you get different wording. Sometimes you also get different brands. This is a property of the medium. It is not a fault in the measurement.

​
What causes the changes

Sampling. Models make text by probability. Two runs of the same prompt can follow different paths. Retrieval. Some answer engines search before they answer. They can find different sources on different days, because the web changed or the retriever ranked the results in a different order. Changes to the answer engines. The providers update models, tune prompts, and rebuild retrieval. They do not announce these changes, and the changes can move a full category at the same time.

​
How to measure in a medium that changes

Read trends, not single readings. A single run is an estimate. A direction over several runs is a finding. For more information, see Trend tracking. Do not change the questions. If you change scenarios between runs, you measure two different things. Keep a stable set, and change it only when you decide to. Find out if your competitors also changed. If all brands changed at the same time, the answer engine changed, not your brand. This one check prevents most unnecessary investigations. Run sufficient conversations. More conversations for each topic give less noise in each reading.

​
When a change is real

A change is real when these three conditions are true:
  • The change continues over several runs.
  • The change applies to specific topics, not to all topics.
  • The citations explain the change. For example, a source appeared or a source disappeared.