Why two people get different answers from the same model

Two apparently identical requests can differ in model version, instructions, memory, history, retrieved evidence or generation behavior. There is no universal ranking of which cause matters most.

One: check the model

Check the model and product first, then inspect the other possible differences.

Record the model or version shown and the product, plan and settings. Different surfaces may use different configurations, and a product label may not expose every implementation detail.

A changed model or configuration is one possible cause. Hold it fixed where the product permits and inspect the other variables too.

Two: hidden context differs per person

Applications may add system instructions, project context, account preferences or memory. API callers may control some of these directly; do not assume every surface adds the same hidden context.

Memory, project context and custom instructions may differ across accounts or conversations. Review the settings and context the product exposes rather than assuming every answer includes the same saved material.

So two people asking “the same question” may be sending materially different requests. The visible text is identical; the actual payload is not. This is worth knowing because it is invisible from inside — neither person can see what the other’s account is adding.

Three: history steers everything after it

Earlier turns can affect the request, but may also be omitted, retrieved or summarized by the application. Compare the actual available history when possible.

A model that has already committed to a framing tends to elaborate on it rather than reconsider. Ask the same question at turn one and turn fifteen and you may get genuinely different answers, both consistent with their context.

Fresh conversations are useful for a single-turn comparison. For an ongoing-workflow test, keep the same history and state instead; the right control depends on the task and the context actually available.

Four: retrieval returned different things

If the tool searches or uses attached files, check whether both runs actually received the same material. Retrieval can differ even when the visible question does not.

Search and connected tools may return different evidence across accounts or times. Preserve the retrieved material when you can, and check whether the answer is supported by it.

If evidence differs, test both retrieval and the model’s use of it. A better retrieval configuration or model can help, but neither removes the need to verify claims.

Five: genuine sampling variation

Generation can vary even with apparently identical inputs. The size and consequences of that variation depend on the model and task, so use repeated runs rather than assuming it affects only phrasing.

Variation is not a calibrated confidence score. Agreement can be consistently wrong, and disagreement can contain a correct answer; compare against a defensible reference.

Repeated runs can expose variation worth investigating. Two responses are a spot check, not a sufficient reliability estimate.

How to get consistency when you need it

For a comparison, control what you can: model version, instructions, history, tool results and settings. Record what the product does not expose.

Use a pinned model version where one is offered, and record it with the results. Version control reduces one source of change but does not promise identical outputs.

Control the context you can. Version the instructions and record inputs and tool results. An API can expose more control than a chat interface, but provider-side behavior may still be outside your visibility.

Choose fresh sessions for a single-turn task or identical saved history for a multi-turn task. Keep memory and project context consistent, and record unavoidable differences.

Fix the retrieval. If sources vary, log which ones were used. You cannot diagnose an answer without knowing what it was based on.

The wording effect, which is larger than people expect

Worth separating out, because it is usually mistaken for randomness.

Two people rarely ask the same question. They ask questions they believe are equivalent, and the differences matter more than they expect. “Is X safe?” and “What are the risks of X?” reliably produce different-feeling answers — the first invites a verdict, the second invites a list. Neither is wrong; they were different requests.

The same applies to what gets omitted. One person mentions the constraint, the other assumes it is obvious. One pastes the document, the other describes it. By the time the answers are compared, nobody has the original text to hand and the difference is attributed to the model.

Compare the exact prompts before attributing the difference to the model. Then check the other variables; wording is one possible explanation, not a demonstrated dominant cause.

When variation is a feature

One honest counterpoint. For anything generative — drafting, naming, brainstorming, finding another angle — variation is the point, and suppressing it makes the tool less useful.

Asking the same question three times and taking the best answer is a legitimate technique. So is asking two different models and comparing. The problem is not variation itself; it is variation in places where somebody assumed reproducibility and nobody checked. Know which mode you are in, and the same behavior goes from a defect to a feature.

What this means organizationally

The practical consequence is that “the AI said” is not a citable source, and it is worth saying that out loud once in a team.

Logs, saved sources and independently checked answers can help adjudicate a disagreement afterward. If the relevant state was not recorded, state that limitation instead of guessing what caused it.

Which brings it back to the same discipline that governs everything else here: if it is going to inform a decision, it needs a source somebody can open. The model can help you find, read and reason about that source. It should not be the source, and the inconsistency is the clearest everyday demonstration of why.

Technical review, October 10, 2026. References: OpenAI: evaluation best practices; OpenAI: memory in ChatGPT; OpenAI: conversation state.

Try this

Have two people record the exact prompt, model, settings and available context for the same task. Compare several runs against a checked reference. Change one variable at a time and keep unknown product settings explicit.