Why you shouldn't take LLM judgments about preferences at face value
Modern language models can reason convincingly about all sorts of things, which is why they are increasingly being brought in as "experts" to assess consumer preferences. For example, people ask ChatGPT how much someone would be willing to pay for a flight with baggage, or how important proximity to the subway is to them. The idea looks tempting: if a model knows human tastes well enough, then based on its answers you could automatically match products, prices, and services. But how do you verify that such assessments actually reflect stable preferences rather than random words?
A group of researchers decided to test this from a scientific standpoint. Instead of simply checking whether the answers look logical at first glance, they asked a more rigorous question: can the model's answers be described at all by a single utility function? A utility function is a mathematical way to order preferences: if it exists, then all answers must be consistent with one another. For example, if you say you are willing to pay an extra 500 rubles for one hour less of travel time, and then choose an option that is both longer and more expensive, that is already a contradiction.
Methodology: how to measure inconsistency
The authors developed statistical tests that make it possible to quantify how far the model's answers deviate from the best-fitting utility function. If the deviations are small, the model is almost consistent in its preferences. If they are large, the judgments are random or situational in nature, and they cannot be reduced to a single scale.
The experiments were run on three types of scenarios: choosing a flight, renting an apartment, and booking a hotel. These are practical cases where estimating willingness to pay matters especially for business. Six different large language models took part in the testing, which makes the sample even more telling.
Results: contradictions everywhere
It turned out that all six models give consistently contradictory answers. That is, if you ask them similar questions in different phrasings, they are highly likely to choose options that do not fit together. For example, a model might indicate that parking availability is extremely important to it, but when comparing two specific apartments, it might refuse to pay extra for parking, even though it had earlier stated its willingness to do so.
Such discrepancies are hard to explain as mere error. The researchers emphasize that this is more of a systemic property: large language models do not have fixed "tastes"; instead, they generate each answer anew, guided by context and probabilistic patterns. So they can be used for rough ideas, but not for precise modeling of individual preferences.
Why this matters in practice
The results strike at several popular ideas at once. In behavioral economics and marketing, there is growing talk of using LLMs for automatic personalization: suppose a model has learned a user's preferences from their queries and can now pick optimal offers. But if the model's own judgments are internally contradictory, then any optimization algorithms built on them will fail.
In addition, this problem also affects research methodologies themselves. If a researcher uses an LLM to assess utility in surveys, they will first have to make sure the model is not giving random answers. Otherwise, the results of such research will be unreliable.
The good news is that the authors did not just point out the problem but also proposed tools for diagnosing it — tests and deviation metrics. These can already be used to check new models or to refine existing ones. Perhaps over time, special training methods will emerge that improve the self-consistency of judgments. But for now, it is too early to rely on LLMs as a trustworthy source of information about what people actually want.
