How close the answers are
Accuracy is measured on held-out questions: questions real people have answered that the model never saw in training. Results are reported at the individual level or the audience level, and tools built on prompting rarely say how they measured. This chapter goes through each level, the published critiques, and what a percentage can mean.
Individual level
Each simulated person’s answer is compared with the real person’s own answer. Two of the six providers report this, as do the research papers on twins and trained models since 2024.
- Ordinal alignment
- Questions with ordered answers, such as scales
- 0.72 to 0.78
- Out of 1, in research papers. Random answers score about 0.6, and the same people asked again score 0.82.
- Toubia et al., 2025; Peng et al., 2025; Dahiya (Semilattice), 2026
- Exact match
- Any multiple-choice question
- 66% to 69%
- Share of answers exactly right. On the General Social Survey chance is 27%, and the same people asked again matched 81%.
- Park et al. (Simile), 2024 (v1); Park et al. (Simile), 2024; Dahiya (Semilattice), 2026
Audience level
The share of simulated people choosing each answer is compared with the real share. Three of the six providers report this.
- Mean absolute error
- Answer shares
- 3.5 to 4 points
- The average gap between simulated and real shares, from one provider’s average and another’s best case. A model can match shares while giving most individuals the wrong answer.
- Aaru, 2026; Electric Twin
Tools built on prompting
Products that write a persona into the prompt of a general-purpose model.
- Method usually not published
- “88% similarity”
- Figures like this are published without the formula behind them, so they cannot be checked or compared.
- Evidenza
Individual level: ordinal alignment in research
Most research papers that score each simulated person use ordinal alignment. Each answer is scored by how close it is to the real one on the scale, and 1 means every answer matched.
The same people, asked twiceOn one panel, real people answering the same questions at least two weeks apart matched their own answers this closely. Scores in published research run from 0.717 to 0.775; papers report a best model, an average or a median, as shown under each. Toubia et al., 2025
Which critiques apply to which models
There are three ways to build a simulated person, and each widely cited critique tested one of them. A product can combine them: Simile grounds a fine-tuned model in each person’s interview. A critique of prompted personas does not show whether grounded or trained models have the same problem.
Prompted
A general-purpose model is told who to be, such as “You are a 34-year-old nurse in Leeds.” AI personas and silicon samples are built this way.
Models flatten identity groups into stereotypes
Four language models were told which of 16 identities to answer as, and their answers were compared with those of 3,200 people who hold those identities.
Other workWhen each simulated person was given the real person’s interview instead of their demographics, accuracy differed less between racial and ideological groups. Park et al. (Simile), 2024
Synthetic answers are not reliable for statistical inference
ChatGPT was given each respondent’s demographics and politics from a US election survey. Its average answers matched the survey, but they varied less, 48% of regression coefficients differed significantly from the survey’s, and the same prompts gave different answers after a model update.
Other workA model fine-tuned on answers from 210 experiments matched the distribution of people’s answers more closely on experiments it had not seen. Kolluri et al. (Simile), 2025
Models’ opinions match some groups and not others
Models’ answers to US opinion polls were compared with those of 60 demographic groups. Telling a model which group to answer as did not close the gaps.
Other workReweighting personas to a human reference is being tested. Hu et al., 2025
Grounded
A general-purpose model reads one real person’s interview or survey answers before it answers as them. The digital twins in Toubia et al. and Peng et al. are built this way.
None of the six providers grounds a general-purpose model on its own. Artificial Societies and Simile ground trained models. Electric Twin constrains general-purpose models with survey data and does not say whether that data is per person.
Twins are only slightly better than a model with no persona
Twins built from each person’s answers to over 500 questions were tested in 19 studies. They scored 0.748, against 0.734 for the same model given an empty persona, and their answers correlated with each person’s at r ≈ 0.20.
Trained
A model is trained on what real people said and did, usually by fine-tuning a language model. Aaru trains a population model that is not a language model.
Not tested by these critiques
None of the critiques above tested a trained model. The published results for trained models come from the teams that built them. Kolluri et al. (Simile), 2025; Dahiya (Semilattice), 2026
LimitTraining needs large sets of real responses, and models trained per audience need a set for each audience. Kolluri et al. (Simile), 2025; Dahiya (Semilattice), 2026
Two levels of accuracy
A score can compare the share of people choosing each answer, or compare each simulated person with the real one. The same simulated answers can be perfect at the first and poor at the second.
Shares match on all five answers
3 of 10 people get their own answer
- Person 1 1 1
- Person 2 2 2
- Person 3 2 2
- Person 4 3 2
- Person 5 2 4
- Person 6 4 3
- Person 7 5 3
- Person 8 3 5
- Person 9 1 2
- Person 10 2 1
What a percentage can mean
The same system can be described as 68.85% or 85% accurate depending on the metric, and an audience-level figure can be high while individuals are wrong.
| Metric | What it measures | Level | Example |
|---|---|---|---|
| Exact match | Share of questions where the simulated person picks the real person’s answer | Individual | 68.85% on the General Social Survey Park et al. (Simile), 2024 (v1) |
| Ratio to human retest | Exact match divided by how often people repeat their own answers when asked again | Individual | 85%, from the same 68.85% Park et al. (Simile), 2024 (v1) |
| Ordinal alignment | 1 minus the average distance between the simulated and real answer on a scale, as a share of the scale | Individual | 0.717 to 0.775 Toubia et al., 2025; Kinzinger and Hartmann, 2026; Dahiya (Semilattice), 2026 |
| Correlation across questions | Whether the model ranks questions or effects the way people do | Audience | r = 0.98 for experiment effect sizes Park et al. (Simile), 2024 (v1) |
| Correlation across people | Whether the model tells people apart on one question | Individual | r ≈ 0.20 on new stimuli Peng et al., 2025 |
| Distribution overlap | How much the simulated and real answer shares overlap, question by question | Audience | 86% Artificial Societies |
| 1 − mean absolute error | 100 minus the average gap, in points, between simulated and real answer shares | Audience | 96%, a best case Electric Twin |
| Mean absolute error | The average gap, in points, between simulated and real answer shares | Audience | 3.53 points Aaru, 2026 |
| Undefined percentage | A single figure published without the formula behind it | Unknown | 88% similarity Evidenza |
Why ordinal alignment
It gives partial credit on scales: answering 6 when the person said 5 scores better than answering 1. It also puts scales of different lengths on one range from 0 to 1, so papers can compare results.
Used in 5 papersToubia et al., 2025; Kolluri et al. (Simile), 2025; Peng et al., 2025; Kinzinger and Hartmann, 2026; Dahiya (Semilattice), 2026
Most of these papers share authors or adopt the score from Twin-2K-500, so the agreement is one research line rather than an independent standard.
- A high floor. Random answers score 0.59 to 0.63, so the useful range is roughly 0.6 to 0.82. Toubia et al., 2025; Peng et al., 2025
- It rewards the middle. A model that answers near the middle of every scale scores well. In one study GPT-4o clustered around mid-scale and outscored the model that best matched the real spread of answers. Kolluri et al. (Simile), 2025
- Personas add little. Giving every twin the same empty persona scored 0.734, against 0.748 with each person’s full profile. Peng et al., 2025
- Only scales. It does not apply to unordered choices, multi-select or open-ended answers, and on two-option questions it is the same as exact match. Peng et al., 2025
An alternative: macro-F1
For each answer option, F1 combines how often the model is right when it picks it (precision) with how often it picks it for the people who chose it (recall). Macro-F1 averages F1 over options, so every option counts equally, however many people chose it.
A model that always predicts the most common answer gets a high accuracy on skewed questions and a low macro-F1. That makes it a direct test of whether the model tells people apart, the failure the ordinal score misses. It is also the standard measure for classification across machine learning, so results compare with the rest of the field.
Used inHoltdirk et al., 2025
Standard referencesSokolova and Lapalme, 2009; Opitz and Burst, 2019
Limits. It treats a near miss on a scale as fully wrong, and it is unstable for options few people choose. None of the survey-simulation benchmarks reviewed here uses it as its headline measure.
Still unsolved
- Variance. Simulated people’s answers vary less than the answers of the real people they model. Being tried: fine-tuning on human responses (Kolluri et al. (Simile), 2025); reporting the spread of answers and correlation across people alongside accuracy (Peng et al., 2025; Kinzinger and Hartmann, 2026).
- Bias. Models lean towards agreeing, and describe groups the way outsiders see them. Fanous et al., 2025; Wang et al., 2025 Being tried: reweighting personas to a human reference (Hu et al., 2025); grounding each agent in the person’s own interview instead of their demographics (Park et al. (Simile), 2024).
- Prompt brittleness. Answers shift with the wording of a question and the order of its options. Being tried: testing whether simulated answers shift with rewording by as much as people’s do (Artificial Societies, 2026); shuffling options and asking each question both ways in training (Dahiya (Semilattice), 2026).
- Reach. Accuracy falls on new stimuli and on questions far from the data a model was built from. Being tried: measuring twins on new stimuli (Peng et al., 2025); predicting each result’s error, or declining to answer (Simile; Listen Labs).