Language-model groups overstate consensus when replaying human deliberation on a reasoning task

Source: arXiv Computers & Society (Academic)

The Challenge

Organizations increasingly contemplate substituting or augmenting human deliberative committees with multi-agent large language model (LLM) simulations to model collective decision-making and forecasting. However, a major socio-technical concern lies in the validity of simulated deliberation: do LLM agent ensembles genuinely reproduce human consensus formation, or do algorithmic artifacts artificially induce agreement? Current research lacks rigorous empirical benchmarks comparing identical human deliberation tasks against belief-seeded agent groups using standardized scoring frameworks. This creates a critical validity problem for leaders seeking to deploy agentic deliberation systems for organizational consensus forecasting, board-level decision modeling, and collaborative problem-solving.

Core Findings

The study evaluated deliberative fidelity by replaying 100 held-out human deliberation groups solving the Wason selection task with matched LLM agent groups. Researchers seeded each agent with the individual human participant's pre-discussion belief and evaluated both populations using identical scoring code. While human consensus estimates ranged from 24.0% to 57.0%—driven partly by disengagement, as roughly 20% of humans never posted—agents exhibited near-universal participation and inflated consensus rates. Sensitivity analyses controlling for submission and participation metrics demonstrated persistent artificial consensus gaps of 34.0 to 44.4 percentage points across chat and reasoning agent architectures. Crucially, when tasks were reparameterized to eliminate memorized answers, reasoning-mode agent groups reached near-unanimous agreement on incorrect solutions. Simulated consensus completely decoupled from collective accuracy, proving that belief-anchored LLM groups introduce systematic pro-consensus bias rather than modeling human cognitive dynamics.

Strategic Takeaway

For digital leaders and organizational sociologists, this research exposes a dangerous illusion of consensus within multi-agent LLM systems. Substituting synthetic deliberative agents for human stakeholder panels risks manufacturing algorithmic conformity and false certainty, often amplifying collective errors. Executive decision-makers cannot use LLM consensus as a proxy for either human social consensus or cognitive correctness. Leaders must recognize that conversational agents lack the social resistance, ambivalence, and non-participation behaviors intrinsic to human groups. Enterprise AI governance frameworks must mandate rigorous behavioral audits before utilizing agentic ensembles to simulate consumer focus groups, policy deliberations, or risk governance discussions.

Deep Dive Q&A

Why do LLM agent groups achieve higher consensus than human teams?

Unlike human groups where members frequently withhold participation, express persistent skepticism, or drop out, LLM agents consistently generate proactive dialogue and exhibit sycophantic convergence tendencies, resulting in artificially elevated agreement rates 34 to 44 percentage points higher than human baselines.

Does artificial agent consensus correlate with improved decision accuracy?

No. The empirical findings reveal that simulated consensus decoupled entirely from factual accuracy; in novel reasoning conditions, agent groups reached near-unanimous consensus on erroneous answers, illustrating severe algorithmic groupthink.

What are the leadership risks of relying on synthetic agent focus groups?

Leaders risk relying on biased estimators of human collective outcomes, mistaking conversational compliance for authentic social alignment, which can lead to misguided strategic bets, flawed product validations, and missed socio-technical friction points.

Related Research & Implementation Playbooks

Cross-disciplinary frameworks and practical blueprints connected to this insight.