Sophistication in GenAI Use: Field Evidence from a Large Firm

Source: arxiv.org

The Challenge

While enterprise adoption and nominal access to generative AI have expanded rapidly, nominal deployment does not guarantee productivity returns or effective integration. Extant socio-technical and technology adoption literature primarily quantifies binary adoption or access rates, offering little empirical clarity on the quality, skill, and behavioral sophistication of employee interactions. Furthermore, organizations lack reliable frameworks to measure whether workforce prompt behaviors improve naturally through organic exposure or whether formalized enterprise training initiatives generate enduring behavioral change across differing professional hierarchies and functional divisions.

Core Findings

Analyzing proprietary field data from KPMG LLP, the authors examine 713,564 prompts and large language model responses generated by nearly 4,000 back-office employees across 15 functional domains over eight months. Sophistication was operationalized through three composite dimensions: use-case diversity, deliberate prompting strategy, and prompt clarity. The empirical results reveal three primary findings: First, interaction sophistication scales positively with employee seniority, indicating that established domain expertise complements genAI capabilities. Second, sophistication exhibits marked variance across organizational functions, peaking in Strategy, Digital Innovation, and Project Management while lagging in transactional units like Accounting and Finance. Third, longitudinal analysis and employee fixed-effects models show no spontaneous maturation in sophistication over time, and formal AI training produces only a temporary uptick during the training month without sustained longitudinal improvements.

Strategic Takeaway

This study shifts digital leadership discourse from adoption volume to interaction sophistication. For organizational leaders, the findings demonstrate that providing enterprise AI access and standardized, one-off training modules is insufficient to develop lasting employee capabilities. Instead, leaders must recognize domain knowledge as a critical enabler of AI efficacy, empowering senior subject-matter experts to model sophisticated prompting behaviors. Furthermore, managers can utilize accessible interaction metadata—such as prompt ambition (length) and persistence (iterative conversational depth)—as cost-effective proxy metrics to evaluate and cultivate workforce AI fluency without requiring intrusive prompt transcript logging.

Deep Dive Q&A

How is sophistication in generative AI use operationalized in this study?

The authors define sophistication using three empirical dimensions: use-case diversity (applying LLMs across diverse functional tasks), deliberate strategy use (applying structured prompting techniques like chain-of-thought or role-assignment), and prompt clarity (providing precise, unambiguous constraints and instructions).

Why do senior personnel demonstrate higher GenAI sophistication than entry-level staff?

Senior professionals possess deeper contextual domain knowledge and strategic problem-structuring abilities, which directly complement generative AI capabilities and enable them to formulate more precise constraints, complex instructions, and iterative refinements.

Does standard corporate GenAI training produce lasting behavioral change?

No. The empirical data indicates that employees exhibit heightened sophistication only during the calendar month in which formal training occurs, reverting to baseline interaction patterns in subsequent months.

Related Research & Implementation Playbooks

Cross-disciplinary frameworks and practical blueprints connected to this insight.