XAI Evaluation Cards: A Practical Method for Designing Human-Centred XAI Evaluations

Source: arXiv Human-Computer Interaction (Academic)

The Challenge

Evaluating explainable AI (XAI) systems from a human-centred perspective remains fragmented, ad hoc, and inconsistently operationalized across technical and social science disciplines. While algorithmic benchmarks such as fidelity and robustness dominate computer science literature, practitioners struggle to evaluate subjective and interactional properties such as trust, cognitive load, and intelligibility. The lack of standardized terminology and collaborative evaluation frameworks creates friction within interdisciplinary teams. Consequently, organizations and research groups either bypass human evaluations entirely or rely on convenient, mismatched metrics that fail to reflect the socio-technical reality of end users.

Core Findings

Kacafírková et al. present the XAI Evaluation Cards, an evidence-informed method and repository designed to guide interdisciplinary teams through systematic evaluation planning. Synthesized from a review of 82 empirical XAI studies and a comparative analysis of seven prominent evaluation taxonomies, the authors consolidated 36 distinct evaluation properties clustered into five core categories: Objective System Aspects, User Experience, Subjective System Aspects, Explanation Aspects, and Interaction. The method operationalizes a three-step card-sorting protocol: identifying relevant properties, collaboratively prioritizing them according to project development stages, and mapping selected properties to an online repository of validated empirical measures. Tested across five projects with two distinct research groups (n = 13), the findings demonstrate that card sorting facilitates cross-disciplinary alignment, reconciles divergent metrics (such as distinguishing behavioral agreement from trust), and prevents premature technical lock-in.

Strategic Takeaway

For digital leaders and socio-technical researchers, the XAI Evaluation Cards provide a structured mechanism for governing AI accountability and transparency. Bridging the divide between machine learning engineers and domain stakeholders, the card-sorting framework transforms abstract ethical AI principles into actionable, prioritized evaluation metrics. Technology leaders can integrate this tool into agile design sprints to systematically measure user trust, cognitive load, and human-AI decision accuracy. This mitigates compliance risks, prevents organizational over-reliance on opaque models, and ensures AI systems deliver measurable, user-centered utility within enterprise workflows.

Deep Dive Q&A

What core problem do the XAI Evaluation Cards address in AI deployment?

They resolve the fragmentation and technical bias in explainable AI evaluations by providing a structured, card-sorting method that helps cross-functional teams select and prioritize human-centered metrics over purely algorithmic benchmarks.

What are the five conceptual categories included in the framework?

The 36 evaluation properties are grouped into Objective System Aspects, User Experience, Subjective System Aspects, Explanation Aspects, and Interaction.

How does the method support interdisciplinary alignment?

By using a tangible three-step card sorting process (identify, prioritize, measure) alongside an empirical measurement repository, it establishes shared terminology and consensus across computer scientists, designers, and domain experts.

Related Research & Implementation Playbooks

Cross-disciplinary frameworks and practical blueprints connected to this insight.