The Effect of Enhancing Software with AI on Its Perceived Usability: The Case of FGPE AuthorKit

Source: AIS eLibrary

The Challenge

Integrating generative artificial intelligence into legacy software systems is widely assumed to dramatically elevate user satisfaction and usability. However, empirical assessments comparing pre- and post-AI user experience remain scarce. Designers and software leaders frequently overlook whether transformative algorithmic capabilities truly translate into proportional gains in perceived system usability. Standard evaluation metrics may fail to capture how deeply altered interaction workflows influence end-user perceptions, creating an empirical blind spot regarding whether heavy technical investments in generative AI genuinely enhance system usability or merely alter the operational cognitive load without significantly shifting standardized usability scores.

Core Findings

The authors conducted an empirical comparative evaluation of FGPE AuthorKit, an educational platform within the Framework for Gamified Programming Education designed for authoring programming exercises. The tool was augmented with GAMAI, an extension utilizing Large Language Models to automatically synthesize gamified exercises from raw narrative inputs. Using the Usability Metric for User Experience (UMUX) framework, the study compared perceived usability scores between the baseline non-AI tool and the AI-augmented edition. Surprisingly, despite fundamentally altering the interaction paradigm from manual specification to automated prompt-based generation, the UMUX score rose only marginally from 73.6 to 79.5 (a gain of under six points). These findings challenge the assumption that profound AI-driven workflow transformations automatically yield dramatic usability improvements, highlighting potential measurement limitations in survey-based UX evaluations applied to AI-enriched software systems.

Strategic Takeaway

For digital leaders and software product strategists, this study serves as an essential reality check against assumptions of guaranteed usability gains from generative AI integrations. Adding automated LLM features significantly alters socio-technical task dynamics but does not automatically ensure high subjective usability satisfaction. Organizations must avoid relying exclusively on standard legacy UX surveys like UMUX when measuring generative system adoption. Leadership must assess qualitative factors such as cognitive trust, prompt fatigue, and task autonomy, ensuring that AI-driven transformation aligns with authentic user needs rather than resting on technological novelty alone.

Deep Dive Q&A

What is FGPE AuthorKit and how was it enriched with AI?

FGPE AuthorKit is an educational authoring tool for building gamified programming exercises, which was extended with GAMAI, an LLM-driven module allowing educators to generate exercises automatically from textual story inputs.

How did the UMUX usability scores change after AI integration?

The UMUX score increased modestly from 73.6 in the original manual version to 79.5 in the AI-enhanced version, representing an unexpectedly small gain of less than six points despite radical workflow shifts.

What are the broader methodological implications for UX evaluation?

The modest increase suggests standard survey instruments like UMUX may not fully reflect the nuanced shifts in user cognitive burden, agency, and interaction models introduced by generative AI enhancements.

Related Research & Implementation Playbooks

Cross-disciplinary frameworks and practical blueprints connected to this insight.