The Challenge
Integrating generative artificial intelligence into legacy software systems is widely assumed to dramatically elevate user satisfaction and usability. However, empirical assessments comparing pre- and post-AI user experience remain scarce. Designers and software leaders frequently overlook whether transformative algorithmic capabilities truly translate into proportional gains in perceived system usability. Standard evaluation metrics may fail to capture how deeply altered interaction workflows influence end-user perceptions, creating an empirical blind spot regarding whether heavy technical investments in generative AI genuinely enhance system usability or merely alter the operational cognitive load without significantly shifting standardized usability scores.
Core Findings
The authors conducted an empirical comparative evaluation of FGPE AuthorKit, an educational platform within the Framework for Gamified Programming Education designed for authoring programming exercises. The tool was augmented with GAMAI, an extension utilizing Large Language Models to automatically synthesize gamified exercises from raw narrative inputs. Using the Usability Metric for User Experience (UMUX) framework, the study compared perceived usability scores between the baseline non-AI tool and the AI-augmented edition. Surprisingly, despite fundamentally altering the interaction paradigm from manual specification to automated prompt-based generation, the UMUX score rose only marginally from 73.6 to 79.5 (a gain of under six points). These findings challenge the assumption that profound AI-driven workflow transformations automatically yield dramatic usability improvements, highlighting potential measurement limitations in survey-based UX evaluations applied to AI-enriched software systems.
Strategic Takeaway
For digital leaders and software product strategists, this study serves as an essential reality check against assumptions of guaranteed usability gains from generative AI integrations. Adding automated LLM features significantly alters socio-technical task dynamics but does not automatically ensure high subjective usability satisfaction. Organizations must avoid relying exclusively on standard legacy UX surveys like UMUX when measuring generative system adoption. Leadership must assess qualitative factors such as cognitive trust, prompt fatigue, and task autonomy, ensuring that AI-driven transformation aligns with authentic user needs rather than resting on technological novelty alone.