A User-Centric Domain-Adaptive Quality Model for Benchmarking Generative AI Systems

Abstract

Generative AI (GenAI) systems operate across diverse application domains where quality priorities shift dynamically in response to user expectations and contextual requirements. This variability calls for a comprehensive quality model that enables stakeholder-driven weight recalibration to support product evaluation and selection. However, existing approaches do not simultaneously account for GenAIspecific attributes, user-centric quality priorities, and domain-adaptive evaluation mechanisms. To bridge this gap, this study proposes the User-Centric Generative AI Quality Model (UC-GAIQM), a domainadaptive framework in which Analytic Hierarchy Process (AHP) weights can be recalibrated to reflect quality priorities across different application scenarios and user profiles. The proposed model was developed through a mixed-methods, three-phase research design. In the first phase, a Systematic Literature Review (SLR) and Multivocal Literature Review (MLR) established the theoretical foundation. In the second phase, a quantitative survey of active GenAI users (n = 111) validated eight quality dimensions through exploratory and confirmatory factor analysis (alpha = 0.94, KMO = 0.88, CFI = 0.943). In the third phase, a three-round expert-driven Delphi study confirmed the structural validity of the model (Kendall's W = 0.84), and an AHP study demonstrated the weight recalibration mechanism. UC-GAIQM comprises eight quality dimensions and thirty sub-dimensions aligned with key ISO/IEC standards, the NIST AI Risk Management Framework, and the EU AI Act. The results demonstrate that the proposed model facilitates dynamic, context-sensitive evaluation of GenAI products by enabling quality priority adaptation across application domains.