To Satisfy or Clarify: Enhancing User Information Satisfaction with AI-Powered ChatGPT ^†

Abstract

The effective use of AI-powered ChatGPT in higher education highlights its potential as a knowledge acquisition tool, yet it brings to the fore the less-explored area of user information satisfaction (UIS), a gap addressing the development of the UIS model tailored to ChatGPT. In this foundational research grounded in UIS theory, a model was proposed based on critical factors to guide educational stakeholders in investigating issues such as plagiarism and ethical use. By analyzing responses from a diverse group of Indonesian users employed in higher education and using Structural Equation Modeling with Smart-PLS 4.0, we identified five key determinants: completeness, precision, timeliness, convenience, and format. The determinants affect user satisfaction, while traditionally emphasized factors such as accuracy and reliability do not significantly impact the academic utilization of ChatGPT. The results advocate for reliance on ChatGPT with limitations in regard to accuracy and suggest a change in the academic use of technology. The results also offer a perspective that blends theoretical innovation with practical application in educational settings.

1. Introduction

Since its emergence in the field of education in late 2022, AI-powered ChatGPT has drawn attention to its benefits and challenges. Studies have underscored risks, including plagiarism, misinformation [1], and ethical dilemmas [2], leading to hesitancy in its adoption in higher education. However, contrasting with these negative implications, there is evidence of ChatGPT’s academic utility. As Roose [3] suggested, ChatGPT could be ‘the teacher’s best friend,’ aiding in tasks such as creating outlines, sparking discussion ideas, and saving time on material preparation. This juxtaposition leads to the provisional view that ChatGPT, while potentially controversial, could be an indispensable, transformative AI tool in academia, bringing about what can be termed a ‘Groundbreaking Performance Transformation’ in future educational environments.

ChatGPT generates information in the form of text and images. ChatGPT is often considered a groundbreaking AI-powered tool in terms of user productivity, but this premise warrants further examination. This urgency arises as previous studies on ChatGPT have primarily focused on its benefits [4], risks [5], and academic integrity implications for users [6], among other aspects [7,8,9]. However, the information satisfaction generated by ChatGPT from a user’s perspective has not yet been thoroughly evaluated. Therefore, to identify a neglected gap, a model introduced by Ives et al. [10] was used to measure User Information Satisfaction (UIS) related to this information system. We investigated UIS concerning the generative information from ChatGPT for academic purposes, too.

A UIS model was constructed for the information and systems of ChatGPT to determine seven measures, namely, completeness, precision, timeliness, convenience, format, accuracy, and reliability, in the UIS framework [10]. This study is the first to examine the UIS of ChatGPT since it was introduced to the market. The urgency of this research lies in the following: Users (consumers) need to attain satisfaction with the ChatGPT system, as measured by various factors. Additionally, a pragmatic evaluation of the current system reflects its sustainability and continued use from the perspectives of educational organization policies, the scholarly publishing environment, and other academic activities. The results of this study contribute to the development of a User Information Satisfaction (UIS) model for ChatGPT systems based on the seven analyzed factors and their impact on user satisfaction. For individual ChatGPT users, organizations, and governments, policies can be made based on the results.

2. Theoretical Background

2.1. UIS

UIS was introduced by Ives et al. [10] to evaluate information systems. Since its inception, numerous studies have been conducted to investigate users’ levels of satisfaction with information systems across various sectors [11,12,13]. Notably, Ang and Soh [14] explored how UIS was measured in organizations, focusing on employee satisfaction for those without a computer studies background. This underscores the importance of UIS for information system users. Consequently, the applicability of the UIS concept can be extended to measure user satisfaction with respect to ChatGPT systems. Ives et al. [10] defined UIS for users as believing in the ability of the information system to meet their information requirements. UIS is a critical tool for evaluating the effectiveness of information systems used in organizations. Ives et al. [10] also emphasized the necessity of efforts to standardize UIS measurements for comparative studies for different systems and contexts. Therefore, applying UIS to the context of ChatGPT is crucial, given the lack of investigation of UIS from the perspective of ChatGPT users. Thus, it is necessary to explore how ChatGPT users perceive this system’s ability to meet their information needs, thereby contributing valuable insights to the effectiveness of ChatGPT as an information tool.

2.2. Theoretical Framework

Previous studies have identified different UIS measures depending on the context and information systems (IS) under investigation. For example, Galletta and Lederer [15] identified 23 measures for assessing user satisfaction regarding the implementation and effectiveness of IS (Figure 1). Laumer et al. [11] identified four factors for measuring users’ satisfaction with enterprise content management systems; they also conducted other diverse studies [10,11,12]. Owing to this research, the theoretical framework for measuring UIS is well established. However, there has been little measurement of UIS for AI-powered systems, including ChatGPT. Thus, we determined UIS measures for ChatGPT, providing information on OpenAI for users, developers, and companies whose operations involve the use of Chatbots. This attempt broadens the scope of UIS applications and contributes to the optimization of AI-powered systems in terms of meeting user needs. The developed measures can be used to understand how these technologies are tailored and improved, enhancing their utility and user satisfaction in various contexts.

Based on the concept of UIS defined in previous studies [4,10,11], we determined and tested seven measures for assessing the ChatGPT. These seven UIS measures were combined into a user satisfaction model. The UIS measures for ChatGPT include completeness, precision, timeliness, convenience, format, accuracy, and reliability. These measures were evaluated in terms of user satisfaction and organized within a conceptual research framework, as depicted in Figure 1. The result provides a holistic understanding of how users interact with and perceive the ChatGPT system. By analyzing these diverse measures, key areas of strength and potential improvement for ChatGPT were identified to enhance its effectiveness as a user-oriented tool. The results are expected to offer valuable insights into the optimization of ChatGPT and similar AI-powered systems.

3. Hypotheses

ChatGPT UIS Measures and Satisfaction

We employed seven User Information Satisfaction (UIS) measures—completeness, accuracy, precision, reliability, timeliness, convenience, and format—to test user satisfaction. By doing so, we examined how each of these dimensions contributes to the overall satisfaction of users interacting with ChatGPT. This approach enables a detailed exploration of this system’s efficacy and user experience, offering insights that can guide future enhancements and modifications to optimize ChatGPT for its users.

The completeness of ChatGPT’s responses refers to its thoroughness and comprehensiveness in addressing user queries. Users value detailed and comprehensive answers, as they foster an understanding of a given subject [16]. The more complete a response provided by ChatGPT to its users, the higher the likelihood of user satisfaction, emphasizing the importance of depth and breadth in information delivery. Therefore, the following hypothesis was proposed.

H1. 

The more complete the responses provided by ChatGPT, the higher the user satisfaction.

Accuracy was conceptualized as the correctness and truthfulness of ChatGPT’s responses. Accurate information is essential for users [4], particularly in decision-making. Thus, we posited a direct positive relationship between the accuracy of information provided and the level of user satisfaction, underlining the significance of providing correct and reliable answers to enhance user experience. Therefore, the following hypothesis was proposed.

H2. 

The higher the accuracy of ChatGPT’s responses, the greater the user satisfaction.

Precision is related to the relevance and specificity with which ChatGPT responds to a user’s inquiries. Users prefer targeted and relevant answers that address their specific concerns [17]. The more precise the information provided, the greater the user satisfaction, highlighting the value of tailor-made and context-specific responses. Therefore, the following hypothesis was proposed.

H3. 

The precision of ChatGPT’s responses in directly addressing the user’s specific queries positively influences user satisfaction.

Reliability refers to the consistency and dependability of ChatGPT’s responses over time. Research on user interaction indicates that consistent performance builds user trust and satisfaction [18]. This underscores the importance of ChatGPT being able to maintain a stable and dependable output to keep users satisfied. Therefore, the following hypothesis was proposed.

H4. 

The more reliable the information provided by ChatGPT, the higher the level of user satisfaction.

Timeliness focuses on the speed of ChatGPT’s responses. Quick and prompt replies have been posited to increase user satisfaction [19]. Timeliness is highly valued in the context of digital communication. Users often equate rapid responses with efficiency and effective service, thus enhancing their overall satisfaction with a tool. Therefore, the following hypothesis was proposed.

H5. 

The greater the timeliness of ChatGPT, the greater the level of user satisfaction.

Convenience in relation to using ChatGPT includes aspects such as ease of use and accessibility. We posited that convenience plays a significant role in user satisfaction. Studies on human–computer interaction have suggested that user-friendly and accessible tools significantly enhance user experience and satisfaction. This implies that the more convenient it is to interact with ChatGPT, the higher the level of user satisfaction. Therefore, the following hypothesis was proposed.

H6. 

Greater convenience in interacting with ChatGPT will significantly boost user satisfaction.

The format of ChatGPT’s responses, including clarity, organization, and presentation, impacts user satisfaction. Well-structured and presented information aids comprehension and user satisfaction [20]. A user-friendly and well-organized response format positively influences the level of user satisfaction, as it facilitates easier understanding and interaction. Therefore, we proposed the following hypothesis.

H7. 

The format in which ChatGPT presents its responses, if clear and well organized, positively impacts user satisfaction.

4. Methodology

4.1. Measures

We adopted the UIS concept [10] as the foundational framework for developing this conceptual research structure. We used the seven UIS measures determined in previous studies, making modifications and adjustments specific to this study’s purposes. This approach was necessary due to the absence of measures for evaluating the UIS of ChatGPT. We formulated 21 questions in four dimensions, modified from the work by Bhattacherjee [21]. For completeness, timeliness, and format, six items were adapted and refined from the work by Laumer et al. [11]. Accuracy was derived from Foroughi et al.’s work [4] and modified from the work by Ives et al. [10], including three items, namely, precision, reliability, and convenience, resulting in eight items. This adaptation ensured the measures were aligned with the unique characteristics and user interactions of ChatGPT, providing a robust and relevant evaluation of user satisfaction. Using this approach, a detailed and nuanced understanding of user satisfaction with respect to ChatGPT was acquired, contributing to the user experience research into AI-powered systems.

4.2. Data Collection

We adopted a quantitative methodology. A survey was designed for specific research participants. The data were gathered through an online survey based on a meticulously crafted questionnaire. This questionnaire comprised three parts. We asked for the consent of the participants, aligning with the study’s use of purposive sampling. To be eligible, the respondents had to meet two key criteria: (a) they had to have used ChatGPT for over six months for academic purposes, and (b) they had to have a background in higher education, serving as faculty members, students (both graduate and undergraduate), or postdoctoral researchers. The second section of the questionnaire was created to collect sociodemographic data such as gender, age, type of university, and occupation. The third section was formulated to allow the respondents to answer specific questions. Using this approach, data were collected to accurately represent the experiences of ChatGPT users in academic environments and to understand how different demographic groups use and perceive ChatGPT.

The data were collected from August to November 2023 from a pool of 508 responses. In total, 73% of the respondents were males. Respondents who were 21 to 35 years old had the highest degree of ChatGPT usage (75%). Altogether, 57% graduated from public universities and 43% from private institutions. Undergraduate students constituted 39% of the participants, and 32% were graduate students (master’s and doctoral candidates). Faculty members accounted for 15%, and 14% were postdoctoral researchers.

4.3. Data Analysis

We set up several stages for data analysis. To ensure validity and reliability, we conducted a Common Method Variance (CMV) test, employing Harman’s Single-Factor method [22], tested using SPSS version 26. We conducted validity, reliability, and hypothesis analyses using Structural Equation Modeling and Smart-PLS 4.0 software. Outer Loadings (OLs), Composite Reliability (CR), Average Variance Extracted (AVE), and Variance Inflation Factor (VIF) were also analyzed [23]. Discriminant validity was tested using the Fornell–Larcker Criterion [24], the Heterotrait/Monotrait Ratio/HTMT [25], a Cross-Loadings Matrix [23], R-squared values [26], and model fit evaluation [23].

5. Results

5.1. CMV

Before data analysis, CMV was tested using Harman’s Single-Factor test, in which all the measures were factored into a single dependent variable. If the total variance explained is below 50%, CMV is not considered a concern [22]. In total, 13.6% CMV was obtained without concern in this study. Thus, the credibility of the subsequent analysis was secured, ensuring the robustness and reliability of our data interpretation and results. This low CMV enhances the validity of this study, reinforcing the integrity of the research findings and their implications.

5.2. Validity and Reliability

Table 1. Convergent validity and reliability.

Constructs	OL	CA	CR	VIF	AVE
Accuracy	0.736–0.858	0.713	0.724	1.288–1.597	0.636
Completeness	0.735–0.964	0.790	0.936	1.385–1.385	0.734
Convenience	0.820–0.841	0.771	0.781	1.444–1.770	0.684
Format	0.870–0.871	0.781	0.681	1.314–1.363	0.758
Precision	0.792–0.850	0.736	0.655	1.044–1.317	0.526
Reliability	0.869–0.908	0.735	0.749	1.510–1.510	0.790
Satisfaction	0.797–0.832	0.822	0.822	1.678–1.913	0.652
Timeliness	0.860–0.889	0.709	0.699	1.392–1.392	0.765

Notes: The threshold for Outer Loadings (OLs) was >0.70; that for Cronbach’s Alpha (CA) was 0.70; that for Composite Reliability (CR) was >0.70; that for Variance Inflation Factor (VIF) was <0.30; and that for Average Variance Extracted (AVE) was >0.50.

displays the results for validity and reliability, which were validated in this study.

Discriminant validity was tested to evaluate the efficacy of the model developed in this study. Three methods were employed, as illustrated in

Table 2. Fornell–Larcker Criterion and HTMT.

Constructs	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)
ACC (1)	0.798	0.352	0.876	0.808	0.873	0.776	0.718	0.858
CMP (2)	0.251	0.857	0.277	0.449	0.655	0.374	0.176	0.282
CVC (3)	0.652	0.210	0.827	0.714	0.644	0.623	0.683	0.788
FMT (4)	0.564	0.299	0.521	0.871	0.875	0.598	0.784	0.691
PRR (5)	0.685	0.315	0.664	0.525	0.725	0.858	0.827	0.895
RLB (6)	0.557	0.284	0.476	0.423	0.558	0.889	0.537	0.249
STS (7)	0.551	0.146	0.555	0.586	0.586	0.420	0.808	0.682
TML (8)	0.601	0.215	0.584	0.475	0.577	0.661	0.517	0.875

Notes: The diagonal bolded values are the square root of the AVE, which was used for Fornell–Larcker Criterion. The italic values indicate the HTMT, with a threshold of <0.90. ACC, accuracy; CMP, completeness; CVC, convenience; FMT, format; PRR, precision; RLB, reliability; STS, satisfaction; TML, timeliness.

and

Table 3. Cross-loadings matrix.

Items/Contructs	ACC	CMP	CVC	FMT	PRR	RLB	STS	TML
ACR.1	0.795	0.253	0.519	0.468	0.511	0.499	0.427	0.514
ACR.2	0.858	0.179	0.523	0.482	0.559	0.411	0.488	0.479
ACR.3	0.736	0.171	0.524	0.396	0.574	0.431	0.400	0.450
CMP.1	0.239	0.964	0.207	0.268	0.313	0.288	0.159	0.224
CMP.2	0.191	0.735	0.141	0.270	0.210	0.171	0.063	0.114
CNV.1	0.496	0.201	0.820	0.407	0.480	0.351	0.364	0.412
CNV.2	0.533	0.173	0.841	0.421	0.600	0.419	0.488	0.480
CNV.3	0.578	0.154	0.820	0.458	0.551	0.402	0.500	0.537
FMR.1	0.489	0.307	0.421	0.870	0.419	0.371	0.510	0.387
FMR.2	0.492	0.213	0.485	0.871	0.494	0.365	0.511	0.440
PRC.1	0.576	0.203	0.568	0.418	0.850	0.457	0.515	0.486
PRC.2	0.577	0.203	0.569	0.439	0.838	0.480	0.491	0.503
PRC.3	0.277	0.473	0.218	0.285	0.792	0.235	0.182	0.191
RLB.1	0.498	0.287	0.398	0.371	0.445	0.869	0.340	0.557
RLB.2	0.494	0.224	0.446	0.381	0.540	0.908	0.402	0.615
STS.1	0.438	0.093	0.441	0.489	0.456	0.295	0.798	0.364
STS.2	0.444	0.082	0.481	0.445	0.489	0.354	0.832	0.444
STS.3	0.459	0.151	0.421	0.493	0.446	0.361	0.797	0.410
STS.4	0.439	0.147	0.448	0.467	0.500	0.345	0.802	0.450
TML.1	0.525	0.202	0.523	0.413	0.502	0.591	0.427	0.860
TML.2	0.528	0.176	0.499	0.418	0.509	0.568	0.475	0.889

. All diagonal and bolded values were greater than the intervariable correlation values. This suggested discriminant validity. Additionally, the HTMT values were below 0.90. These results confirm the distinctiveness of the factors of the model, ensuring that each measure captured a unique aspect of the phenomenon, which was crucial for ensuring the overall validity of the research findings.

Discriminant validity was tested using the Cross-Loading Matrix method (Table 3). The correlation of each construct with its respective measurement items was greater than that with cross items. Thus, discriminant validity was confirmed in this study. The differentiation of the constructs ensured the reliability and accuracy of the measurement model, ensuring that each construct was distinctively and accurately captured in the research framework.

The model was tested using an evaluation of model fit and R-squared values. The SRMR value was 0.070, the Chi-Square value was 1631.471, and the NFI was 0.783, all of which fell within the threshold limits [23]. Subsequently, the R-squared value was assessed to determine the power of independent constructs for predicting the dependent constructs. An R-squared value of 0.581 was obtained, indicating that the seven ChatGPT UIS measures accounted for 58.1% of the variance in satisfaction. This meets the criteria set by Falk and Miller [26], as the R-squared value was above the 0.10 benchmark. These results demonstrate the model’s good fit and underline the considerable explanatory power of the identified factors in determining user satisfaction with ChatGPT.

5.3. Hypothesis Testing

Table 4. Summary of hypothesis testing.

Hypothesis	β	T	Bootstrapping CI 97.5%		Decision
			Lower	Upper
CMP → STS	0.096 **	2.863	0.168	0.040	Accept
AC → STS	0.070	1.247	0.037	0.182	Reject
PRC → STS	0.245 ***	3.681	0.117	0.378	Accept
RLB → STS	0.016	0.400	0.118	0.086	Reject
TML → STS	0.138 **	2.396	0.025	0.249	Accept
CVN → STS	0.126 **	2.192	0.014	0.239	Accept
FMT → STS	0.323 ***	5.115	0.202	0.444	Accept

Notes: AC, accuracy; CMP, completeness; CVN, convenience; FMT, format; PRC, precision; RLB, reliability; TML, timeliness; STS, satisfaction. Significance levels: *** p < 0.001; ** p < 0.010.

and Figure 2 present a summary of the hypothesis testing. Accuracy (β = 0.070; T = 1.247) and reliability (β = 0.016; T = 0.400) were not significant predictors of satisfaction, leading to the rejection of hypotheses H2 and H4. Completeness (β = 0.096; T = 2.863), convenience (β = 0.126; T = 2.192), format (β = 0.323; T = 5.115), precision (β = 0.245; T = 3.681), and timeliness (β = 0.138; T = 2.396) significantly impacted satisfaction, supporting hypotheses H1, H3, H5, H6, and H7. Format and precision impacted satisfaction more than the other factors. These results highlight the varying degrees of influence of different UIS measures on user satisfaction, with several factors playing a more critical role than others in shaping user experience with respect to ChatGPT.

6. Discussion

We examined the UIS framework proposed by Ives et al. [10], which suggested that accuracy and reliability are essential for user satisfaction with respect to information systems. Contradicting this long-standing belief, the study’s results revealed that these aspects did not significantly shape user satisfaction in the utilization of ChatGPT. This indicated that the parameters determining satisfaction are changing with advancements in generative AI technologies. This is contradictory to findings reported in previous studies [10]. Other dimensions including completeness, convenience, format, precision, and timeliness influenced users’ satisfaction with ChatGPT. Users of ChatGPT valued the presentation and immediacy of information, not just its correctness or dependability. This underscores that with recent UIS theory adaptations, the changing landscape of user expectations is becoming more sophisticated [10,11].

7. Conclusions

Implication

The UIS framework was applied theoretically by examining the aspects of accuracy and reliability regarding user satisfaction as discussed by Ives et al. [10]. These factors did not significantly influence user satisfaction in the context of ChatGPT, pointing to a potential shift in the UIS paradigm as generative AI technology progresses. Such a result challenges and potentially redefines the established parameters of user satisfaction. The results of this study contribute to UIS by identifying the importance of completeness, convenience, format, precision, and timeliness. The evolving priorities of users were also redefined by advanced technology. The theoretical framework of UIS contributes attributes to the current digital interaction milieu and enhances our understanding of user satisfaction in regard to information systems.

The results of this study can be used to understand the application of ChatGPT in organizational and individual contexts. At an organizational level, the traditional emphasis on accuracy and reliability is changing, encouraging entities to recalibrate their focus toward completeness, convenience, format, precision, and timeliness in their information systems. Such a realignment leads to the creation of interfaces that are more attuned to user needs, potentially improving users’ utilization of and satisfaction with AI-powered tools. This will result in the development of more intuitive and efficient systems, thereby enhancing the return on investment for AI technologies.

For those who regularly engage with AI programs such as ChatGPT, the results reported herein underscore the importance of mastering the features of this system that enhance its immediate utility. Understanding how to effectively navigate and extract value from AI interactions is crucial, suggesting that user education is mandatory in these aspects. As end-users become more adept at utilizing the streamlined and user-friendly features of such technologies, they can optimize efficiency and productivity, reaping the full benefits of the evolving digital tools at their disposal. This knowledge enables users to have realistic expectations and maximize the utility of AI in their work and learning environments, solidifying their role as competent navigators in the advancing technological landscape.

Despite its contributions, this study had limitations, such as a confined focus on ChatGPT while disregarding various other generative AI platforms. Furthermore, this study’s reliance on self-reported measures of satisfaction introduced biases that hindered its ability to capture the nuanced reactions of users to AI interfaces. Further studies are necessary to include a broader range of AI tools and technologies in this field of research, thereby enriching the generalizability of the findings. Additionally, it is necessary to incorporate objective usage data to provide an understanding of user satisfaction and behavior and offer a holistic view of how individuals and organizations interact with AI systems. A longitudinal study must be conducted to assess users’ satisfaction with AI over time as they become more accustomed to these technologies.