Next Article in Journal
Assess Spatial Equity Considering the Similarity Between GIS-Based Supply and Demand Maps: A New Framework with Case Study in Beijing
Previous Article in Journal
A Model of Building Changes to Support Comparative Studies and Open Discussions on Densification
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJGI
Volume 14
Issue 4
10.3390/ijgi14040156
ijgi-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Use and Effectiveness of Chatbots as Support Tools in GIS Programming Course Assignments

by
Hartwig H. Hochmair
Geomatics Sciences, Fort Lauderdale Research and Education Center, University of Florida, 3205 College Ave, Davie, FL 33314, USA
ISPRS Int. J. Geo-Inf. 2025, 14(4), 156; https://doi.org/10.3390/ijgi14040156
Submission received: 6 January 2025 / Revised: 16 March 2025 / Accepted: 30 March 2025 / Published: 2 April 2025
Browse Figures
Review Reports Versions Notes

Abstract

:
Advancements in large language models have significantly transformed higher education by integrating AI chatbots into course design, teaching, administration, and student support. This study evaluates the use, effectiveness, and perceptions of chatbots in a Python-based graduate-level GIS programming course at a U.S. university. Students self-reported perceived improvements in skills and the use of different help resources across three home assignments of varying complexity and spatial context. In group discussions, students shared their experiences, strategies, and envisioned future applications of chatbots in GIS programming and beyond. The results indicate that prior programming experience enhances students’ perception of assignment usefulness, and that chatbots serve as a partial replacement for traditional help resources (e.g., websites) in completing assignments. Overall, students expressed positive sentiments regarding chatbot effectiveness, especially for complex spatial tasks. While students were optimistic about the potential of chatbots to enhance future learning, concerns were raised about overreliance on AI, which could hinder the development of independent problem-solving and programming skills. In conclusion, this study offers valuable insights into optimizing chatbot integration in GIS education.

1. Introduction

A chatbot is an advanced computer program designed to enable conversational interactions with humans, utilizing a range of artificial intelligence (AI) technologies such as natural language processing (NLP), machine learning, deep learning, artificial neural networks, and generative AI [1,2,3]. The primary goal of a chatbot is to provide real-time assistance by interpreting user inquiries phrased in natural language and generating contextually appropriate responses [4]. Chatbots and virtual assistants have been increasingly integrated into various sectors including education, where the widespread adoption of large language models (LLMs), such as in ChatGPT, has transformed this field by augmenting traditional teaching techniques with AI-enabled personalized learning solutions and effective student interactions [5,6]. Numerous literature reviews emphasize the advantages of AI-powered chatbots in areas such as homework support, personalized learning, and skill development [3] while also examining their applications, platforms, and interaction styles [7].
Chatbot technology has been shown to significantly improve student engagement, learning outcomes, and knowledge retention across various educational subjects, such as computer programming [8,9,10]. AI, including chatbots, offers also a wide range of education opportunities in geomatics, spanning application areas such as GIS, spatial analysis, remote sensing, computer vision, and geodesy [11]. Students in computer programming courses use chatbots primarily for tasks such as code generation, debugging, checking, and explanation, as well as to deepen their conceptual understanding [12]. While some studies have reported improved student performance and knowledge retention through the integration of chatbots in programming courses [13,14], others have found no significant effect [15,16] or even a decline in code quality [12]. Research has also shown that students with prior programming experience use chatbots more effectively for a broader range of tasks compared to programming novices [12]. The quality of chatbot-generated code varies, with some assignments yielding highly readable and well-structured code that requires minimal modification [17,18], while others produce code with a low correctness rate (68%) on the first attempt and a relatively small portion (57%) of readable code [19].
The spatial problem-solving abilities of LLMs and multimodal LLMs have been explored across a variety of tasks, including map reading and analysis, toponym resolution, spatial reasoning, spatial literacy, GIS theory, multi-criteria decision making, and the generation of programming code for spatial operations [20,21,22,23,24]. Despite these advancements, the integration of chatbots into geoscience education remains underexplored, especially when compared to the field of computer programming. A few geoscience-related studies have explored the use of chatbots in creating assignment questions, such as in the context of sea-level rise and inundation mapping [25] or geography curriculum development [26]. They also offer insights into how “Geography,” a GPT-4-based, fine-tuned chatbot, could be utilized for classroom learning tasks [27]. Table 1 summarizes related studies that discuss the application of chatbots in programming, GIS, and geography education.

1.1. Research Motivation and Contributions

A review of previous research highlighted several gaps, which motivated this study. The most prominent gap is the lack of studies examining students’ behavioral patterns in using chatbots for coursework in a GIS context. Specifically, little is known about how students use chatbots in their coursework, for which tasks, how chatbot usage impacts learning and comprehension, and how students perceive chatbot usefulness in completing assignments. Existing studies on chatbots in GIS and geography education primarily focus on their role in assisting educators with lecture or curriculum preparation [25,26] without exploring students’ actual use of chatbots during coursework. This study aims to address this gap by gathering data on students’ experiences with chatbots for both spatial and non-spatial programming tasks, their use of help resources, and their learning progress through GIS programming assignments.
Additionally, while previous research has explored the effectiveness of chatbots as teaching tools in programming courses and the quality of chatbot-generated code in class assignments [17,18,19], these studies have not considered the spatial context. Programming tasks in GIS and spatial analysis require specialized knowledge of spatial concepts like coordinate systems, topology, and overlay operations, making these tasks particularly challenging. Therefore, another key contribution of this research is to examine how chatbot usefulness and the use of help resources may vary based on the spatial context of programming tasks.
Specifically, this study aims to address these research gaps by exploring the following six key questions:
  • RQ1: How is prior student programming experience related to perceived improvements in programming skills?
  • RQ2: How does the use of programming resources evolve throughout the course?
  • RQ3: Does the introduction of chatbots prior to programming tasks negatively affect students’ code comprehension abilities?
  • RQ4: How is chatbot helpfulness related to the complexity of assignments and their spatial focus?
  • RQ5: Is the quality of chatbot responses linked to the extent of code modifications needed to achieve functional code?
  • RQ6: What sentiment scores emerge from student discussions regarding the usefulness of chatbot responses?
The study will also gather insights from student discussions on the usefulness of chatbot-generated code (Topic 1), strategies for enhancing chatbot responses (Topic 2), and students’ envisioned future applications of chatbots in education (Topic 3).

1.2. Paper Organization

The remainder of the paper is structured as follows: The next section provides an overview of the GIS programming course, including the student tasks from which the data are drawn, and the analysis methods used to study perceptions and use of chatbots throughout the course. Section 3 presents the results from the programming tasks and group discussions, followed by a discussion and interpretation of the findings in Section 4. Section 5 concludes the paper and offers directions for future research.

2. Materials and Methods

Data to address the six research questions were collected through several graded tasks in a 3-credit GIS programming graduate course at the University of Florida, offered during the Fall 2024 semester (Figure 1). Specifically, the data were drawn from: (1) surveys completed by students after they finished the programming portion of three selected assignments (see Section 2.2.1), (2) a quiz with eight programming-related questions taken after completing Assignment 2 (see Section 2.2.1), and (3) a group discussion with 2–3 students on chatbot usage (see Section 2.2.2). The analysis methods included statistical tests and linear regression models based on survey responses and quiz results (see Section 2.3.1), as well as sentiment analysis and summaries of group discussions (see Section 2.3.2).

2.1. Course Content and Enrollment

The GIS programming course is an elective, open to students from all disciplines, and requires students to have a working knowledge of ESRI’s ArcGIS Pro and basic programming experience in any language. The course enrolled 26 students, of whom 24 consented to the use of their assignment materials for this research. The curriculum included 12 weekly modules, each accompanied by a home assignment covering the following topics:
  • Week/Assignment 1–3: Basics of Python programming and editing, including Python editors, syntax, variables, control structures (loops, conditional statements), data types, object oriented programming, and reading/writing from/to text files.
  • Week/Assignment 4–8: ArcPy functionality, including spatial data management, table and feature class manipulation through cursors, manipulation of feature geometries, raster processing, and Python access to built-in geoprocessing tools.
  • Week/Assignment 9: Customized ArcGIS script tools.
  • Week/Assignment 10: ArcGIS and Jupyter notebooks.
  • Week/Assignment 11: Open libraries for spatial analysis, including Shapely and GeoPandas.
  • Week/Assignment 12: Open-source geospatial scripting in QGIS.
Home assignments accounted for 87% of the course grade, an AI-related group discussion and written report for 5%, and contributions to weekly assignment-related online discussion posts for 8%.

2.2. Student Tasks

2.2.1. Assignments

The weekly topics and assignments progressively increased in complexity, gradually incorporating more spatial context and related data handling and operation methods. Since part of the study aimed to assess how the use of programming resources evolved throughout the course and how chatbot helpfulness correlated with assignment complexity and spatial context, three assignments (2, 3, and 7) with varying spatial complexity were selected for analysis, as described below. Each of the three assessed assignments consisted of a programming task followed by a survey.
Assignment 2 required students to develop a Python script that read a user-provided number between 1 and 20 and computed its factorial using both a for loop and a while loop (Figure 2a). The program also needed to verify that the input was numeric and within the specified range. This task focused on foundational Python concepts, such as defining functions, using loops, and applying if statements, without incorporating spatial or GIS-related concepts. For half of the students, the handout instructions directed them to first view two brief introductory videos on generative AI. The first video provided an overview of LLMs and introduced six chatbots available through the university’s NaviGator AI platform, including GPT-4, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama 3.1. The second video demonstrated practical applications of these chatbots, such as analyzing spreadsheet data or generating Python code for tasks like solving linear equations or computing distances between points in .json format. For this group, the handout explicitly stated that they could use any chatbot as an auxiliary tool to complete the coding task.
The other half of the class was directed to the same videos in Assignment 3, but not during Assignment 2. This division allowed the study to assess whether promoting chatbot usage influenced students’ code comprehension, which was evaluated through an online quiz administered with Assignment 2.
The programming task in Assignment 3 required students to develop a Python script that read topological road network data from a text file and computed the degree of each network node using a Python dictionary. The text file provided, for each road segment, the arc feature ID, the FROM-node ID, and the TO-node ID (Figure 2b). By counting the number of roads incident to each node, the degree could be computed. The script was also required to return, based on user input, the degree of a specified node and handle incorrect data inputs. This task was more complex than Assignment 2, as it involved aspects of topological network analysis (without utilizing the ArcPy module) and spatial data manipulation.
The programming task in Assignment 7 involved constructing polygons from point features grouped by a common street block attribute value (Figure 2c). The resulting polygons had to be saved as a feature class in a file geodatabase, and their areas were to be printed in the output window. This task required reading and writing features from and to a file geodatabase using search and insert cursors, adding point geometries to arrays and converting them into polygons, and specifying spatial reference systems. These steps utilized various ArcPy object classes. As the most complex of the three assignments, it encompassed a range of spatial data handling and processing tasks. For this assignment, students were instructed to first input the handout information into a chatbot to generate code for the task. Half of the class used GPT-4 Turbo, while the other half used Gemini 1.5 Pro. In the next step, students modified the chatbot-generated code to ensure it was fully functional and submitted both versions as part of their assignment.
The quiz associated with Assignment 2 consisted of 8 questions and assessed the programming skills necessary to complete the assignment. For example, one of the quiz questions, along with possible solutions, is shown in Figure 3. Students received full credit for the quiz portion of the assignment as long as they answered all 8 questions, regardless of whether their answers were correct or incorrect. This approach was intended to reduce pressure on students who might otherwise rely on chatbots to answer the questions. Additionally, to encourage students to think through the given problem and discourage reliance on running the code in a Python editor or chatbot, the code in the quiz questions was provided as an image rather than as text.
Along with the online quiz for Assignment 2, students were required to complete an online survey after finishing the programming tasks for Assignments 2, 3, and 7 to receive full course credit. To encourage honest responses and prevent students from providing expected answers, the survey introduction emphasized that there were no right or wrong answers and that only the completeness of the responses would be graded, not the content itself. The surveys collected various types of information, with specific questions varying across the different assignments:
  • Prior programming experience (0…10)—Assignment 2 only.
  • Prior Python experience (0…10)—Assignment 2 only.
  • Assignment completion time—Assignments 2 and 3 only.
  • Perceived enhancement in programming skills (0…10)—Assignments 2 and 3 only.
  • Use of help resources: Lecture material (Yes/No), other websites (Yes/No), chatbot (Yes/No), instructor/TA (Yes/No).
  • Helpfulness of resources: Lecture materials (0…10), other websites (0…10), chatbot (0…10), instructor/TA (0…10).
  • Quality of chatbot generated code (0…10)—Assignment 7 only.
  • Handling of problems with chatbot generated code (Text)—Assignment 7 only.

2.2.2. Group Discussion

While the analyzed surveys and quiz results were tied to specific assignments and corresponding programming tasks, the group discussion was scheduled toward the end of the semester to provide a more comprehensive student perspective on chatbot use throughout the course. For the discussion, students were randomly grouped into pairs or small groups of three. Each group was tasked with discussing their experiences with chatbots during the course and summarizing their findings in a report of 300 words or more. The essay-style structure of the completed discussions allows for a deeper understanding of chatbot and resource usage from the students’ perspective, offering more nuanced insights than the numerical responses from the surveys. The group discussion was instructed to address the following three topics:
  • Usability of chatbots for course work (e.g., relevance, accuracy, completeness of responses).
  • Strategies or techniques for improving the usability of chatbot responses.
  • The potential use of chatbots for educational purposes in the future.

2.3. Analysis Methods

2.3.1. Assignment Surveys

The results presented in this study are primarily based on the analysis of data collected from three surveys associated with programming Assignments 2, 3, and 7. The analysis explores the statistical relationships between various response variables within each survey (e.g., prior programming experience vs. perceived skill improvement) and compares responses across surveys (e.g., use of help resources for different assignments). The statistical methods employed include Pearson correlation, the independent samples t-test, the Chi-square test of independence, the Kruskal–Wallis test, followed by Dunn’s post hoc test, and linear regression. The statistical tests aimed to draw statistical inferences that could be generalized to all students in the course, based on the sample of students who consented to data use and completed the survey questions (which some students occasionally omitted).
Effect sizes were computed to evaluate the magnitude of the observed effect relative to the variability inherent in the data. For the independent samples t-test, the effect size (Cohen’s d) is calculated as d = t (sqrt(1/n1 + 1/n2)), where n1 and n2 are the size of the first and second sample, respectively, and t is the test statistic value from the t-test [29]. The effect size for the Chi-squared test is assessed using Cramer’s V, computed as V = sqrt(X2/(n × f*), where df* is the smaller value between the number of rows minus 1 or columns minus 1 [30]. The interpretation Cramer’s V depends on df* and the effect size can be considered medium if V falls between 0.30 and 0.50 for df* = 1 [31].
For the Kruskal–Wallis test, the effect size (η2) is computed as η2 = (H − k + 1)/(n − k), where H is the Kruskal–Wallis test statistic, k is the number of groups, and n is the total number of observations [30]. η2 values range from 0 to 1, with values between 0.01 and 0.06 indicating a small effect, 0.06 to 0.14 a moderate effect, and values ≥ 0.14 a large effect.
The survey for Assignment 7 asked students to assess the perceived quality of the code generated by the chatbot. Using Pearson correlation, the perceived code quality was compared to the similarity between the initial chatbot-generated code and the modified code submitted by students. The submitted codes were compared using the Levenshtein distance, which measures the minimum number of single-character edits (insertions, deletions, or substitutions) needed to convert one string into another [32]. A higher Levenshtein distance indicated that the code underwent more changes compared to a lower distance. These modifications could include necessary adjustments to ensure the code executed correctly, such as correcting file paths to data sources. Additionally, optional changes might involve renaming variables, adding or removing comments, or incorporating error-handling procedures.

2.3.2. Group Discussions and Sentiment Analysis

To assess students’ sentiments regarding the usefulness of chatbot responses (Topic 1), 63 sentences related to this topic were extracted from nine group discussion reports and analyzed using VADER (Valence Aware Dictionary and sEntiment Reasoner) [33]. VADER is a rule-based sentiment analysis tool specifically designed to capture sentiments expressed in social media. It does not require training, as the model is based on a generalizable, valence-based, human-curated sentiment lexicon. The tool assigns positive, negative, and neutral sentiment scores to a given text, each ranging from 0 to 1, with the sum of all scores equaling 1. VADER also provides a composite score, known as the compound score, which is calculated by summing the valence scores of all words in the lexicon, adjusting according to predefined rules, and normalizing the result between −1 and +1. Text with a compound score of 0.05 or higher is classified as having a positive sentiment, scores of −0.05 or lower indicate negative sentiment, and scores between −0.05 and 0.05 are considered neutral [34,35]. Table 2 provides examples of VADER sentiment scores for sentences with positive, neutral, and negative sentiments from student discussions.
In addition to the sentiment scores for Topic 1, a summary of student discussion comments is provided, covering the quality of chatbot responses (Topic 1), strategies for improving chatbot responses (Topic 2), and considerations for the future use of chatbots in education (Topic 3).

3. Results

3.1. Programming Tasks

3.1.1. RQ1: Skill Improvement

The surveys for Assignments 2 and 3 asked students to rate their perceived improvement in programming skills associated with each assignment. The skill improvement ratings showed a moderate positive correlation with prior programming experience for both Assignment 2 (r(21) = 0.56, p = 0.005) and Assignment 3 (r(21) = 0.67, p < 0.001) (see Figure 4). These results indicated that students with prior programming experience perceived the assignments as more effective in enhancing their programming skills compared to students with little or no prior experience. Despite these correlations, responses in the survey for Assignments 2 showed that students had limited prior programming experience (M = 3.83, SD = 2.64), especially with Python (M = 2.45, SD = 2.50).

3.1.2. RQ2: Programming Resources

Figure 5 displays the percentages of students who consulted various help resources to complete the three programming tasks associated with the different home assignments.
While most students relied on lecture materials to complete all three assignments, the use of websites decreased significantly from 87.0% in Assignment 2 to 36.4% in Assignment 7. This suggested that online resources may be less effective for more specialized programming tasks, such as handling spatial data or working with proprietary libraries like ArcPy. In contrast to the declining use of websites, chatbot usage increased over the course, rising from 73.9% in Assignment 2 to 100% in Assignment 7. Although the increase in chatbot use across Assignments 2, 3, and 7 was partly influenced by assignment design, the decrease in website use was more likely a result of growing reliance on chatbots, especially for more complex tasks. The percentage of students (9.5%) who consulted the instructor or TA for Assignment 2 was low. This can be attributed to the basic nature of the Python programming task (loops, conditions), which is widely covered in online tutorials and websites, as it did not involve data reading or spatial data handling. As a result, students primarily relied on lecture materials and external websites. By contrast, Assignment 3, which involved topological and data processing operations on road network data, saw an increase in instructor/TA consultation to 40.9%, reflecting the task’s increased complexity and reduced reliance on external websites. For Assignment 7, which required finding an initial solution with a chatbot, instructor/TA consultation dropped again to 10.0%, indicating that chatbots can, to some extent, replace instructor feedback even for more challenging and specific programming tasks. A chi-square test of independence revealed that the change in the use of lecture materials across the three assignments was not significant (X2 (2, N = 68) = 2.204, p = 0.332). However, the changes in website use (X2 (2, N = 68) = 12.439, p = 0.002), chatbot use (X2 (2, N = 68) = 7.687, p = 0.021), and consultation of instructor/TA (X2 (2, N = 63) = 8.486, p = 0.014) were statistically significant, with medium effect sizes of V = 0.43, V = 0.34, and V = 0.37, respectively. Student performance on the three analyzed assignments was similar, as shown by the scores (maximum = 100) for Assignment 1 (M = 91.8, SD = 9.9), Assignment 2 (M = 90.4, SD = 13.2), and Assignment 3 (M = 93.8, SD = 10.0). This suggested that the various resources available for the assignments (e.g., chatbot solutions vs. instructor input) complemented each other, and that students did not need to utilize all available resources simultaneously to successfully complete an assignment.

3.1.3. RQ3: Code Comprehension

The results of the code interpretation quiz (maximum score = 8 points) were collected for two groups of students: those who were introduced to chatbots as part of Assignment 2 (Group 1) and those who were not (Group 2). The quiz results were found to be positively correlated with prior Python programming experience (r(18) = 0.45, p = 0.046) and negatively correlated with introduction to chatbots (r(18) = −0.45, p = 0.046). To examine the combined effect of these factors and their interaction on the quiz results, both were included as predictors in a linear regression model (Table 3).
The results suggested that the introduction of chatbots as a tool for programming assignments may have led to a reduced understanding of basic code structures, while prior Python experience did not significantly predict quiz scores. Although exposure to chatbots may have hindered the learning progress, students in Group 1 may have been less motivated to carefully consider the quiz questions, knowing that chatbots could assist with similar tasks in the future and that the quiz answers were not graded. To address this potential impact on student motivation, future studies should consider using graded quizzes to assess student learning, while ensuring that no chatbots or other technical aids are permitted.

3.1.4. RQ4: Chatbot Helpfulness

Figure 6 illustrates the mean perceived helpfulness of different resources in completing Assignments 2, 3, and 7, showing an increase in perceived helpfulness of chatbots as the complexity of the coding tasks escalated. A Kruskal–Wallis test indicated a significant difference in central tendency of perceived helpfulness of chatbots among the three assignments (H (2, n = 59) = 11.26, p = 0.004), with a large effect size of η2 = 0.017. Post hoc analysis using Dunn’s method with a Bonferroni correction for multiple comparisons revealed a significant difference between Assignment 2 and 7 (adjusted p = 0.002). By contrast, Kruskal–Wallis tests conducted on the other three help resources (lecture materials, websites, and instructor/TA) did not reveal a significant change in perceived helpfulness across assignments at the 5% significance level. This highlighted the particular effectiveness of chatbots in assisting students, especially with increasingly complex spatial programming tasks.
However, despite this merit of chatbots, the completion time for Assignment 2 was not significantly shorter when using chatbots (M = 3.55, SD = 2.66) compared to completing the assignment without them (M = 4.38, SD = 2.84), as indicated by the Wilcoxon rank sum test (w = 64, p = 0.378).

3.1.5. RQ5: Code Readiness

The instructions for Assignment 7 advised students to incorporate the programming guidelines from the handout into the chatbot prompt to generate Python code. However, some students chose to rephrase these instructions instead of copying them verbatim, leading to variations in the chatbot-generated code. The Levenshtein distance between the chatbot-generated code and the final submitted code showed a strong negative correlation with the perceived quality of the chatbot code (r(19) = −0.73, p < 0.001) (see Figure 7). This suggested that students associated a greater number of code edits with lower quality chatbot output, implying that the extent of code modifications can serve as an indicator of the generated code’s readiness and functionality. The perceived quality of the chatbot-generated code was similar for both GPT-4 Turbo (M = 8.0, SD = 1.3) and Gemini 1.5 Pro (M = 7.7, SD = 1.7). Similarly, the Levenshtein distance was comparable between the two models (GPT-4 Turbo: M = 565.2, SD = 470.6; Gemini 1.5 Pro: M = 546.2, SD = 618.7).
The survey for Assignment 7 asked students to report any issues encountered during the programming task, specifically related to the code generated by either GPT-4 Turbo or Gemini 1.5 Pro, and how they resolved those issues. The students’ comments did not highlight any issues specific to either GPT-4 Turbo or Gemini 1.5 Pro. Analysis of this feedback, along with a comparison between the original chatbot-generated code and the final submitted code, revealed that the most common modification involved updating the file path to the geodatabase. Students typically replaced placeholder paths with the correct, actual paths to ensure that the code functioned properly.
Many of the other code modifications were optional and aimed at enhancing functionality or improving code structure and readability without altering the code outcome. These changes included adjusting environment settings to allow overwriting of existing feature classes, implementing error-handling procedures (e.g., try and except statements) to check for the presence or successful creation of feature classes or fields, adding or removing comments, reformatting printed messages, modifying syntax for handling function call arguments, using different classes for spatial data processing (e.g., replacing PointGeometry with Geometry objects), and introducing a main entry point for standalone execution through the if __name__ == ‘__main__’ syntax.

3.2. Discussion Results

3.2.1. Assessment of Response Usefulness (Topic 1, RQ6)

After conducting a manual plausibility check of the compound scores for the 63 sentences related to chatbot quality sentiments, four sentences were excluded from the analysis due to discrepancies. These sentences expressed a positive tone but received negative compound scores. For instance, the sentence “The chatbots acted as a personal tutor, answering questions and allowing us to learn through trial and error” received a compound score of −0.4019. Among the remaining 59 sentences, 66.10% were categorized as having a positive overall sentiment, 16.95% were neutral, and another 16.95% reflected a negative sentiment. The mean sentiment score of 0.30 indicated a generally favorable perception of chatbot usability (Figure 8).
Students expressed appreciation for chatbot effectiveness in debugging and the time saved in completing tasks. They highlighted the interactive nature of the chatbots and their constant availability, which provided valuable support at any time. The chatbots were recognized as helpful supplementary tools, best utilized alongside critical thinking and independent learning strategies. Additionally, students noted that subject knowledge played a crucial role in efficiently checking and refining chatbot-generated code when necessary.
Several student comments with negative sentiments raised concerns about the limited ability of chatbots to handle complex programming tasks. These challenges sometimes led to runtime errors or infinite loops, requiring multiple prompts and human intervention to resolve. Such issues were more frequent in tasks with fewer online resources for the AI model to reference.

3.2.2. Strategies for Response Improvement (Topic 2)

Students shared several prompting techniques they found effective in improving the quality of chatbot-generated code. Foremost among these was providing specific instructions, including details on the expected script structure, formatting style, and required functions or libraries. This highlighted the importance of formulating clear prompts and evaluating chatbot responses, which requires both expertise and subject knowledge [36]. Follow-up questions were also seen as valuable for refining code. For instance, when dealing with execution errors, students found it helpful to include error messages in the prompt or number sections of lengthy code to facilitate error explanation and resolution. As chatbots are capable of multi-turn conversations, students discovered that breaking complex queries into smaller parts often led to better results. However, they also observed that long conversations with numerous code snippets could cause the chatbot responses to become repetitive or erratic. In such cases, starting a new chat session and pasting the entire code often helped to resolve the issue.

3.2.3. Use of Chatbots for Future Education (Topic 3)

Students expressed a general consensus that they would use chatbots for educational purposes in the future. They valued the instant feedback that chatbots provide and their ability to adapt to various learning styles by offering information in different formats, such as text, images, and interactive simulations. However, many students cautioned about the need to strike a balance between utilizing AI’s capabilities and avoiding overreliance on it. While generative AI can accelerate problem solving and support learning, students expressed concern that excessive dependence on AI might stifle creativity and hinder the development of independent problem-solving and coding skills. Additionally, students emphasized the importance for instructors to highlight the value of self-improvement over easy solutions and to promote a balance between chatbot use and the development of human learning and problem-solving abilities. They also noted the necessity for users to critically assess whether they possess the appropriate background knowledge to evaluate the accuracy of chatbot responses, as these may contain errors or inaccuracies. Ultimately, students concluded that chatbots should serve as a complementary tool rather than a replacement for foundational programming skills.

4. Discussion

Regarding RQ1, analysis of both related surveys revealed that students with prior programming experience rated their improvement in programming skills through the assignments higher than students with less experience. Previous research supports the idea that prior knowledge plays a significant role in how students engage with learning technologies or tutoring systems [37]. Specifically, foundational subject knowledge and existing programming skills enable students to more effectively utilize AI chatbots for programming tasks [12]. Based on these findings, a practical course design recommendation is to break down programming assignments into smaller, more manageable exercises at the start of the course. This approach could enhance learning outcomes and increase the effectiveness of chatbots, particularly for students with less programming experience.
Regarding the use of help resources for assignment completion, the survey results showed consistent reliance on course materials across all three assignments, with a notable increase in chatbot usage and a corresponding decline in the use of external websites. While the increase in chatbot use can be partly attributed to specific assignment instructions (e.g., Assignment 7 directed students to start the task using an assigned chatbot), the significant drop in website usage—from 87% of students in Assignment 2 to just 36% in Assignment 7—may suggest that, for more specialized tasks (such as utilizing geometry classes from the proprietary ArcPy package or processing road network data), online resources become less relevant. In these cases, more general programming websites may not offer easily applicable code examples. Chatbot applications have the potential to replace certain aspects of human interaction and the role of instructors in education [38]. Our findings suggest that, in educational settings, chatbots are increasingly serving as substitutes for traditional help resources, such as programming websites or blogs, in assisting with assignment completion. This aligns with previous studies, which found that students tend to explore fewer traditional educational resources once they start using ChatGPT, often relying solely on it [28].
Previous research showed mixed results when it comes to the integration of chatbots in classroom activities and learning outcomes. Some studies have shown that ChatGPT enhances knowledge acquisition and retention across various disciplines, including engineering [35] and programming [10,14]. However, other research observed a decline in coding quality in student assignments due to ChatGPT access [12] or found no statistically significant impact of ChatGPT usage on student performance in programming courses [15,16,28]. The performance on the code interpretation quiz associated with Assignment 2 showed that introduction to chatbots was associated with a lower score on code comprehension after controlling for prior Python programming experience using a linear regression model. This finding aligns with concerns raised in the student discussions. For instance, one group noted that “overreliance on chatbots could hinder skill development,” while another mentioned, “we might have gained a stronger understanding of the programming language if we hadn’t relied on them so much.” These concerns mirror previous studies suggesting that dependence on chatbots may limit students’ exploration of problem-solving strategies and hinder the development of critical thinking skills [39,40]. Students also expressed worries that relying on ChatGPT for programming exercises could prevent them from acquiring the essential skills needed to succeed in exams [17].
Further supporting this, prior research found that dependency on chatbots for programming assignments can diminish the quality of output and negatively impact the learning of programming fundamentals, likely due to reduced time investment in the assignments [12]. Another study also highlighted a negative association between chatbot use and total number of study hours per day [41], potentially explaining the observed decrease in quiz performance. However, assignment designs that prevent chatbots from offering direct solutions—such as complex, step-by-step laboratory assignments or visual instructional materials—may help to foster greater computational reasoning abilities [10]. In their discussion reports, students emphasized the importance of having a solid understanding of the subject matter before using chatbots for coursework. This knowledge enables students to critically evaluate chatbot responses, which may sometimes be incomplete or inaccurate. The limited reliability and accuracy of chatbot responses remain a significant concern for their integration into educational environments [3].
The increase in the perceived helpfulness of chatbots across Assignments 2, 3, and 7 may stem from several factors. First, students’ growing familiarity with chatbots likely played a role, as they became more comfortable using the tool over time. Additionally, the limited availability of online resources for the more complex spatial tasks in Assignments 3 and 7 may have prompted students to rely more heavily on chatbots, increasing their perceived value. Moreover, as students’ knowledge of GIS programming expanded throughout the course, they may have been able to use chatbots more effectively [12], further enhancing the perceived usefulness of these tools.
The comparison of chatbot-generated code and finalized submitted code for Assignment 7, using the Levenshtein distance, revealed a wide range of code modifications. Fewer code edits made to the chatbot-generated code were positively correlated with higher perceived code quality. Some students made optional improvements, such as adding new functions, rearranging existing ones, inserting comments, or renaming variables, none of which were critical for the code’s functionality. These findings align with results from a previous study in a Java programming course, where ChatGPT’s generated code could be adapted with minimal effort for the programming tasks at hand [17]. This contrasts with an analysis of ChatGPT-generated code for 72 Scala coding exercises in a functional programming course, which found that nearly half of the correct answers were of limited use to students due to their difficulty in understanding or inefficiency [19]. While prior research has indicated that ChatGPT outperforms other chatbots like Bard/Gemini, Claude, or Copilot in coding and spatial analysis tasks [23,42], no significant differences in the perceived code quality were observed between GPT-4 Turbo and Gemini 1.5 Pro in Assignment 7 of this study.
The helpfulness of chatbot responses was largely linked to positive sentiments in the student discussions. This aligns with previous research, which found that students generally rated ChatGPT positively for its suitability in programming tasks, even when the code provided was occasionally incorrect, non-executing, or misleading [17].

5. Conclusions

Chatbots are increasingly transforming higher education by providing immediate assistance, detailed explanations, and additional educational resources. They have the potential to enhance critical thinking by introducing novel ways to engage with content, but they can also hinder it by encouraging passive acceptance of information [39]. In this study, students in a graduate GIS programming course used their critical thinking skills to evaluate the usability and helpfulness of chatbots for specific programming tasks and engaged in group discussions about their findings. The ability to critically analyze and assess information is a crucial skill that may be underdeveloped in STEM fields, where the emphasis often lies on learning and memorizing facts rather than engaging in research and the practical application of knowledge [43]. AI and chatbot technologies are continuously evolving and playing an increasingly significant role in educational settings, making it unrealistic to ignore their presence in education [44]. Therefore, ongoing research is essential to fully harness their potential for enhancing student learning outcomes. This study shows that students with a foundational understanding of the subject learn more effectively from chatbots. A basic grasp of the material enables students to verify chatbot responses and deepen their knowledge more efficiently. Therefore, AI tools should be introduced gradually, after foundational concepts are taught, to prevent overreliance on chatbots. Additionally, integrating critical thinking and discussion components is essential to enhance the overall learning experience [3]. By doing so, chatbots can be transformed into powerful pedagogical tools, rather than merely passive sources of information.
The average self-reported completion times of around three to four hours for the programming portion of Assignments 2 and 3 in this study indicated that, despite using chatbots, students still need to invest their own efforts to complete coding tasks. Chatbots can serve as a useful starting point for building basic code structures, but these are typically refined and improved through the students’ own problem-solving skills. In other words, chatbots are unlikely to fully replace a student’s effort in completing assignments.
Use of AI in the classroom environment also requires an introduction to ethical concerns, such as issues related to copyright, bias, academic integrity, and data privacy. Providing alternative help resources like discussion boards, lecture notes, and online tutorials can support diverse student learning styles and prevent overreliance on chatbots. Future studies will explore the use of chatbots in other GIS and geomatics courses, offering insights into the transferability of findings from the current GIS programming course, which had a relatively small enrollment and limited generalizability compared to larger courses. Future iterations may also incorporate additional methods to measure actual learning improvements through chatbot use in GIS coursework, rather than relying on student self-assessments, which were the primary focus of this study.

Funding

This research received no external funding.

Institutional Review Board Statement

The protocol was approved by the university’s Institutional Review Board (Protocol #: ET00043005) on 9 April 2024, which concluded that the research design met the criteria for exempt research.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The survey responses related to Home Assignments 2, 3, and 7 and the quiz questions related to Home Assignment 2 are available on FigShare at https://doi.org/10.6084/m9.figshare.28551017.v1 (accessed on 3 January 2025).

Conflicts of Interest

The author declares no conflicts of interest.

References

  1. Griol, D.; Callejas, Z. A Neural Network Approach to Intention Modeling for User-Adapted Conversational Agents. Comput. Intell. Neurosci. 2016, 2016, 8402127. [Google Scholar] [PubMed]
  2. Nirala, K.K.; Singh, N.K.; Purani, V.S. A survey on providing customer and public administration based services using AI: Chatbot. Multimed. Tools Appl. 2022, 81, 22215–22246. [Google Scholar] [PubMed]
  3. Labadze, L.; Grigolia, M.; Machaidze, L. Role of AI chatbots in education: Systematic literature review. Int. J. Educ. Technol. High. Educ. 2023, 20, 56. [Google Scholar]
  4. Pham, X.L.; Pham, T.; Nguyen, Q.M.; Nguyen, T.H.; Cao, T.T.H. Chatbot as an intelligent personal assistant for mobile language learning. In Proceedings of the 2nd International Conference on Education and E-Learning (ICEEL 2018), Bali, Indonesia, 5–7 November 2018; pp. 16–21. [Google Scholar]
  5. Rahman, A.; Alqahtani, L.; Albooq, A.; Ainousah, A. A Survey on Security and Privacy of Large Multimodal Deep Learning Models: Teaching and Learning Perspective. In Proceedings of the 21st Learning and Technology Conference, Jeddah, Saudi Arabia, 15 January 2024. [Google Scholar]
  6. Ilieva, G.; Yankova, T.; Klisarova-Belcheva, S.; Dimitrov, A.; Bratkov, M.; Angelov, D. Effects of Generative Chatbots in Higher Education. Inf. Fusion 2023, 14, 492. [Google Scholar]
  7. Kuhail, M.A.; Alturki, N.; Alramlawi, S.; Alhejori, K. Interacting with educational chatbots: A systematic review. Educ. Inf. Technol. 2023, 28, 973–1018. [Google Scholar]
  8. Deng, X.; Yu, Z. A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education. Sustainability 2023, 154, 2940. [Google Scholar]
  9. Tegos, S.; Demetriadis, S.; Karakostas, A. Promoting academically productive talk with conversational agent interventions in collaborative learning settings. Comput. Educ. 2015, 87, 309–325. [Google Scholar]
  10. Yilmaz, R.; Yilmaz, F.G.K. The effect of generative artificial intelligence (AI) based tool use on students’ computational thinking skills, programming self-efficacy and motivation. Comput. Educ. Artif. Intell. 2023, 4, 100147. [Google Scholar]
  11. Retscher, G. Exploring the intersection of artificial intelligence and higher education: Opportunities and challenges in the context of geomatics education. Appl. Geomat. 2025, 17, 49–61. [Google Scholar]
  12. Groothuijsen, S.; Beemt, A.v.d.; Remmers, J.C.; Meeuwen, L.W.v. AI chatbots in programming education: Students’ use in a scientific computing course and consequences for learning. Comput. Educ. Artif. Intell. 2024, 7, 100290. [Google Scholar]
  13. Winkler, R.; Hobert, S.; Salovaara, A.; Söllner, M.; Leimeister, J.M. Sara, the Lecturer: Improving Learning in Online Education with a Scaffolding-Based Conversational Agent. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 25–30 April 2020. [Google Scholar]
  14. Essel, H.B.; Vlachopoulos, D.; Tachie-Menson, A.; Johnson, E.E.; Baah, P.K. The impact of a virtual teaching assistant (chatbot) on students’ learning in Ghanaian higher education. Int. J. Educ. Technol. High. Educ. 2022, 19, 57. [Google Scholar] [CrossRef]
  15. Kosar, T.; Ostojić, D.; Liu, Y.D.; Mernik, M. Computer Science Education in ChatGPT Era: Experiences from an Experiment in a Programming Course for Novice Programmers. Mathematics 2024, 12, 629. [Google Scholar] [CrossRef]
  16. Sun, D.; Boudouaia, A.; Zhu, C.; Li, Y. Would ChatGPT-facilitated programming mode impact college students’ programming behaviors, performances, and perceptions? An empirical study. Int. J. Educ. Technol. High. Educ. 2024, 21, 14. [Google Scholar] [CrossRef]
  17. Haindl, P.; Weinberger, G. Students’ Experiences of Using ChatGPT in an Undergraduate Programming Course. IEEE Access 2024, 12, 43519–43529. [Google Scholar] [CrossRef]
  18. Ouh, E.L.; Gan, B.K.S.; Shim, K.J.; Wlodkowski, S. ChatGPT, Can You Generate Solutions for my Coding Exercises? An Evaluation on its Effectiveness in an undergraduate Java Programming Course. In Proceedings of the ITiCSE: Innovation and Technology in Computer Science Education, Turku, Finland, 7–12 July 2023. [Google Scholar]
  19. Popovici, M.-D. ChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course. Int. J. Hum.–Comput. Interact. 2024, 40, 7743–7754. [Google Scholar] [CrossRef]
  20. Tao, R.; Xu, J. Mapping with ChatGPT. ISPRS Int. J. Geo-Inf. 2023, 12, 284. [Google Scholar] [CrossRef]
  21. Hu, X.; Kersten, J.; Klan, F.; Farzana, S.M. Toponym resolution leveraging lightweight and open-source large language models and geo-knowledge. Int. J. Geogr. Inf. Sci. 2024, 1–28. [Google Scholar] [CrossRef]
  22. Mooney, P.; Cui, W.; Guan, B.; Juhász, L. Towards Understanding the Spatial Literacy of ChatGPT. In Proceedings of the ACM SIGSPATIAL International Conference, Hamburg, Germany, 13–16 November 2023. [Google Scholar]
  23. Hochmair, H.H.; Juhász, L.; Kemp, T. Correctness Comparison of ChatGPT-4, Gemini, Claude-3, and Copilot for Spatial Tasks. Trans. GIS 2024, 28, 2219–2231. [Google Scholar] [CrossRef]
  24. Li, F.; Hogg, D.C.; Cohn, A.G. Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark. In Proceedings of the The Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24), 20–27 February 2024; pp. 18500–18507. [Google Scholar]
  25. Redican, K.; Gonzalez, M.; Zizzamia, B. Assessing ChatGPT for GIS education and assignment creation. J. Geogr. High. Educ. 2025, 49, 113–129. [Google Scholar] [CrossRef]
  26. Wilby, R.L.; Esson, J. AI literacy in geographic education and research: Capabilities, caveats, and criticality. Geogr. J. 2024, 190, e12548. [Google Scholar] [CrossRef]
  27. van Staden, C.J. Geography’s ability to interpret topographic maps and orthophotographs, adoption prediction, and learning activities to promote responsible usage in classrooms. J. Geogr. Educ. Afr. 2025, 8, 16–32. [Google Scholar] [CrossRef]
  28. Xue, Y.; Chen, H.; Bai, G.R.; Tairas, R.; Huang, Y. Does ChatGPT Help With Introductory Programming? An Experiment of Students Using ChatGPT in CS1. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET ’24), Lisbon, Portugal, 14–20 April 2024; pp. 331–341. [Google Scholar]
  29. Enzmann, D. Notes on Effect Size Measures for the Difference of Means from Two Independent Groups: The Case of Cohen’s d and Hedges’ g. Available online: https://www.researchgate.net/publication/270703432_Notes_on_Effect_Size_Measures_for_the_Difference_of_Means_From_Two_Independent_Groups_The_Case_of_Cohen’s_d_and_Hedges’_g (accessed on 14 March 2025).
  30. Tomczak, M.; Tomczak, E. The need to report effect size estimates revisited. An overview of some recommended measures of effect size. Trends Sport Sci. 2014, 1, 19–25. [Google Scholar]
  31. Bobbitt, Z. How to Interpret Cramer’s V (With Examples). Available online: https://www.statology.org/interpret-cramers-v/ (accessed on 14 March 2025).
  32. Shuwandy, M.L.; Zaidan, B.B.; Zaidan, A.A.; Albahri, A.S.; Alamoodi, A.H.; Alazab, O.S.A.M. mHealth Authentication Approach Based 3D Touchscreen and Microphone Sensors for Real-Time Remote Healthcare Monitoring System: Comprehensive Review, Open Issues and Methodological Aspects. Comput. Sci. Rev. 2020, 38, 100300. [Google Scholar] [CrossRef]
  33. Hutto, C.J.; Gilbert, E. VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. Proc. Int. AAAI Conf. Web Soc. Media 2014, 8, 216–225. [Google Scholar] [CrossRef]
  34. Pano, T.; Kashef, R. A Complete VADER-Based Sentiment Analysis of Bitcoin (BTC) Tweets during the Era of COVID-19. Big Data Cogn. Comput. 2020, 4, 33. [Google Scholar] [CrossRef]
  35. Uddin, S.M.J.; Albert, A.; Tamanna, M.; Ovid, A.; Alsharef, A. ChatGPT as an educational resource for civil engineering students. Comput. Appl. Eng. Educ. 2024, 32, e22747. [Google Scholar]
  36. Zamfrescu-Pereira, J.D.; Wong, R.; Hartmann, B.; Yang, Q. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the CHI conference on human factors in computing systems (CHI ’23), Hamburg, Germany, 23–28 April 2023. [Google Scholar]
  37. Xia, Q.; Chiu, T.K.F.; Chai, C.S.; Xie, K. The mediating effects of needs satisfaction on the relationships between prior knowledge and self-regulated learning through artificial intelligence chatbot. Br. J. Educ. Technol. 2023, 54, 967–986. [Google Scholar]
  38. Alam, A. Should Robots Replace Teachers? Mobilisation of AI and Learning Analytics in Education. In Proceedings of the International Conference on Advances in Computing, Communication, and Control (ICAC3), Mumbai, India, 3–4 December 2021. [Google Scholar]
  39. Parsakia, K. The Effect of Chatbots and AI on The Self-Efficacy, Self-Esteem, Problem-Solving and Critical Thinking of Students. Health Nexus 2023, 1, 71–76. [Google Scholar] [CrossRef]
  40. Kasneci, E.; Sessler, K.; Küchemann, S.; Bannert, M.; Dementieva, D.; Fischer, F.; Gasser, U.; Groh, G.; Günnemann, S.; Hüllermeier, E.; et al. ChatGPT for good? On opportunities and challenges of large language models for education. Learn. Individ. Differ. 2023, 103, 102274. [Google Scholar] [CrossRef]
  41. Mosleh, S.M.; Alsaadi, F.A.; Alnaqbi, F.K.; Alkhzaimi, M.A.; Alnaqbi, S.W.; Alsereidi, W.M. Examining the association between emotional intelligence and chatbot utilization in education: A cross-sectional examination of undergraduate students in the UAE. Heliyon 2024, 10, e31952. [Google Scholar] [CrossRef]
  42. Manvi, R.; Khanna, S.; Mai, G.; Burke, M.; Lobell, D.; Ermon, S. GeoLLM: Extracting Geospatial Knowledge from Large Language Models. In Proceedings of the International Conference on Learning Representations (ICLR 2024), Vienna Austria, 7–11 May 2024. [Google Scholar]
  43. Sternberg, R.J. Critical Thinking in STEM Disciplines. In Critical Thinking in Psychology, 2nd ed.; Sternberg, R.J., Halpern, D.F., Eds.; Cambridge University Press: Cambridge, UK, 2020; pp. 309–327. [Google Scholar]
  44. Kooli, C. Chatbots in Education and Research: A Critical Examination of Ethical Implications and Solutions. Sustainability 2023, 15, 5614. [Google Scholar] [CrossRef]
Ijgi 14 00156 g001
Figure 1. Analysis flowchart.
Figure 1. Analysis flowchart.
Ijgi 14 00156 g001
Ijgi 14 00156 g002
Figure 2. Evaluated programming assignments.
Figure 2. Evaluated programming assignments.
Ijgi 14 00156 g002
Ijgi 14 00156 g003
Figure 3. Quiz question as part of Assignment 2.
Figure 3. Quiz question as part of Assignment 2.
Ijgi 14 00156 g003
Ijgi 14 00156 g004
Figure 4. Perceived programming skill improvement associated with Assignment 2 (a) and Assignment 3 (b).
Figure 4. Perceived programming skill improvement associated with Assignment 2 (a) and Assignment 3 (b).
Ijgi 14 00156 g004
Ijgi 14 00156 g005
Figure 5. Use of four help resources for the three assignments.
Figure 5. Use of four help resources for the three assignments.
Ijgi 14 00156 g005
Ijgi 14 00156 g006
Figure 6. Mean rated helpfulness of different resources for the three assessed assignments with standard error bars.
Figure 6. Mean rated helpfulness of different resources for the three assessed assignments with standard error bars.
Ijgi 14 00156 g006
Ijgi 14 00156 g007
Figure 7. Perceived chatbot code quality and Levenshtein distance between chatbot-generated code and finalized code submissions.
Figure 7. Perceived chatbot code quality and Levenshtein distance between chatbot-generated code and finalized code submissions.
Ijgi 14 00156 g007
Ijgi 14 00156 g008
Figure 8. Distribution of the normalized, weighted composite scores for sentences related to the usefulness of chatbot responses.
Figure 8. Distribution of the normalized, weighted composite scores for sentences related to the usefulness of chatbot responses.
Ijgi 14 00156 g008
Table 1. Existing research on the use of chatbots in educational settings.
Table 1. Existing research on the use of chatbots in educational settings.
Subject AreaStudy DesignFindingsReference
Programming
Skill retention in Python programming course2 × 2 (scaffold/no scaffold, text/voice)Conversational, scaffolding agent in lecture recordings facilitated retention of Python knowledge[13]
Student performance in multimedia programming course (HTML, CSS)Experimental vs. control student cohort Better course performance with chatbot assistant than with instructor[14]
Student performance in C++ programming courseExperimental vs. control student cohortStudent performance not influenced by ChatGPT usage[15]
Student performance in Java programming courseExperimental vs. control student cohortStudent performance not influenced by ChatGPT usage[28]
Student performance in Python programming courseExperimental vs. control student cohortStudent performance not influenced by ChatGPT usage[16]
Student attitudes in Java programming courseExperimental vs. control student cohortUsing ChatGPT in weekly programming practice enhanced student morale, programming confidence, and computational reasoning competence[10]
Performance and chatbot use in the course scientific computing for mechanical engineering (C and Python) Student questionnaires, semi-structured interviews with students and teacherStudents with prior experience used ChatGPT for more varied tasks; decline in code quality and student learning with increased ChatGPT usage[12]
Quality of ChatGPT generated Java code Student questionnairesChatGPT-generated code required little modification[17]
Readability of ChatGPT-generated Java codeAnalysis of ChatGPT code solutions for course programming tasksChatGPT-generated code was readable and well structured[18]
Code correctness of ChatGPT-generated Scala functional programming code Analysis of ChatGPT code solutions for course programming tasksInitial ChatGPT solutions were correct in 68% of cases; only 57% of correct solutions were legible[19]
GIS/Geography
Quality of ChatGPT-generated GIS lab assignmentAnalysis of generated lab instructionsPrompt refining led to more useful GIS assignment instructions[25]
Completeness of a ChatGPT-generated 2 h lecture program on climate and societyComparison of generated topics with existing curriculumChatGPT offered some useful suggestions to supplement existing lecture topics[26]
Table 2. VADER sentiment analysis scores.
Table 2. VADER sentiment analysis scores.
SentenceNegativeNeutralPositiveCompoundLabel
The code resulting from a chatbot can be easily improved according to the developer’s needs.00.7030.2970.6705Positive
Despite these concerns, using chatbots helped [to] develop technical skills, particularly in automating tasks in GIS with ArcPy.0100Neutral
Ultimately, students agreed that chatbots are not a perfect solution0.3530.4980.149−0.4329Negative
Table 3. Regression results for the prediction of quiz points.
Table 3. Regression results for the prediction of quiz points.
CoefficientsEstimateStd. Errortp
Intercept4.520.617.33<0.001 **
Python experience0.170.190.890.387
Chatbot introduction−2.220.95−2.330.033 *
Python × chatbot0.600.411.480.158
N20
R20.405
Adj. R20.293
** p < 0.001, * p < 0.05.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Hochmair, H.H. Use and Effectiveness of Chatbots as Support Tools in GIS Programming Course Assignments. ISPRS Int. J. Geo-Inf. 2025, 14, 156. https://doi.org/10.3390/ijgi14040156

AMA Style

Hochmair HH. Use and Effectiveness of Chatbots as Support Tools in GIS Programming Course Assignments. ISPRS International Journal of Geo-Information. 2025; 14(4):156. https://doi.org/10.3390/ijgi14040156

Chicago/Turabian Style

Hochmair, Hartwig H. 2025. "Use and Effectiveness of Chatbots as Support Tools in GIS Programming Course Assignments" ISPRS International Journal of Geo-Information 14, no. 4: 156. https://doi.org/10.3390/ijgi14040156

APA Style

Hochmair, H. H. (2025). Use and Effectiveness of Chatbots as Support Tools in GIS Programming Course Assignments. ISPRS International Journal of Geo-Information, 14(4), 156. https://doi.org/10.3390/ijgi14040156

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop