Integrating chatbots in education: insights from the Chatbot-Human Interaction Satisfaction Model CHISM Full Text

Such a strategy was used to ensure that the instructor could guide the students the next day if there were any issues. Qualitative data were collected through class discussions and assessment reports of the AICS following a template provided through the Moodle platform. During the 1-month intervention period in each educational setting, participants independently completed the assessment reports. They were instructed to provide personal feedback on their interaction with each AIC, using the template to note both positive and negative aspects. Additionally, they were asked to attach 12 screenshots illustrating their interaction, three with each AIC, to support their assessment. QDA Miner Software was used for textual analysis of students’ written evaluations on each AIC, adhering to a provided template.

Such a streamlined approach will assist learning centers in reducing manual efforts required for materials update, thereby fostering convenient resource utilization. By leveraging this valuable feedback, teachers can continuously improve their teaching methods, ensuring that students grasp concepts effectively and ultimately succeed in their academic pursuits. Users should be aware of potential biases in the training data that chatbots are based on and take measures to mitigate the amplification of biases in the generated content. Universities must establish clear guidelines and policies to ensure that students use AI tools appropriately and give proper credit to original sources. In the assisting role (Assisting), chatbot actions can be summarized as simplifying the student’s everyday life, i.e., taking tasks off the student’s hands in whole or in part.

In this study, we carefully look at the interaction style in terms of who is in control of the conversation, i.e., the chatbot or the user. Hobert and Meyer von Wolff (2019), Pérez et al. (2020), and Hwang and Chang (2021) examined the evaluation methods used to assess the effectiveness of educational chatbots. The authors identified that several evaluation methods such as surveys, experiments, and evaluation studies measure acceptance, motivation, and usability.

These AI-driven tools create an inclusive studying environment by catering to diverse educational styles and abilities. They offer adaptable content formats, such as audio, visual, and text-based materials, ensuring accessibility for all users, regardless of their needs. In 2023, AI chatbots are transforming the education industry with their versatile applications.

Drawing from extensive systematic literature reviews, as summarized in Table 1, AI chatbots possess the potential to profoundly influence diverse aspects of education. However, it is essential to address concerns regarding the irrational use of technology and the challenges that education systems encounter while striving to harness its capacity and make the best use of it. Three categories of research gaps were identified from empirical findings (i) learning outcomes, (ii) design issues, and (iii) assessment and testing issues. EC studies have primarily focused on language learning, programming, and health courses, implying that EC application and the investigation of learning outcomes have not been investigated in various educational domains and levels of education.

Furthermore, a designer should also consider chatbot’s capabilities for natural language conversation and how it can aid instructors, especially in repetitive and low cognitive level tasks such as answering FAQs (Garcia Brustenga et al., 2018). As mentioned previously, the goal can be purely administrative (Chocarro et al., 2021) or pedagogical (Sandoval, 2018). This study aims to investigate the effects of ECs for an Instructional Design course that applies team-based project towards learning outcomes, namely learning performance, perception of learning, need for cognition, motivation, creative self-efficacy, and teamwork. Learning performance is defined as the students’ combined scores accumulated from the project-based learning activities in this study. Next, perception of the learning process is described as perceived benefits obtained from the course (Wei & Chou, 2020) and the need for cognition as an individual’s tendency to participate and take pleasure in cognitive activities (de Holanda Coelho et al., 2020).

Exceptionally, a chatbot found in (D’mello & Graesser, 2013) is both a teaching and motivational agent. In general, most desktop-based chatbots were built in or before 2013, probably because desktop-based systems are cumbersome to modern users as they must be downloaded and installed, need frequent updates, and are dependent on operating systems. Unsurprisingly, most chatbots were web-based, probably because the web-based applications are operating system independent, do not require downloading, installing, or updating.

App-Integrated Chatbots (AICs) in language learning

The data that support the findings of this study are available from the corresponding author upon reasonable request. Georgia State University has effectively implemented a personalized communication system. They introduced Pounce, a bespoke smart assistant created to actively engage admitted students.

Chatbots can provide virtual tutoring and mentoring services, guiding students through coursework, assignments, and career advice. They can supplement the support offered by faculty members and academic advisors. Two recent articles in the journal Nature described its application to weather forecasting. Currently, it is difficult and time-consuming because to make predictions, meteorologists must analyze weather variables such as temperature, precipitation, pressure, wind, humidity, and cloudiness individually, but new AI systems can significantly speed up the process.

Begin by telling the chatbot that you would like to develop a fictional short story and that you’d like its assistance in developing your ideas. Try different ways of interacting and responding to the chatbot to get a sense of its capabilities. ChatGPT, developed by OpenAI, uses the Generative Pre-training Transformer (GPT) large language model.

Chatbots are digital systems that can be interacted with entirely through natural language via text or voice interfaces. They are intended to automate conversations by simulating a human conversation partner and can be integrated into software, such as online platforms, digital assistants, or be interfaced through messaging services. There is also a bias towards empirically evaluated articles https://chat.openai.com/ as we only selected articles that have an empirical evaluation, such as experiments, evaluation studies, etc. Further, we only analyzed the most recent articles when many articles discussed the same concept by the same researchers. Since different researchers with diverse research experience participated in this study, article classification may have been somewhat inaccurate.

When you think of advancements in technology, edtech might not be the first thing that pops into your head. But during the COVID-19 pandemic, edtech became a true lifeline for education by making it accessible and easy to use despite there being numerous physical restrictions. Today, technologies like conversational AI and natural language processing (NLP) continue to help educators and students world over teach and learn better.

Teacher candidates’ perceptions of app-integrated chatbots

Such a contribution also offers networking opportunities and support for current students. Additionally, this will positively impact the brand image, attracting potential applicants and stakeholders. Overloaded due to tight scheduling and plenty of daily duties, educators often face challenges. Invaluable teaching assistants can give a hand with automation tasks like tests, assessments, and assignment tracking. EdWeek reports that, according to Impact Research, nearly 50% of teachers utilized ChatGPT for lesson planning and generated creative ideas for their classes. Education reaches far beyond the classroom, requiring guidance and support across the entire campus life.

It engaged in text-based conversations and demonstrated the ability to exhibit delusional behavior, offering insights into natural language processing and AI. Later in 2001 ActiveBuddy, Inc. developed the chatbot SmarterChild that operated on instant messaging platforms such as AOL Instant Messenger and MSN Messenger (Hoffer et al., 2001). SmarterChild was a chatbot that could carry on conversations with users about a variety of topics. It was also able to learn from its interactions with users, which made it more and more sophisticated over time. In 2011 Apple introduced Siri as a voice-activated personal assistant for its iPhone (Aron, 2011).

Considering this, the University of Murcia in Spain used an AI chat assistant that successfully addressed more than 38,708 inquiries with an accuracy rate of 91%. Educational chatbots serve as personal assistants, offering individual guidance to everyone. Through intelligent tutoring systems, these models analyze responses, learning patterns, and overall performance, fostering tailored teaching. Bots are particularly beneficial for neurodivergent people, as they address individual comprehension disabilities and adapt study plans accordingly. AI systems may lack the emotional understanding and sensitivity required for dealing with complex sentimental concerns. In educational establishments where mental support is essential, the absence of sensitive intelligence in chatbots can limit their effectiveness in addressing users’ personal needs.

After coding a larger set of publications, it became clear that the code for service-oriented chatbots needed to be further distinguished. This was because it summarized e.g. automation activities with activities related to self-regulated learning and thus could not be distinguished sharply enough from the learning role. After refining the code set in the next iteration into a learning role, an assistance role, and a mentoring role, it was then possible to ensure the separation of the individual codes. Research in this area has recently focused on chatbot technology, a subtype of dialog systems, as several technological platforms have matured and led to applications in various domains. You can foun additiona information about ai customer service and artificial intelligence and NLP. Chatbots incorporate generic language models extracted from large parts of the Internet and enable feedback by limiting themselves to text or voice interfaces. For this reason, they have also been proposed and researched for a variety of applications in education (Winkler and Soellner, 2018).

As data sources, Scopus, Web of Science, Google Scholar, Microsoft Academics, and the educational research database “Fachportal Pädagogik” (including ERIC) were selected, all of which incorporate all major publishers and journals. In (Martín-Martín et al., 2018) it was shown that for the social sciences only 29.8% and for engineering and computer science, 46.8% of relevant literature is included in all of the first three databases. For the topic of chatbots in education, a value between these two numbers can be assumed, which is why an approach of integrating several publisher-independent databases was employed here.

Each has some unique characteristics and nuanced differences in how developers built and trained them, though these differences are not significant for our purposes as educators. We encourage you to try accessing these chatbots as you explore their capabilities. The authors declare that this research paper did not receive any funding from external organizations.

What Are Educational Chatbots All About?

These AI-driven programs, tailored for educational settings, aim to provide enriched learning experiences. It’s incredible, but chatbots have been used in education since the early 1970s. A chatbot in the education industry is an AI-powered virtual assistant designed to interact with students, teachers, and other stakeholders in the educational ecosystem. Using advanced Conversational AI and Generative AI technologies, chatbots can engage in natural language conversations, providing personalized support and delivering relevant information on various educational topics. Chatbots in education offer unparalleled accessibility, functioning as reliable virtual assistants that remain accessible around the clock.

Therefore, looking at our results and the challenges presented, we conclude, “No, we are not there yet! ” – There is still much to be done in terms of research on chatbots in education. Still, development in this area seems to have just begun to gain momentum and we expect to see new insights in the coming years. The teaching agents presented in the different studies used various approaches. For instance, some teaching agents recommended tutorials to students based upon learning styles (Redondo-Hernández & Pérez-Marín, 2011), students’ historical learning (Coronado et al., 2018), and pattern matching (Ondáš et al., 2019).

• Modern chatbots are trained to conduct very complex tasks, yet they can be easily built without coding.
• Chatbots in the education sector can help collect feedback from all the stakeholders after each conversation or completion of every process.
• Although we have focused on ensuring that codes used contribute to a strong understanding, the determination of the abstraction level might have affected the level of detail of the resulting data representation.
• For example, while Buddy.ai is oriented towards developing oral skills in children at a lower level, John Bot and Andy are designed for vocabulary and grammar building through role-playing interactions at more intermediate levels.
• Today chatbots can understand natural language, respond to user input, and provide feedback in the form of text or audio (text-based and voice-enabled).

Chatbots deployed through MIM applications are simplistic bots known as messenger bots (Schmulian & Coetzee, 2019). These platforms, such as Facebook, WhatsApp, and Telegram, have largely introduced chatbots to facilitate automatic around-the-clock interaction and communication, primarily focusing on the service industries. Even though MIM applications were not intended for pedagogical use, but due to affordance and their undemanding role in facilitating communication, they have established themselves as a learning platform (Kumar et al., 2020; Pereira et al., 2019).

They can offer learners the possibility to engage in simulated conversational interactions in a non-judgmental environment (El Shazly, 2021; Skjuve et al., 2021). For these reasons, chatbots are being increasingly used as virtual tutors to facilitate the development of language skills and communicative competence in the target language (Huang et al., 2022; Hwang & Chang, 2021; Zhang et al., 2023). Navigating the expansive world of educational chatbots reveals a realm where technology meets academia, fostering student engagement, and offering support.

The study was conducted independently and without financial support from any source. The authors have no financial interests or affiliations that could have influenced the design, execution, analysis, or reporting of the research. It’s important to note that some papers raise concerns about excessive reliance on AI-generated information, potentially leading to a negative impact on student’s critical thinking and problem-solving skills (Kasneci et al., 2023). For instance, if students consistently receive solutions or information effortlessly through AI assistance, they might not engage deeply in understanding the topic.

The value was determined by looking at the search results in detail using several queries to exclude as few relevant works as possible. This approach showed promising results and, at the same time, did not burden the literature list with irrelevant items. Assignment of coded research topics identified in related literature reviews to research categories.

Subsequently, the assessment of specific topics is presented where the user is expected to fill out values, and the chatbot responds with feedback. The level of the assessment becomes more challenging as the student makes progress. A slightly different interaction is explained in (Winkler et al., 2020), where the chatbot challenges the students with a question. If they answer incorrectly, they are explained why the answer is incorrect and then get asked a scaffolding question. The proliferation of smartphones in the late 2000s led to the integration of educational chatbots into mobile applications.

Most importantly, chatbots played a critical role in the education field, in which most researchers (12 articles; 33.33%) developed chatbots used to teach computer science topics (Fig. 4). Chatbots, also known as conversational agents, enable the interaction of humans with computers through natural language, by applying the technology of natural language processing (NLP) (Bradeško & Mladenić, 2012). In fact, the size of the chatbot market worldwide is expected to be 1.23 billion dollars in 2025 (Kaczorowska-Spychalska, 2019). In the US alone, the chatbot industry was valued at 113 million US dollars and is expected to reach 994.5 million US dollars in 2024 Footnote 1. I do not see chatbots as a replacement for the teacher, but as one more tool in their toolbox, or a new medium that can be used to design learning experiences in a way that extends the capacity and unique abilities of the teacher. In addition, the responses of the learner not only determine the chatbot’s responses, but provide data for the teacher to get to know the learner better.

Meanwhile, North Korea, China, and Russia, in particular, contended that the U.S. might employ ChatGPT for disseminating misinformation. Conversely, OpenAI restricts access to ChatGPT in certain countries, such as Afghanistan and Iran, citing geopolitical constraints, legal considerations, data protection regulations, and internet accessibility as the basis for this decision. Italy became the first Western country to ban ChatGPT (Browne, 2023) after the country’s data protection authority called on OpenAI to stop processing Italian residents’ data. They claimed that ChatGPT did not comply with the European General Data Protection Regulation. However, after OpenAI clarified the data privacy issues with Italian data protection authority, ChatGPT returned to Italy.

4 RQ4 – What are the interaction styles supported by the educational chatbots?

This can be achieved by making information more easily available (Sugondo and Bahana, 2019) or by simplifying processes through the chatbot’s automation (Suwannatee and Suwanyangyuen, 2019). An example of this is the chatbot in (Sandoval, 2018) that answers general questions about a course, such as an exam date or office hours. Most articles (13; 36.11%) used an experiment to establish the validity of the used approach, while 10 articles (27.77%) used an evaluation study to validate the usefulness and usability of their approach. The remaining articles used a questionnaire (10; 27.7%) and a focus group (3; 8.22%) as their evaluation methods.

Chatbots can facilitate online discussions, group projects, and collaborative learning experiences, allowing students to engage with peers and share ideas, fostering community and active participation. Chatbots’ ease of use and ability to rapidly create human-like text, including everything from reports, essays, and recipes to computer code, ensure that the AI revolution will be a powerful tool for students at every level to improve their capabilities and expertise. But, like most powerful technologies, the use of chatbots offers challenges as well as opportunities. By grouping the resulting relevant publications according to their date of publication, it is apparent that chatbots in education are currently in a phase of increased attention. The release distribution shows slightly lower publication numbers in the current than in the previous year (Figure 6), which could be attributed to a time lag between the actual publication of manuscripts and their dissemination in databases.

The aim was not to compare the four AICs, but rather to present teacher candidates with a broad overview of these virtual tutors, providing a variety of options and examples. Some studies have emphasized that interactions with AICs can seem detached and lack the human element (Rapp et al., 2021). Additionally, while AICs can handle a wide range of queries, they may struggle with complex language nuances, which could potentially lead to misunderstandings or incorrect language usage. It has also been observed that some students’ interest dwindled after the initial period of engagement due to repetitive conversation patterns and redundancies, making the interaction less natural compared to student–teacher exchanges (Fryer et al., 2019). AI chatbots for education offer backup throughout university life, from the admission process to post-course assistance. They act beyond classroom activities as campus guides, providing valuable information on facilities and helping students.

Nevertheless, because the tool did not produce answers to some questions, some students decided to abandon it and instead use standard search engines to find answers. This paper will help to better understand how educational chatbots can be effectively utilized to enhance education and address the specific needs and challenges of students and educators. As a digital assistant, the EC was designed to aid in managing the team-based project where it was intended to communicate with students to inquire about challenges and provide support and guidance in completing their tasks. According to Cunningham-Nelson et al. (2019), such a role improves academic performance as students prioritize such needs. Therefore, supporting the outcome of this study that observed that the EC groups learning performance and teamwork outcome had a more significant effect size than the CT group.

Making up a quarter of all publications, Efficiency of Education is the second most popular objective (25%), while addressing Students’ Motivation and Availability of Education are third (13%) and fourth (11%), respectively. Other objectives also make up a substantial amount of these publications (19%), although they were too diverse to categorize in a uniform way. Examples of these are inclusivity (Heo and Lee, 2019) or the promotion of student teacher interactions (Mendoza Chat PG et al., 2020). Applying the curve presented in Figure 6 to Gartner’s Hype Cycle (Linden and Fenn, 2003) suggests that technology around chatbots in education may currently be in the “Innovation Trigger” phase. This phase is where many expectations are placed on the technology, but the practical in-depth experience is still largely lacking. At last, we could have missed articles that report an educational chatbot that could not be found in the selected search databases.

Roughly 92% of students worldwide demonstrate a desire for personalized assistance and updates concerning their academic advancement. By analyzing pupils’ learning patterns, these tools customize content and training paths. Such a unique approach ensures that everyone receives tailored support, promoting better comprehension and knowledge retention. Although chatbots can provide information, they should not act as a substitute for, instead of spurring the development of students’ critical thinking and analytical skills. Universities need to emphasize the importance of independent research, critical evaluation, and synthesis of knowledge.

With artificial intelligence, the complete process of enrollment and admissions can be smoother and more streamlined. Administrators can take up other complex, time-consuming tasks that need human attention. From teachers to syllabus, admissions to hygiene, schools can collect information on all the aspects and become champions in their sector. Users should provide feedback to OpenAI, Google, and other relevant creators and stakeholders regarding any concerns or issues they encounter while using chatbots. Reporting any instances of misuse or ethical violations will help to improve the system and its guidelines.

The comprehensive list of included studies, along with relevant data extracted from these studies, is available from the corresponding author upon request. The datasets generated and/or analysed during the current study are not publicly available due privacy reasons but are available from the corresponding author on reasonable request. The American Council on Science and Health is a research and education organization operating under Section 501(c)(3) of the Internal Revenue Code.

Secondly, we study selected articles and synthesize results and lastly, we report and discuss the findings. Subsequently, we delve into the methodology, encompassing aspects such as research questions, the search process, inclusion and exclusion criteria, as well as the data extraction strategy. Moving on, we present a comprehensive analysis of the results in the subsequent section. Finally, we conclude by addressing the limitations encountered during the study and offering insights into potential future research directions.

SPACE10 (IKEA’s research and design lab) published a fascinating survey asking people what characteristics they would like to see in a virtual AI assistant. Beyond gender and form of the bot, the survey revealed many open questions in the growing field of human-robot interaction (HRI). In the images below you can see two sections of the flowchart of one of my chatbots. In the first one you can see that the chatbot is asking the person how they are feeling, and responding differently according to their answer.

Only one study pointed to high usefulness and subjective satisfaction (Lee et al., 2020), while the others reported low to moderate subjective satisfaction (Table 13). For instance, the chatbot presented in (Lee et al., 2020) aims to increase learning effectiveness by allowing students to ask questions related to the course materials. It turned out that most of the participants agreed that the chatbot is a valuable educational tool that facilitates real-time problem solving and provides a quick recap on course material. The study mentioned in (Mendez et al., 2020) conducted two focus groups to evaluate the efficacy of chatbot used for academic advising. While students were largely satisfied with the answers given by the chatbot, they thought it lacked personalization and the human touch of real academic advisors. Finally, the chatbot discussed by (Verleger & Pembridge, 2018) was built upon a Q&A database related to a programming course.

Examples of these are chatbots simulating a virtual pen pal abroad (Na-Young, 2019). Conversations with this kind of chatbot aim to motivate the students to look up vocabulary, check their grammar, and gain confidence in the foreign language. To understand and underline the current need for research in the use of chatbots in education, we first examined the existing literature, focusing on comprehensive literature reviews. By looking at research questions in these literature reviews, we identified 21 different research topics and extracted findings accordingly.

This allows the teacher to tweak the chatbot’s design to improve the experience. Equally if not more importantly, it can reveal gaps in knowledge or flawed assumptions the learners hold, which can inform the design of new learning experiences — chatbot-mediated or not. Tutoring, which focuses on skill-building in small groups or one-on-one settings, can benefit learning (Kraft, Schueler, Loeb, & Robinson, 2021). Effective tutors may use questioning techniques, collaborative problem-solving, and personalized instruction to support their students. While Stanford provides a range of tutoring services, not all students use them regularly; students might use AI chatbots as a supplement to tutoring services.

One of the takeaways is that the emerging field around educational chatbots has seen much activity in the last two years. Based on the experience of this preliminary search, search terms, queries, and filters were constructed for the actual structured literature review. This structured literature review follows the PRISMA framework (Liberati et al., 2009), a guideline for reporting systematic reviews and meta-analyses. The framework consists of an elaborated structure for systematic literature reviews and sets requirements for reporting information about the review process (see section 3.2 to 3.4).

The goal was to analyse and identify the main benefits and drawbacks of each AIC as perceived by teacher candidates. These themes were cross-referenced with the different components of the CHISM model to establish correlations as shown in Table 7. Frequency in the table refers to the number of observations made in the sample of textual data based on the written assessments provided by participants. Regarding the frequency of use of the four AICs employed in the intervention, the post-survey results shown in Table 3 indicated that Andy was the most frequently used, averaging nearly 4 h per week, followed by John Bot and Mondly, while Buddy.ai was the least used. Concerning the educational setting, Spanish participants interacted more frequently with all four AICs compared to Czech students.

Looking ahead, allowing students to select specific design aspects of AICs, similar to choosing linguistic features such as target level or accent, could be a crucial step in creating a more adaptive and personalized learning experience. The CHISM results, particularly in the Language Experience (LEX) dimension, revealed significant insights about the teacher candidates’ perceptions of the four evaluated chatbots. When examining why none chatbot in education of the AICs achieved moderate satisfaction in the LEX dimension, it is crucial to consider each AIC’s design and target audience limitations, as pointed out in previous research (Gokturk, 2017; Hajizadeh, 2023). For instance, Mondly’s reliance on pre-programmed responses and Buddy.ai’s focus on repetitive drills for children limit dynamic conversation, resulting in lower satisfaction in maintaining contextually relevant dialogues.