In "Introduction to Agricultural Policy" (D-USYS), we tested a ChatGPT-4.0-based tutor for exam preparation in the spring semester of 2024. The knowledge base of the AI tutor was an open-source textbook on Swiss agricultural policy, which we supplemented with custom instructions for the bot. As our analysis of students' interactions showed, the tutor was used primarily for concept clarification, explanations, summaries, and quizzes. The accuracy of the tutor was relatively high (96%), but it did provide incorrect responses occasionally, which is why it was important that students had been sensitized to potential problems before working with the tutor. Overall, students rated the tutor at 3.8 out of 5 and acknowledged its potential as a supplementary tool for learning.
Implementation of the Project/Course
Interviewer: Why did you decide to experiment with an AI tutor in the first place?
Robert: I saw that students were struggling to process all the study material for the exam. As I’ve heard about the potential of AI as a learning aid, I reached out to the Innovedum project team at D-USYS (Dr. Melanie Paschke, Dr. Manuel Sudau, and Réka), who were looking for volunteer lecturers to experiment with AI in their courses. My goal was to assess students’ AI-assisted learning experience and gauge the accuracy of the tool on subject-specific questions.
Réka: When Robert reached out, I saw it as an opportunity to develop know-how for creating individualized learning paths with AI assistance. As there were no clear blueprints for the educational implementation of an AI tutor, this course provided us with the opportunity to experiment with one. Since the study material for the exam was an open-access book, we could set up the tutor without having to worry about data protection. (In 2024, only ChatGPT offered a technical solution for the kind of implementation we had in mind, and data protection would have been problematic on that platform if the text had been behind a paywall.)
Interviewer: Did students use the tutor the way you envisioned it?
Robert: Not quite. We designed a guided learning experience for students, but students used the tutor mostly for concept clarification, summarizing, and, occasionally, for mock exams. Apparently, they wanted a supplementary tool rather than a new, more interactive approach to learning. Nevertheless, students were satisfied with the tutor: their average satisfaction with the tool was 3.8 out of 5, and they also found it very motivating to learn with it (4.07 out of 5). Whether learning preferences will change with the availability of new options remains to be seen.
Interviewer: How has the technology changed since you first implemented the project?
Réka: A problem with the tutor was that it would obtain information from other sources than the coursebook, which is why Notebook LM may work better at present. In Notebook LM, only the sources you share with the tool are used. This platform also helps the students fact-check the outputs by linking all statements to the source material. Even though our tutor “only” hallucinated 4% of the time (e.g., confusing similar-sounding terms such as ‘decoupling’ and ‘decoupled payments’), this verification step can further help students mitigate the risk of learning inaccurate information.
ETH Competence Framework
-
Method-specific
Competencies -
Personal
Competencies
- Media and digital technologies: students receive training on how to use large language models effectively and responsibly and then gain practical experience with the tools
- Critical thinking: students are encouraged to stay vigilant and continuously monitor the accuracy of AI-generated answers
- Self-awareness and self-reflection: students were asked to reflect on their experiences with the AI tutor, which helped them identify their learning preferences and strategies
