Manual versus AI-Assisted Test Design in Language Assessment: Insights from Pre-service English Teachers
6 th International Conference on Educational Technology and Online Learning – ICETOL 2026, Bremen, Almanya, 17 - 20 Ağustos 2026, ss.156, (Özet Bildiri)
- Yayın Türü: Bildiri / Özet Bildiri
- Basıldığı Şehir: Bremen
- Basıldığı Ülke: Almanya
- Sayfa Sayıları: ss.156
- Erzincan Binali Yıldırım Üniversitesi Adresli: Evet
Özet
Reshaping the roles of educators through the power of language education, Artificial Intelligence (AI) is increasingly praised for its abilities in exam preparation, including designing test items, generating practice tests, and providing instant, personalised feedback. In addition to language assessment literacy, language educators need AI literacy to effectively incorporate AI-generated outputs into their critical evaluation, and teacher training programmes should prioritise developing this literacy to support AI-assisted language assessment. With a focus on the professional development of pre-service English teachers, this study compared their experiences and perceptions of manual versus AI-generated language assessment practices. Adopting a qualitative case study design, the study included 61 pre-service English teachers. All participants were first asked to design a language test based on the content provided in class, with freedom to choose item techniques and quantity. After completing their manual test design, they used a GenAI tool (ChatGPT) to produce a test on the same content, applying the effective prompt writing strategies they had been trained on. Following the submission of both their manually designed and AI-generated tests, they reflected on their experiences and perceptions of the test development process comparatively. Their tests and reflections were analysed thematically. The findings revealed that pre-service English teachers mostly preferred structured-response or extensive response items rather than multiple-choice items in their manually designed tests since they had difficulty writing distractors; however, their AI-generated tests included more variety and quantity in terms of items. Furthermore, pre-service English teachers mostly found the AIgenerated language test design process more practical, time-efficient, and effective in ensuring content validity but also emphasised their roles in checking the appropriateness of the items, particularly for the levels of students and the objectives of the course. They also pointed out that they needed more practice in assessment design as well as theoretical aspects of language assessment to have assessment literacy and digital literacies, particularly AI literacy, for writing effective prompts to obtain expected outcomes. Suggesting a balanced approach in assessment design where AI does not replace but supports the teacher through its capabilities in assessment design, this study emphasises the development of AI literacy in teacher training to promote the ethical and effective use of these systems in language education.