Evaluation of pre-trained models for pedagogical assessment of novel AI-assisted educational questions
The surge in AI-assisted generation of educational materials has outpaced our capacity to validate their pedagogical quality. Automated evaluation using Bloom Classifier models is a promising approach to assess educational materials at scale. These models show high accuracy within-distribution dataset (IID Dataset). However, applying the same models to new out-of-distribution (OOD) datasets such as AI-assisted generated questions could show performance degradation. To identify robust classifiers under dataset shift, we evaluated traditional Machine Learning (ML), transformer, and Large Language models on the Bloom level classification task. We also explored feature-engineering strategies incorporating NLP metrics, appending the learning objectives as part of the input, and text splicing to
评估预训练模型在教学评估新型 AI 辅助教育问题上的表现。
- 核心方法
- 使用 Bloom 分类器模型对传统机器学习、转换器和大型语言模型进行评估，探索包括 NLP 指标、附加学习目标和文本拼接在内的特征工程技术。
- 适合谁读
- 教育技术研究者、AI 研究人员、教育工作者
- 要解决的问题
- AI 辅助生成的教育材料激增，但对其教学质量的验证能力不足。
- 关键实验
- 对传统 ML、变换器和大语言模型在 Bloom 水平分类任务上进行评估，测试了不同特征工程策略的效果。
- 主要贡献
- 识别了在数据集变化下仍能保持性能的鲁棒分类器，为大规模评估教育材料提供了方法。
- 意义与局限
- 该研究为教育材料的质量评估提供了新的自动化手段，但需进一步验证在真实应用场景中的效果。