Rubric-Based Evaluation of Tool-Augmented LLMs on Curriculum-Aligned Textbook Projects

초록

Tool-augmented ('deep-research') large language models (LLMs) are increasingly integrated into educational technology for inquiry and project-based learning, yet educators still lack curriculum-aligned analytics that distinguish ordinary response fluency from process-competency evidence and isolate the contribution of tool augmentation. We evaluate two deep-research systems (GPT-DR and Gemini-DR) and matched general-mode baselines (GPT-BL and Gemini-BL) on 73 authentic secondary textbook project tasks drawn from seven Korean high-school textbooks. Using an exact fixed prompt ('Please solve this problem.'), we collected three independent responses per task for each model (876 total responses). A 17-indicator rubric spanning problem solving, reasoning, communication, connections, and information processing was used as the analytic lens; two human coders independently coded the 438 baseline-control responses to establish rubric reliability. Task-clustered analyses showed that deep-research augmentation substantially increased mean response score within both vendors. On the deep-research condition, Gemini-DR covered 70.9% of indicator opportunities and GPT-DR 54.7%; Gemini-DR showed higher coverage in problem solving, reasoning, communication, and connections, while the information-processing dimension was similar. Multi-run analysis also showed greater stability for Gemini-DR. These coverage rates should be interpreted as visible process-evidence coverage rather than as scores of mathematical correctness, response depth, or instructional quality. We discuss how rubric-guided prompting, repeated generation, and teacher-facing verification pipelines can support accountable classroom use of LLM-generated project work. © 2026 The Authors.

키워드