Evaluating GPT’s Ability to Understand Syntactic Minimal Pairs in Korean

초록

This study investigates the syntactic knowledge of three GPT models GPT-3.5, GPT-4, and GPT-4o using a Korean dataset to assess whether their human-comparable performance extends to low-resource languages like Korean. Through Forced Choice, Yes/No, and Likert Scale tasks on Korean syntactic minimal pairs, we evaluated the models’ alignment with native Korean speaker judgments. Syntactic knowledge of GPT-4 and GPT-4o was largely comparable to that of native speakers, while GPT-3.5 was not. It is likely due to its lower sensitivity to low-resource languages. Qualitative analyses revealed strong performance in canonical structures but declines in complex and less frequent ones, suggesting limitations in training data. Our findings highlight the potential of GPT models for use in Korean linguistic research. The entire dataset and experimental code used in this study publicly available online.