\n<\/p>\n
AI-simulated students consistently outperform real students\u2014and make different kinds of mistakes\u2014in math and reading comprehension, according to a new study.<\/p>\n
That could cause problems for teachers, who increasingly use general prompt-based artificial intelligence platforms to save time on daily instructional tasks. Sixty percent of K-12 teachers report using AI in the classroom, according to a June Gallup study, with more than 1 in 4 regularly using the tools to generate quizzes and more than 1 in 5 using AI for tutoring programs. Even when prompted to cater to students of a particular grade or ability level, the findings suggest underlying large language models may create inaccurate portrayals of how real students think and learn.<\/p>\n
\u201cWe were interested in finding out whether we can actually trust the models when we try to simulate any specific types of students. What we are showing is that the answer is in many cases, no,\u201d said Ekaterina Kochmar, co-author of the study and an assistant professor of natural-language processing at the Mohamed bin Zayed University of Artificial Intelligence in the United Arab Emirates, the first university dedicated entirely to AI research. <\/p>\n
Kochmar and her colleagues prompted 11 large language models (LLMs), including those underlying generative AI platforms like ChatGPT, Qwen, and SocraticLM, to answer 249 mathematics and 240 reading grade-level questions on the National Assessment of Educational Progress in reading and math using the persona of typical students in grades 4, 8, and 12. The researchers then compared the models\u2019 answers to NAEP\u2019s database of real student answers to the same questions to measure how closely AI-simulated students\u2019 answers mirrored those of actual student performance.<\/p>\n
The LLMs that underlie AI tools do not think but generate the most likely next word in a given context based on massive pools of training data, which might include real test items, state standards, and transcripts of lessons. By and large, Kochmar said, the models are trained to favor correct answers.<\/p>\n
\u201cIn any context, for any task, [LLMs] are actually much more strongly primed to answer it correctly,\u201d Kochmar said. \u201cThat\u2019s why it\u2019s very difficult to force them to answer anything incorrectly. And we\u2019re asking them to not only answer incorrectly but fall in a particular pattern\u2014and then it becomes even harder.\u201d<\/p>\n
For example, while a student might miss a math problem because he misunderstood the order of operations, an LLM would have to be specifically prompted to misuse the order of operations.<\/p>\n
None of the tested LLMs created simulated students that aligned with real students\u2019 math and reading performance in 4th, 8th, or 12th grades. Without specific grade-level prompts, the proxy students performed significantly higher than real students in both math and reading\u2014scoring, for example, 33 percentile points to 40 percentile points higher than the average real student in reading. <\/p>\n
Kochmar also found that simulated students \u201cfail in different ways than humans.\u201d While specifying specific grades in prompts did make simulated students perform more like real students with regard to how many answers they got correct, they did not necessarily follow patterns related to particular human misconceptions, such as order of operations in math. <\/p>\n
The researchers found no prompt that fully aligned simulated and real student answers across different grades and models.<\/p>\n
For educators, the findings highlight both the potential and the pitfalls of relying on AI-simulated students, underscoring the need for careful use and professional judgment.<\/p>\n
\u201cWhen you think about what a model knows, these models have probably read every book about pedagogy, but that doesn\u2019t mean that they know how to make choices about how to teach,\u201d said Robbie Torney, the senior director of AI programs at Common Sense Media, which studies children and technology. <\/p>\n
Torney was not connected to the current study, but last month released a study of AI-based teaching assistants that similarly found alignment problems. AI models produce answers based on their training data, not professional expertise, he said. \u201cThat might not be bad per se, but it might also not be a good fit for your learners, for your curriculum, and it might not be a good fit for the type of conceptual knowledge that you\u2019re trying to develop.\u201d<\/p>\n
This doesn\u2019t mean teachers shouldn\u2019t use general prompt-based AI to develop tools or tests for their classes, the researchers said, but that educators need to prompt AI carefully and use their own professional judgement when deciding if AI outputs match their students\u2019 needs.<\/p>\n
\u201cThe great advantage of the current technologies is that it is relatively easy to use, so anyone can access [them],\u201d Kochmar said. \u201cIt\u2019s just at this point, I would not trust the models out of the box to mimic students\u2019 actual ability to solve tasks at a specific level.\u201d<\/p>\n
Torney said educators need more training to understand not just the basics of how to use AI tools but their underlying infrastructure. \u201cTo be able to optimize use of these tools, it\u2019s really important for educators to recognize what they don\u2019t have, so that they can provide some of those things to the models and use their professional judgement.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"
AI-simulated students consistently outperform real students\u2014and make different kinds of mistakes\u2014in math and reading comprehension, according to a new study. That could cause problems for teachers, who increasingly use general prompt-based artificial intelligence platforms to save time on daily instructional tasks. Sixty percent of K-12 teachers report using AI in the classroom, according to a<\/p>\n","protected":false},"author":1,"featured_media":20467,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[57],"tags":[1978,9090,455,12574,12575],"class_list":{"0":"post-20466","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-education","8":"tag-match","9":"tag-matters","10":"tag-real","11":"tag-simulations","12":"tag-studentsand"},"_links":{"self":[{"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=\/wp\/v2\/posts\/20466","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=20466"}],"version-history":[{"count":0,"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=\/wp\/v2\/posts\/20466\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=\/wp\/v2\/media\/20467"}],"wp:attachment":[{"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=20466"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=20466"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=20466"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}