What the most detailed study of peer-reviewed AI in the classroom teaches us
The outstanding capabilities of widely available LLMs have sparked a heated debate in the education sector. On the one hand, they provide a 24/7 tutor for the student who can always help; however, of course, students can use LLM to cheat! My students and I have seen both sides of the coin. Yes, even the bad side, even at the college level.
Despite extensive discussion of the potential benefits and problems of LLM in education, there is an urgent need for important, empirical evidence to guide the integration of these technologies in classrooms, curriculums and overall research. Going beyond anecdotal accounts and rather limited research, this is a work titled “The Impact of Chatgpt on Students’ Learning Performance, Learning Perception, and Higher-Stage Thinking: Insights of Meta-Analyticals” that provides one of the most comprehensive quantitative assessments to date. Articles by Jin Wang and Wenxiang fans of the Institute of Chinese Education Modernization at Hangzhou Normal University were published in Magazine this month Humanities and Social Sciences Communication From the Natural Publishing Group. Its details are complex, so here I will dig into the results reported therein, also on methodology, and on the impact on people who develop and deploy AI in an educational environment.
Enter: Quantitative impact of chatgpt on student learning
Wang and Fan’s study is a meta-analysis that combines data from 51 research papers published between November 2022 and February 2025, examining the impact of Chatgpt on three critical student outcomes: learning performance, learning perception, and advanced thinking. For AI practitioners and data scientists, this meta-analysis provides a valuable, evidence-based lens through which current LLM capabilities can be evaluated and inform the future development of educational technology.
The main research questions attempt to determine the overall effectiveness of Chatgpt in three key educational outcomes. The meta-analysis produced statistically significant and noteworthy results:
Regarding learning performance, data from 44 studies suggest a huge positive impact attributed to CHATGPT use. In fact, it turns out that on average, students who incorporated CHATGPT into their learning process showed significantly improved academic outcomes compared to the control group.
To learn perceptions, covering students’ attitudes, motivations, and participation, an analysis of 19 studies revealed moderate but significant positive effects. This means that despite the prior limitations and questions related to the tools students can use to cheat, Chatgpt can promote a more favorable learning experience from the student’s perspective.
Similarly, the impact on advanced thinking skills such as critical analysis, problem solving, and creativity is also moderately positive, according to nine studies. The good news is that Chatgpt can support the development of these key cognitive abilities, although its impact is obviously not as obvious as direct learning performance.
How different factors affect chatgpt learning
In addition to overall efficacy, Wang and Fan also examined how various research characteristics affect the impact of Chatgpt on learning. Let me summarize the core results for you.
First of all, this type has a strong effect. The greatest effect was observed in courses involving skills and ability development, followed by STEM (Science/Technology) and related disciplines, followed by language learning/academic writing.
The course’s learning model also plays a crucial role in regulating how many assisted students. Problem-based learning makes Chatgpt particularly powerful enhancement, and has produced great results. Personalized learning environments also show great results, while project-based learning shows smaller but still positive results.
The duration of ChatGPT usage is also an important regulator of Chatgpt’s impact on learning performance. A short duration of one week has produced a small effect, while extended use within 4-8 weeks has the greatest impact and will not grow more if used further expansion. This suggests that sustained interactions and familiarity are essential for fostering positive affective responses to LLM assisted learning.
Interestingly, in any analytical study, the student’s grades, the specific role Chatgpt plays in the activity, and the application areas, did not significantly affect learning performance.
Other factors, including grade level, type, learning model, specific roles adopted by Chatgpt, and application areas, did not significantly mitigate the impact on learning perception.
The study further shows that Chatgpt’s impact on cultivating higher-order thinking is most obvious when it acts as an intelligent mentor, providing personalized guidance and feedback.
Impact on the development of AI-based educational technology
The findings of Wang & Fan’s meta-analysis have significant implications for the design, development, and strategic deployment of AI in an educational environment:
First, strategic scaffolding about deeper cognition. The impact on thinking skills development is lower than performance, which means that LLM is not inherently deep farmer, even if they do have a positive global impact on learning. Therefore, AI-based educational tools should integrate clear scaffolding mechanisms to facilitate the development of thinking processes to guide students in advanced analytics, synthesis and evaluations parallel to the direct help of AI systems.
Therefore, the implementation of AI tools in education must be properly constructed, and as we saw above, the framework will depend on the exact type and content of the course, the learning model you wish to apply, and how long it is available. A particularly interesting setup is where AI tools support queries, assumptions testing and collaboratively solve problems. Note that the findings of optimal duration suggest the need for introductory strategies and adaptive engagement techniques to maximize impact and mitigate potential overdependence.
Excellent influence is documented in the direction of AI in education when Chatgpt acts as an intelligent tutor. It is crucial to develop LLM-based systems that can provide adaptive feedback, constitute diagnostic and reflective problems, and guide learners through complex cognitive tasks. This requires moving beyond simple Q&A capabilities toward more complex dialogue AI and teaching reasoning.
Most importantly, there are still some non-small problems to be solved. Although LLM performs well in information delivery and task assistance (which leads to high performance improvements), enhancing its impact on emotional domains (perception) and advanced cognitive skills requires better interactive design. Critical considerations, combined with elements that develop student agents, provide meaningful feedback and effectively manage cognitive burdens.
Limitations and where to go for future research
The authors of the study carefully acknowledge some limitations, which also elucidates the avenues for future research. Although the total sample size is the largest ever, it is still small and very small for some specific issues. More research is needed and new meta-analysis may be required when more data is available. A difficult point, which is my personal addition, is that as the technology progresses so quickly, unfortunately, the results can be very quickly obsolete.
Another limitation in the research analyzed in this article is that they are largely biased towards university-level students, while data on primary education are very limited.
Wang and Fan also discuss AI, data science, and pedagogy that should be considered in future research. First, they should try to break down the effects based on a specific LLM version, which is critical because they develop so quickly. Second, they should study how students and teachers usually “cure” LLM and then examine the impact of differences cues on the final learning outcomes. They then need to develop and evaluate the adaptive scaffolding mechanism embedded in LLM-based educational tools in some way. Finally, in the long run, we need to explore the impact of LLM integration on the development of knowledge retention and self-regulating learning skills.
Personally, I say at this point, I think research needs to dig more about how students use LLM cheats, not necessarily willing, but may be by looking for shortcuts that lead them to mistakes or allow them to get out of the situation but without really learning anything. In this case, I think AI scientists lack in developing masquerading systems to detect text generated by AI, which they can use to quickly and confidently determine, for example, whether an assignment is completed using LLM. Yes, there are some watermarks and similar systems out there (I’ll cover them one day!), but I don’t seem to deploy them in a way that educators can easily take advantage of.
Conclusion: Towards the integration of evidence for AI in education
The meta-analysis I present for you here provides you with the key, data-driven contributions to the discussion of educational AI. It confirms the great potential of LLM, especially in CHATGPT in these studies, to improve student learning performance and positively influence learning perception and higher-order thinking. However, the study also strongly illustrates that the effectiveness of these tools is not uniform, but is significantly regulated by contextual factors and their nature incorporating them into the learning process.
For the AI and data science community, these findings are both affirmative and challenging. It is definitely due to the effectiveness of LLM technology. The challenge lies in harnessing this potential through thoughtful, evidence-based design designs that go beyond universal applications toward refined, adaptable and teaching-making educational tools. The path forward requires continued commitment to rigorous research and a nuanced understanding of the complex interactions between AI, pedagogy and human learning.
refer to
Wang Hefan:
The impact of CHATGPT on students’ learning performance, learning perception, and higher-order thinking: Meta-analytical insights. Jin Wang & Wenxiang Fans Humanities and Social Sciences Communication Volume 12, 621 (2025)
If you like this,.
