The AI Who Taught Me

There has been a lot of drama and misinterpretation about a recent study preprint put out by MIT - Your Brain on ChatGPT. In this episode, we will look at that study in more detail, as well as a few others that have received less attention but I think do a better job of telling us how to use AI to teach writing.
We begin this episode by looking at a recent MIT pre-print, Your Brain on ChatGPT - preprint paper and the study site with FAQ and figures.
TIME Magazine covered this here, and then interviewed the study's main author, Dr. Nataliya Kosmyna - interview here. Fox News exaggerated the claims in their coverage of the story. It has been covered also by CNN and many other publications worldwide.
But the study doesn't actually make the claims that this breathless coverage suggests! Instead, we list three other studies that are more helpful at discerning how to use AI to teach writing:
Beware of Metacognitive Laziness.  https://arxiv.org/abs/2412.09315
In this experiment, some students used ChatGPT to write essays, while others did not. Those who used ChatGPT's help had better essays, but didn't have a significantly different level of learning or retention.
Modifying AI, Enhancing Essays. https://arxiv.org/pdf/2412.07200
A retrospective look at the CoAuthor dataset - a set of transcripts of students who used GPT-3 to help with their essays. This paper showed that students who engaged with feedback, rather than just accepting it blindly, had the most improvement.
Harnessing AI in Secondary Education to Enhance Writing Competence. https://arxiv.org/pdf/2412.12117v1
Not an experiment, but an overview and summary of other research.
"Research on feedback on writing shows that it is particularly valuable for students to receive help while they are engaged in the writing task. If students find themselves stuck or wonder how they have performed so far, feedback can be a great help. ... Unlike the human teacher, generative AI can provide instant feedback on a student’s writing (Jeon, 2023). This might include highlighting strengths, suggesting improvements, and pointing out areas where the student's voice shines through."
--
These three papers are available in the Stanford GenAI for Education Hub, which is an invaluable resource for finding quality research on AI in education.

Since the launch of ChatGPT in 2022, several AI education companies have sprung up to help teachers use these new tools in their classrooms. MagicSchool, founded by Adeel Khan, has been used by over 5 million teachers around the world. Meanwhile, Brisk Teaching has been growing fast and has over a million teachers using their extension as well.
These new platforms are the way that many teachers encounter and use AI in their classrooms. In this episode, we'll describe how they work, tell their story of how they began, and evaluate the evidence (or lack of evidence) on student learning.
 
Sources mentioned:
 
 
Tiers of Evidence
ESSA tiers of evidence (MagicSchool and Brisk both have earned the LOWEST tier, Tier 4, requiring just that there is a rationale behind their product but no evidence yet).
Common Sense privacy reviews for MagicSchool and Brisk are both 93%.
 
MagicSchool sources
 
Case studies from Aurora Public Schools(and the Aurora edtech page) and Innova. MagicSchool study from University of Jhang. Youtube short advertising rubric generator and Adeel Khan interview 
 
Brisk sources
Arman Jaffer interview on the Linkup Podcast and Brisk case studies.
 

Using LLMs can seem like a safe bet - but they have biases built in at every stage of the process. In this episode, we'll talk about how bias sneaks in and look at some studies that detail how LLMs are trained and how to watch out for and correct bias in an educational context.
Notes:
Follow up on the Tutor CoPilot study from the first week: https://lukeshepard.com/blog/preregistration-of-outcomes
Reserach about LLMs in classrooms: LLMs Are Biased Teachers
Meta's study on A Systematic Study of Bias Amplification
OpenAI discusses their own dataset: OpenAI: Is ChatGPT Biased?
I was influenced by Dr. Timnit Gebru- on AI Bias and the recently published The AI Con Book, which goes into more detail about how bias works in LLMs and the risks across industries.
 

There has been a lot of discussion about a recent article in NY Magazine, titled Everyone Is Cheating Their Way Through College. It contains many anecdotes about the nearly omnipresent use of ChatGPT and other AI tools on college campuses. Even two years ago, a survey found that nearly all students had used ChatGPT in some form, and the use has just expanded. "Generative-AI chatbots — ChatGPT but also Google’s Gemini, Anthropic’s Claude, Microsoft’s Copilot, and others — take their notes during class, devise their study guides and practice tests, summarize novels and textbooks, and brainstorm, outline, and draft their essays."
One of the students hit the nail on the head: “Most assignments in college are not relevant,” he told me. “They’re hackable by AI, and I just had no interest in doing them."
One teacher interviewed claims that "Massive numbers of students are going to emerge from university with degrees, and into the workforce, who are essentially illiterate. ... It’s short-circuiting the learning process, and it’s happening fast.”
In this episode, we'll examine whether this is true: is the use of AI in schools really short-circuiting the learning process? We look at two repositories of incredible research that help us answer that question.
First, we look at the meta-analysis published on May 6 in Nature: The effect of ChatGPT on students' learning performance, learning perception, and higher order thinking. The authors looked at 51 experimental studies from around the world, and they found "that ChatGPT has a large positive impact on improving learning performance and a moderately positive impact on enhancing learning perception and fostering higher-order thinking."
 
* In Taiwan, researchers saw strong learning when they used ChatGPT embedded in a game to help seventh graders learn science concepts.
* In Dubai, a set of eleventh graders were studying electromagnetism in their physics class. Those who used ChatGPT saw larger learning gains.
* In Australia researchers found that AI helped boost scores on student writing - but only when students actually made edits and engaged with the suggestions. When they just copy/pasted the text, their learning was much worse.
 
We also discuss the Stanford GenAI repository which serves as a great resource for finding up to date studies that analyze what's happening with AI in education.
 
See more on my blog:
https://lukeshepard.com/blog/chatgpt-in-schools

In this kickoff episode, host Luke Shepard and his AI co-host Emerald dive into how AI is entering classrooms. We will be looking at how AI is used in classrooms, starting with some recent research out of Harvard, Standford and Penn.
 
Studies
We look at four studies published in the last year:
Pew Research study on use of chatbots in school. About a quarter of US teens have used ChatGPT for schoolwork
A pre-print of the pilot in the Harvard Physics class in which students used a chatbot to help with studying. AI Tutoring Outperforms Active Learning
From University of Pennsylvania, How kids who used ChatGPT did better on practice but worse on the eventual tests. Kids who use ChatGPT as a study assistant do worse on tests
Stanford saw gains from a study of 1,800 students across multiple school districts in the use of a new AI tutor assistant. How AI can improve tutor effectiveness and the paper PDF
 
Further Listening:
 Listen to this recent interview between Jane Rosenzweig and Michael Horn about how AI is affecting writing.
 
Sal Khan's TED talk: How AI Could Save and Not Destroy Education