#5: Beyond the MIT ChatGPT Study: What The Research Tells Us About How To Use AI To Teach Writing

There has been a lot of drama and misinterpretation about a recent study preprint put out by MIT - Your Brain on ChatGPT. In this episode, we will look at that study in more detail, as well as a few others that have received less attention but I think do a better job of telling us how to use AI to teach writing.

We begin this episode by looking at a recent MIT pre-print, Your Brain on ChatGPT - preprint paper and the study site with FAQ and figures.

TIME Magazine covered this here, and then interviewed the study's main author, Dr. Nataliya Kosmyna - interview here. Fox News exaggerated the claims in their coverage of the story. It has been covered also by CNN and many other publications worldwide.

But the study doesn't actually make the claims that this breathless coverage suggests! Instead, we list three other studies that are more helpful at discerning how to use AI to teach writing:

Beware of Metacognitive Laziness.  https://arxiv.org/abs/2412.09315

In this experiment, some students used ChatGPT to write essays, while others did not. Those who used ChatGPT's help had better essays, but didn't have a significantly different level of learning or retention.

Modifying AI, Enhancing Essays. https://arxiv.org/pdf/2412.07200

A retrospective look at the CoAuthor dataset - a set of transcripts of students who used GPT-3 to help with their essays. This paper showed that students who engaged with feedback, rather than just accepting it blindly, had the most improvement.

Harnessing AI in Secondary Education to Enhance Writing Competence. https://arxiv.org/pdf/2412.12117v1

Not an experiment, but an overview and summary of other research.

"Research on feedback on writing shows that it is particularly valuable for students to receive help while they are engaged in the writing task. If students find themselves stuck or wonder how they have performed so far, feedback can be a great help. ... Unlike the human teacher, generative AI can provide instant feedback on a student’s writing (Jeon, 2023). This might include highlighting strengths, suggesting improvements, and pointing out areas where the student's voice shines through."

--

These three papers are available in the Stanford GenAI for Education Hub, which is an invaluable resource for finding quality research on AI in education.