When given permission and guidance to use artificial intelligence tools in college writing classes, students rely primarily on the software for brainstorming and research rather than having them write massive essays. These findings, published in the Journal of Writing Research, suggest that students are selectively using computerized text generation tools to enhance the learning process. The study also revealed unexpected differences in how non-native English speakers use technology compared to other people.
The public availability of ChatGPT in late 2022 has sparked a heated debate about its place in higher education. Many educators were concerned that these tools would undermine original thinking and academic integrity. Instructors on the other side argued that generative artificial intelligence could act as a personal tutor, outlining ideas and supporting language learners. Researchers say these discussions often frame technology as a simple binary, assuming that the paper is either fully human or fully machine.
Because of this gap, instructors needed detailed data about how students were actively interacting with these programs behind the scenes. Sarah Madsen Hardy, an instructor in Boston University’s writing program, led a research team investigating this question. She and her colleagues wanted to observe how students instruct artificial intelligence systems and what type of machine-generated text they ultimately choose to include in their final academic papers. They sought to create an environment where students felt safe exploring these new technologies without fear of punishment.
Researchers designed an observational study of students enrolled in an introductory writing and research course at Boston University. The course section was specifically set up as a pilot to experiment with the integration of artificial intelligence. All students in these pilot sections received a subscription to ChatGPT Plus. Instructors guided students through exercises that focused on basic technical understanding and ethical implications of software, such as identifying bias and validating digital sources.
Instead of banning the tools, instructors allowed students to submit work containing up to 50 percent machine-generated text. To track software usage, instructors required students to highlight each word of machine-generated text in blue font in their submitted assignments. Students were not asked to emphasize fine grammatical adjustments, but only the substantive language directly produced by the chat interface.
After the semester ended, Madsen-Hardy and her team recruited 50 of these students to share their essays. A subset of the 34 participants also provided chat logs detailing their interactions with ChatGPT during the drafting process. Approximately one-quarter of study participants identified themselves as studying English as a foreign language. The research team specifically wanted to document whether these international students utilized text generators differently than native English speakers.
The team removed personally identifiable information from the collected essays and chat logs and performed a thematic content analysis. This process involves categorizing student texts into specific functional categories such as brainstorming, revision, and direct writing. To increase efficiency, the researchers utilized large-scale language models as additional raters alongside human experts. They measured agreement between human raters and the computerized system and found a high level of consistency in how chat logs were classified.
When the researchers analyzed these chat logs, they found that a small number of student requests were for the program to generate the original text for an assignment. Of the 290 prompts analyzed, only 18.6 percent asked ChatGPT to write the word from scratch. Most of the students’ interactions involved background work leading up to the writing stage.
Students most frequently used the chat feature to ask for help with revisions, such as shortening sentences or changing tone. This accounted for about a quarter of the total prompts. Another very common use was to ask programs to explain course material, define concepts, or clarify academic readings. When the researchers grouped the prompts, they found that students were far more likely to ask the software to provide advice, resources, and explanations than they were to ask the software to create text.
The chat logs also revealed a timeline of how students used the tools as the assignment progressed. Most students began the conversation by asking artificial intelligence for help with planning and locating information. The prompt asking the machine to compose a sentence typically occurs in the last quarter of a chat session. This indicates that direct text production occurred only after long conversations that addressed the traditional stages of the drafting process.
When the team looked at actual submitted papers, the data showed high levels of self-control among authors. More than half of the students in the pilot program chose not to include machine-generated verbatim text in their final draft. Across all 50 papers analyzed, only 8.2 percent of the total words submitted were blue-flagged to indicate artificial intelligence authorship. This usage was significantly less than the generous half price allowed by the instructor.
When students chose to paste text directly from ChatGPT into their reports, they rarely dropped entire paragraphs of blocks. Only about 6% of the blue text consisted of large paragraph chunks. Instead, students primarily wove small machine-generated phrases into their own original writing. The most common rhetorical purpose for incorporating this generated text was to support discussion, analysis, and synthesis of ideas.
The data revealed a striking contrast between students studying English as a foreign language and the native English speakers in their classes. The authors anticipated that non-native speakers might rely heavily on text generators to clarify confusing academic reading material. The analysis showed exactly the opposite pattern. Prompts for understanding or clarification were the least common requests submitted by foreign language learners.
Native English speakers asked text generators to clarify concepts seven times more often than other foreigners. Instead, foreign language students were about twice as likely to use a chat interface to ask for help revising existing prose. They were also more likely to initiate a chat session seeking direct feedback on their writing.
These interaction differences carried over into the final essay. In fact, foreign language-speaking students incorporated less machine-generated text into their final submissions than native speakers. Additionally, native speakers frequently used the program to create summaries of academic sources to include in their papers. International students rarely used this tool for summarization, choosing instead to handle these tasks without direct mechanical assistance.
The researchers noted that these patterns call into question the assumption that non-native speakers rely heavily on writing software to complete tasks. Because taking classes in a second language imposes an extra cognitive load, these writers primarily used the tool to hone their sentence structure. This study suggests that international students strategically utilize artificial intelligence resources to meet their specific language needs, rather than relying on artificial intelligence for basic understanding.
The authors emphasized that providing students with a structured environment to explore tools helps them make active decisions. By treating the software as a collaborative assistant, students retained ownership of their assignments. “Our findings indicate that students who receive continued instruction are able to engage strongly with the GAI and selectively incorporate its outcomes in ways that suggest both judgment of its effectiveness and investment in their own learning,” the authors write.
The researchers outlined several caveats for interpreting the data. Participation in post-semester data collection was voluntary, so the 50 students who volunteered their assignments may not be representative of the typical college freshman. Students who have struggled with courses or relied heavily on automatic transcription for illegal reasons may have decided not to share their data with researchers.
Another limitation arises from the requirement for students to self-report the generated text using blue font. The writer may have inconsistently highlighted or forgotten to label sentences inspired by the chat interface. Still, instructors met regularly with students to review drafts in progress, giving teachers confidence that the data reported reflected reality.
Future studies should include larger groups of participants to see if the observed language use patterns hold on a broader scale. Researchers can combine this content analysis with student interviews. By speaking directly with writers, you may find out exactly why non-native speakers are hesitant to use chatbots for reading comprehension.
Until then, current data provides an optimistic snapshot of modern college classrooms. When instructors model safe and effective ways to operate text generators, students appear to integrate the technology as a helpful assistant rather than a substitute for their own schoolwork.
The study, “Using Generative AI in College Writing Classes: An Analysis of Student Chat Logs and Writing Projects,” was authored by Sarah Madsen Hardy, Pary Fassihi, Shuang Geng, Christopher McVey, and Matt Parfitt.
