ChatGPT Can Grade your Essays. You just have to know how to ask.
As a former English teacher, I know firsthand the struggle of quickly grading stacks of end-of-term papers and finals. When I first started experimenting with ChatGPT I read several blogs and posts that showed it did a good job grading essays on a rubric. I recently had the opportunity to help a teacher try to develop a prompt to grade essays, and I can say it didn't live up to the hype. It was wildly unpredictable and unreliable. But I have found a few tips and tricks to significantly improve its ability to grade.
I must preface this by saying that anyone trying this must check all the results–it's not foolproof. Also, the best use of this might be for students to use it to get feedback before submitting their final drafts. These techniques could serve to create a personalized writing tutor. But I'm also all for improving efficiency. If this helps a few teachers get a Sunday back, I'm here for it!
Doing two things improved the ability to get valid scores and feedback:
Using a simple rubric
Adding a word count as a heuristic
Use a Simplified Rubric
The teacher whom I was helping had an extremely detailed rubric. There were six criteria with six levels each. The rubric created two issues. The first is also true with human graders–grading writing is subjective. Without calibration, it is unlikely that ChatGPT would have the same definition of "emerging" as the teacher, even with this detailed rubric.
The second problem created by the detailed rubric is that ChatGPT has a limited memory (4096 tokens, to be exact, where a token is ~¾ of a word). The prompt, the rubric, the student writing, and the response share this limit. This entire rubric was over 2500 tokens, leaving very little room for the student writing.
Simplifying our rubric fixes both of these problems and improves the output. Here is part of a prompt with a simple rubric for comparison.
For each criterion, provide a numerical score and a justification for your score. This justification should be specific and in natural language. It should be friendly and helpful, and understandable by a typical 8th grader. Speak as if you are talking directly to the student.
Give the total score at the end.
RUBRIC
Criterion 1: Thesis/Evidence (5 points)
Criterion 2: Organization (5 points)
Criterion 3: Voice (5 points)
Criterion 4: Word Choice (5 points)
Criterion 5: Sentence Fluency (5 points)
Criterion 6: Conventions (5 points)Each criterion is scored from 1-5.
0: Insufficient
1: Beginning
2: Emerging
3: Developing
4: Proficient|
5: Exemplary
Add to this some additional instructions, and students now get personalized feedback on each criterion.
Adding a word count as a heuristic
There is still one major issue–ChatGPT will score the writing with no regard for the requirements in terms of length. For example, the first paragraph of a 3-page paper will likely get the same word choice score as the entire essay. ChatGPT lacks the heuristics of a teacher scoring a paper–those mental shortcuts we all use to determine scores. Introducing just one of those–word count–improved the reliability.
When I first attempted this trick, I simply asked ChatGPT to count the words. What I didn't know at the time was that ChatGPT cannot count. No really, it can't. Give it 500 words and ask it for a word count. It is just as likely to say 300 as 1300. If you give it an accurate word count in the prompt and also prompt it to take that into account in the scores, it gets much more consistent. Do this by placing the following at the end of the student submission, "[word count= x]", where x is the accurate word count from your word processor.
See this in all in action here: example grading prompt
Try this in Grobots.ai
I’ve been using ChatGPT almost daily this semester with my students. Based on my experiences, I made the tool that makes ChatGPT do what I wanted it to do–including using word count. If you turn on word count as an option in Grobots.ai, it will first count the number of words submitted and then add that to the message to ChatGPT. You can create a link to have students submit their responses directly to the AI and then get a log of the chat. You can also just use it to submit the responses yourself. Here is a video showing how easy it is to set up.