Yes, ChatGPT can answer exam questions. But can it write them too?

There have many stories published since ChatGPT came out last November about the potential of college students using the AI to write essays, answer exam questions, and otherwise skirt the educational honor system.

But what about the converse - can we as instructors use ChatGPT to write exam questions?

After a decade of teaching introductory biology at the college level, I find it more difficult with each passing semester to come up with new ideas for quiz questions each week. So after playing with ChatGPT with my kids and asking it to write stories and songs about hamsters, Fortnite, and John Cena, I decided to ask it if it can write some biology questions for me. The following is what I found out (TLDR: yes, ChatGPT can write some pretty darn good biology questions).

First, I wanted to know if it knew what Bloom's Taxonomy is, as this is an important construct for writing questions that assess different cognitive levels.

OK check, ChatGPT is knowledgeable of Bloom's. Notice that it used specific tenses of the six Bloom's levels (remembering, understanding, applying, analyzing, evaluating, and creating) so I used those specific verbs when asking it to write questions for me. Next I picked a topic that I would hope it was familiar with (Darwin's theory of natural selection) and asked it write me a Bloom's remembering level multiple choice question.

Goodness gracious, not only can ChatGPT write a multiple choice question, but it tells me the correct answer, and I would definitely classify this as a remembering level question. Moving on to the level of understanding.

Alright, that is a decent understanding question. But now let's really challenge ChatGPT. Applying and analyzing questions usually require students to transfer their knowledge to new situations or scenarios. Let's see how it did.

I almost fell out of my chair when I read these. Not only did ChatGPT up the game and write true application and analysis questions, but it gave novel scenarios for students to assess without being prompted to do so!

Now to the highest two levels, evaluating and creating. Putting aside the argument for a moment that some make that it isn't possible to write these levels of multiple choice questions, let's see how ChatGPT fared.

Well my oh my. ChatGPT definitely worked at higher levels and asked questions that assess students' abilities to evaluate different explanations and to come up with new experimental designs to test a hypothesis. I'm not a fan of "all of the above" options, but I won't nit pick too much now.

So while ChatGPT can write some pretty good multiple choice questions of all six Bloom's levels, it cannot write questions that incorporate figures or drawings, data tables, or graphs from scientific publications, all of which I use to further assess my students' scientific analysis abilities. It also has a hard time writing complex, open-ended, numerical-based problems, of which I use in my chemical engineering courses (but it seems to do a pretty good job at writing straightforward numerical problems, think plug and chug). But text-only multiple choice questions are very commonly used on quizzes and exams in STEM courses (including by me), so now the question has to be asked - should we as instructors use ChatGPT to help us write assessments?

I don't know if I have a strong feeling about this yet, but it sure is tempting to think about. Is using ChatGPT to write questions any different than borrowing old questions from a colleague, using the test bank of questions that comes with your textbook, or googling for questions that are posted online? In all of these cases, you obtain a question that is ready to use, and you may use it verbatim, but you may just as likely edit it to fit your style. So perhaps that is the way to go with ChatGPT - ask it to write you a question and use its response as a foundation to which you edit and make the question more of your own. Or maybe we don't give in to the AI and keep our questions 100% human-generated.

What do you think? Is using ChatGPT to write exam questions fair, ethical, or justifiable? Should you tell your students if you do choose to use ChatGPT to write questions? Please comment below or email me and share your thoughts, would love to hear what you think!

Justin Shaffer is a teaching associate professor in chemical and biological engineering and in quantitative biosciences and engineering at the Colorado School of Mines and is the the founder of Recombinant Education. This article may or may not have been written by ChatGPT.