5 Chapter 5: Getting More Detailed

As we emphasized in chapter 3, our goal as professors should be to help students understand that the process, and not the final product, is the point. We still need them to write about assigned reading, as a way of ensuring that they engage with it. The main thing to do in order to eliminate the need to police AI usage is to ask students to answer highly specific and detailed questions to which one cannot find ready-made answers online. If you have in the past required that students submit short overnight writes or blog posts about reading from a textbook, that assignment can still work if you add specific questions about details on specific pages. Such prompts have long been used by professors in many classes to deter students from being vague and superficial. While the advent of AI may alert you that this technique is essential, it has been necessary for some time. Online sources such as SparkNotes had already made it possible for students to find online summaries of books they are supposed to read. The use of such sources is related to the topic we explored earlier in the book, that students think the product rather than the process is the point.

The solutions we offer here are not merely addressing an immediate crisis caused by AI. They are solutions to longstanding challenges educators have faced in the form of students who use shortcuts to bypass doing assigned reading.^[1] As we emphasized earlier in the book, if the product is the course grade, it is both natural and rational to simply produce that. If a student realizes or becomes convinced that it is the process that is the goal, then you may no longer be fighting against natural instinct. What we want students to feel, deep in their bones, is that simply knowing what happens in a novel is different than experiencing the story first-hand. Reading conclusions from a work of nonfiction leaves out the reasoning behind how one arrived there. This is the joy and lifeblood of the humanities. This is why it matters.^[2] In this chapter, we provide details on the kinds of adjustments you might make to preserve these ideals, given that AI is here to stay.

Prompts for Short Writes about Assigned Reading

Let me begin with a concrete example related to a course that I teach and experiments I have conducted with ChatGPT.^[3] Let’s say that the following is one of the questions I ask students about the reading:

Why does Jaime Clark-Soles, on p.156 of her book Women in the Bible, say that references in Luke 8:19-21 can be a sensitive one?

The LLM made a decent guess based on the passage I provided from Clark-Soles but was wrong. Nonetheless, students might still think the kind of content it generated was relevant enough to the focus of the course and the book to earn them points. Now, imagine that instead I asked the following question:

On p.156 of her book Women in the Bible, what does Jaime Clark-Soles say is potentially sensitive, and why?

ChatGPT gave the following response, which I think is worth quoting:

I don’t have access to specific page numbers or passages of copyrighted books like Women in the Bible by Jaime Clark-Soles. However, if you provide context or a specific excerpt from the text, I can help analyze it or clarify its meaning. Alternatively, I can guide you on how to access this information through your local library, an online bookstore, or a scholarly database. Let me know how you’d like to proceed!

In its response to the first question, ChatGPT did not make explicit that which it did in its response to the second question—namely, that it does not have access to recent books that are under copyright. That may change, of course, but until it does, ChatGPT will only offer guesses. Its first answer (which I did not quote above) noted what Jaime Clark-Soles likely had in mind. Many students will fail to pick up on this difference, and the ones who don’t make any more effort than providing your question prompt and pasting ChatGPT’s response will thus end up including similar qualifiers. Students have no reason to talk about what an author was “likely” addressing if the answer to the question is explicit in the reading. Students who do so have not succeeded in the assignment and should not be given credit for that kind of attempt. You don’t need to determine whether they are doing the guessing or an LLM is. Just grade the assignment as submitted and move on.

Another trick you can use is to require quotes from the reading, which often makes it literally impossible for AI to produce a satisfactory response. An LLM can fabricate quotes, but it rarely has access to the specific content that you have assigned. You can make sure this is so, if you intentionally use copyrighted and paywalled material.^[4] Once again, this is something that addresses not only AI use but also the use of SparkNotes and other such sources. To interact with the exact wording of a novel, one simply must consult the novel. You can fail a student who includes fake quotes in their answer, once again without the need to worry whether this was a result of the student or an AI doing the fabricating. Both scenarios equally merit failure.

Reading Together

Every educator in the humanities will have noticed that students no longer read well, nor do they engage with assigned reading effectively. This would be true even if AI were to disappear tomorrow and be forever forgotten. The problem started before the appearance of generative AI and is not a result of it. However, there are ways of addressing this problem that simultaneously address the challenges posed by AI. There is a way to get students to engage with assigned reading more effectively, one that avoids the possibility of them using AI to write blog posts or other short writes which they then submit, pretending they have done the reading. The solution? Read together in class.

We know readers have likely conjured up images of children in kindergarten sitting in a sharing circle as a teacher reads them a book (ideally with pictures). That is not what we are envisioning. Instead, the template would be for students to do the first couple of readings together in class, annotating them and asking the instructor questions along the way. A grade would be assigned to engagement, whether through an assignment written outside of class afterwards or the very act of annotating and discussing in person. There are tools that make it possible for students to annotate a pdf text not only individually but collectively online in real time. You could also require that students take notes on the reading during class and turn them in at the end of the class period. Activities like these are designed to require students to attempt to read closely and critically. If you selected texts that are behind some sort of paywall, any secretive attempt by a student to engage an LLM will almost certainly not be able to generate an acceptable assignment.

Once students have become accustomed to reading in this way, you can then shift to having them do the reading on their own outside of class time. You can still expect them to annotate the pdf (whether individually or collectively as a class using one of the tools that makes this possible) or submit explanations, summaries, and questions about it prior to the next class meeting. In some courses it may make sense to do in-class reading periodically such as at the start of each new novel or subject unit. In foreign language classes, it may work to read together throughout the semester. Once again, these are activities that can be adapted to your specific needs.

Another assignment option that deserves serious consideration is having students read with an AI conversation partner and assistant. While a student left to their own devices or unguided might simply ask an AI to summarize a reading for them rather than reading it themselves, this approach necessitates reading the article. Without reading it, even if not thoroughly, a student will not be able to ask an LLM to explain unfamiliar concepts. Students can be required to submit a transcript of their interaction with an LLM about what they have read, asking it for clarifications of unfamiliar concepts and words, asking it for more information about something that is mentioned, and so on. This type of assignment increases the likelihood that students will do the reading and demonstrate that they have done so. It also makes it more likely that they will look up unfamiliar words and seek clarification of unfamiliar or difficult concepts. LLMs can sometimes do a good job when given an article from the past and asked to contextualize it historically, although as always the output needs to be verified. Integrating AI in this way thus supports traditional educational goals in the humanities, and provides a way of verifying that students have engaged in relevant activities in ways that were not previously possible.

Journaling an Essay

Early in my academic career, I participated in a workshop that required us to do a research project and keep a journal about it. I was skeptical of the value of the exercise. I imagined myself writing “I am writing in my journal when I should be writing my article” or perhaps “I am procrastinating” in entry after entry. In fact, the exercise was transformative and remained with me ever since. I had been under the (as it turned out mistaken) impression that I would procrastinate for extended periods, followed by a flurry of activity before a deadline arrived.^[5]

It turned out that I wasn’t procrastinating. I was reading and thinking, and all of that was what made possible the more visibly productive part of the research process. This is something that it is important for students to learn as well, making this a valuable exercise even apart from AI. If required to write and post daily journal entries about their research on a blog, for instance, some of them will demonstrate that they are indeed procrastinating, while others may have an epiphany. The worst student writing—including that generated at the last minute by AI—is often a result of students not realizing that the amount of time needed to create an essay by a deadline includes not only the writing but the finding, reading, and thinking processes. Require students to provide commentary on the writing process, how they searched for sources, how their thinking developed. It will give you access to the student learning process that you miss when all you see is the final product. This assignment also opens the door for you to share with them about your own research. You could even journal about whatever your current research project is and share that with them as an example.

What makes this a particularly useful assignment in the era of LLMs is that it will be difficult for a student to persuasively bluff. If an LLM generates your essay, trying to do daily blog posts that accurately match the stages a human author would have gone through to produce that final product will be challenging. For a student who has no idea what the actual research involved would entail, it may be literally impossible to do convincingly. They may try, to be sure, but you will be able to tell. Better still is to require journaling in conjunction with requiring students to show their work by having their saved drafts accessible to you, as described in an earlier chapter. The amount of time and effort that would be involved in creating an LLM-generated essay, adapting it into forms that look like successive earlier drafts, and then create journal entries that mimic what the student would have done if they had written the essay themselves, exceeds that required to do the work by a significant order of magnitude. Having them work in a document to which you have access (as outlined in the “Show Your Work” section of chapter 3) will also help prevent this. With such constraints in place that require their own human involvement, you might feel free to let them experiment with LLM usage for brainstorming, drafting, and copyediting. As long as you and your students understand the capabilities and limitations of this technology and agree on what constitutes ethical use thereof in the context of your course, you and they have a wide array of options. Assignments such as this one are useful in preventing LLM use if that is your aim, but also in ensuring appropriate use if that is permitted (or encouraged or required) in your course.

Oral Exams in the Video Age

Oral examinations or presentations are another approach that helps an educator establish what and how students studied and learned the relevant content and skills. Furthermore, depending on the level of interaction with the student, educators can really dial in on a student’s effort. Oral exams are much less common these days than they once were. In the context of the growing AI landscape, they are due to make a comeback, even if not in their traditional form. The biggest hurdle to using this extremely rigorous evaluation method (and related variants) is the massive time commitment on the part of the educator in examining and assessing each student individually. This tradeoff makes sense – the more personalized (and human) one gets, the more time and attention it takes to execute. However, the enhanced learning environment created through individual interaction between educator and student produces strong outcomes among students. Examinations are typically perceived by students as formal and impersonal. A more conversational form of assessment can potentially be more friendly while at the same time being more rigorous.

The ubiquitous nature of video technology allows an instructor to make some sensible changes to this traditional oral exam model that are more flexible and less cumbersome to implement. Students can be asked to provide verbal commentary and responses in video form in response to prompts provided by the instructor.^[6] Though this sacrifices the interactive nature of a true oral exam, it does allow students to practice communication in a different form, as well as allow the instructor to infer more about the preparation of a student. What is lost is an educator’s ability to guide a student to a correct answer with follow-up questions when they might be on the right track but haven’t expressed things clearly. (Or alternatively to determine whether a student can improvise beyond a rigidly prepared answer in advance.) These “video exams” are not a like-for-like replacement for an oral exam, but they may be an acceptable substitute. One could, depending on one’s enthusiasm for this approach, allow for or require students to submit a series of clarifying answers to leading questions from a professor’s written or video feedback.

The key to this kind of assignment, just as in the written examples we have discussed before, is to ask students detailed and focused enough questions to test whatever content and skills you are evaluating. An exam of this sort can be open or closed book, with questions provided ahead of time or in a limited time window that corresponds to an exam time. Students increasingly write the way they talk and so what you receive will in many cases be very similar to what you would have received in written form. Once the video submission has been received, you can leverage technology to streamline your evaluation process. Two common options are playing videos at a faster speed or using AI to automatically generate a written transcript of the recording. Each of these approaches moves the assignment and the educator further away from the traditional oral exam, which is not necessarily a bad thing! Instructors can decide where they feel most comfortable along that continuum; carefully done, students may often not even be aware that you have done so.

This video approach can be incorporated in different forms in other assignments as well. Video submissions that require students to explain their process in creating an essay, having them provide a written follow-up or response video after they receive the graded video assignment, are activities that do more than just reduce the likelihood of students submitting AI-generated work. These assignments encourage students to actually see the extensive feedback that educators often provide, rather than simply look at their grade. Requiring students to revisit an assignment and respond to feedback on it is extremely valuable. Once again, this feedback cycle would be worth doing even if AI were not a consideration. In some versions of the activity, you can not only ask each student to comment on their own work and respond to your comments, but you can also ask additional follow-up questions or have them respond to and comment on one another’s videos.

It used to be the case that oral examinations were time consuming and cumbersome both to give and to assess. Thanks to modern technology that is less the case. Even taking the time to have a Zoom session with each student may be worth the time spent. The additional time can be made by removing other assignments. There is evidence that students who engage more thoroughly and more thoughtfully with a smaller number of assignments can learn as much or even more as they might through a larger number of assignments with less active learning and personal interaction. If you are concerned, automated multiple-choice quizzes about broader course content can be used to supplement the essays and video oral exams. Another option that is implemented far less frequently than one would expect is the use of the flipped classroom model to incorporate things like oral exams or evaluated student discussions and debates into the regular class meetings. You won’t have time in most classes to have something like a traditional oral exam or thesis defense for every student. However, you can have graded speech, expecting students to present a paper and defend it, or come prepared to debate and then do so in a manner that is graded. It may also be an assignment for students who are not presenting on a given day to ask challenging questions of their peers who are presenting. All of these are forms of oral examination that can be incorporated within rather than outside of class meetings.

When it comes to oral examinations related to students’ own written work, these are especially interesting and rewarding types of assignments. They mirror what is expected of students who go on to do PhDs. Even when their own ideas are still in the process of development, they can be asked to defend their claims, prompted to draw conclusions if they failed to do so. You can ask them to clarify and justify their arguments that seem to go too far or not far enough. Forcing students to revisit and defend or critically evaluate their own earlier work is an extremely rewarding exercise that nothing else can substitute for. Whether through video, or having students speak during class in ways that are evaluated in the context of a flipped classroom, oral presentation is a good way to evaluate in the era of AI precisely because the question is not how did students study and familiarize themselves with content and skills, but how well have they mastered them so as to be able to proceed to higher levels of learning and achievement. Preventing inappropriate AI usage is only one reason to explore assessments using oral presentation and video. Oral exams are rigorous and push students to demonstrate mastery of higher order learning outcomes.

LLM as Discussion Partner

This book is not only about creating assignments in which AI usage is prevented, but also assignments in which students use AI in meaningful ways. By providing initial prompts or utilizing custom GPTs, an educator can have students use an LLM as a conversation partner. Like small group discussions, this practice is beneficial in the ways that all conversations can be when they require us to talk about and apply things we have been learning. As with online discussion forums, you can require students to have similar types of conversations with an AI. You should stipulate, just as you probably already do, the level at which their own engagement needs to be so as to demonstrate that they are not just going through the motions. You can also ask them to reflect on what they learned as a result of the exchange, as well as asking them to submit the discussion itself. Many have begun to refer to the potential of LLMs to serve as tutors. That term must be used with caution, not because it is fallible (after all, so are human tutors) but because a good human tutor will not confidently make things up when they do not know. LLMs do precisely that with some regularity.

Unless your students are very different from mine, their discussion of a topic in class may be, shall we say, hit or miss. Some topics may get vibrant conversations going. In other cases, it soon becomes clear that they have moved on to talking about something else entirely unrelated to the class. In the case of face-to-face conversations (including in breakout rooms on Zoom) there is no way to hear what each student is contributing in every group. Online discussion forums do better since the instructor can (eventually) read every comment. The quality of discussion online, however, varies as much as it does in person. A conversation with an LLM will, in a best-case scenario, force the conversation up to a higher level than it might otherwise occur at.^[7] Let’s be honest: even Wikipedia-level detail is better than some discussions we’ve heard among our students about course content. If no student in an exchange really grasps the content, then discussion among them may reinforce rather than challenge misconceptions. An LLM conversation partner will not do worse, although ultimately the effectiveness of an LLM when used in this way will vary depending on numerous factors including not only the topic but also what any individual student types in.

Some intellectuals have had impressively deep conversations with LLMs like Claude, which have led them to insights they might not have had otherwise.^[8] Note what is being said. The LLM did not have the insight. The human participant had the kind of insight that most or all of us have had when talking with someone. Their questions and responses to what we say lead us to think, formulate, and understand in new ways. You can have students submit their conversations, and then you evaluate the human part of them.

In addition to using discussion with an LLM as an end in itself, educators can also use this as a training ground for discussion with students. Students find certain types of speech in class difficult. They find it difficult not only to stand up and present in front of an audience of their peers, but also to talk about meaningful things in a way that makes room for substantive yet respectful disagreement. Often students have not yet learned that it is possible to disagree with someone’s ideas and yet for that not to be an attack on the person. An AI “argument clinic” will be more profitable than the one in the famous Monty Python sketch of the same name.

A variation on this is for students to get two separate LLMs to interact with one another. If the same LLM is used then it is important that they be two separate instances on two separate computers, so that the interaction with one does not impact the other. Require two groups of students to get each LLM to independently make the best case for and against the point under discussion. The results are copied and pasted into a Google Doc, allowing each side to generate counterarguments and critiques using the LLM. This can be framed or used as preparation for an actual in-class debate, with each side then evaluating the arguments and counterarguments offered by the LLMs and using them to prepare to make the best case they can in their interaction with the other group. Each group’s desire to win (however you incentivize it) will motivate them to look carefully for fake references, false information, and weak arguments in the text produced by the other side, as well as in their own lest it be found by the opposing team and used against them.

Students will at some point discover that, just because an LLM consistently produces grammatical prose on a topic, it may not be offering the most compelling arguments, since it imitates patterns of words and does not understand their meaning. For an added layer of engagement with and evaluation of LLMs, after the instructor and the class have each had an opportunity to evaluate who won the debate, a transcript of the debate can be given to two more separate instances of LLMs and asked to evaluate which side made the more persuasive case. An LLM, of course, cannot evaluate persuasiveness in any meaningful way, yet it will emulate the patterns of text that are found in such evaluations. This provides a good test case for getting students to reflect on whether mechanistic imitation of the product of reasoning can ever be a substitute for actual human reasoning.

In addition to practicing discussion skills with LLMs, research suggests that students who used generative AI at the brainstorming phase of a writing project found it helpful to do so.^[9] In this book we offer many suggestions on how to craft assignments that minimize the likelihood of inappropriate use of generative AI. Once that is accomplished, it opens the door to an exciting possibility, namely the appropriate use of generative AI. Brainstorming a topic and even getting suggestions for reading (the actual existence of which sources one verifies in seeking to gain access to them and read them) may constitute such appropriate use, so long as the actual research, the heart of the thinking and writing, is the work of the student.