In today’s column, I will showcase an important prompting technique that can demonstrably improve your results when using generative AI such as ChatGPT. The technique is commonly known as the checklist prompt. In brief, you tell generative AI to produce a checklist for whatever question or problem you want the AI to solve. Doing so aids the AI in undertaking a more structured approach to the matter.
A secondary plus is that you have the AI verify via the derived checklist that all the questions or parts of the problem were indeed solved or at least considered by the AI during the solving process. This is akin to someone using a checklist to make sure that nothing was inadvertently left by the wayside. We’ll make that the job of the AI.
Anyone using generative AI on a weekly or daily basis ought to be familiar with and always ready to employ the checklist prompt.
Let’s talk about it.
This analysis of an innovative proposition is part of my ongoing Forbes.com column coverage on the latest in AI including identifying and explaining various impactful AI complexities (see the link here).
The Immense Value Of Checklists
Before we leap into the generative AI aspects, I’d like to cover the use of checklists in a conventional human-related context.
I dare say that most people would readily agree that checklists are great. If you are going to the store to buy groceries, a checklist is a handy means of ensuring you don’t forget to purchase those fresh eggs or newly baked bread. While at work, you undoubtedly make checklists too. A checklist at the office might remind you of work tasks that need to be accomplished.
Checklists, checklists, checklists.
They are all around us.
There are a few conventional rules that go along with checklists, especially checklists that are reasonably well put together. First, the checklist ought to be relevant to the matter at hand. Putting your work tasks on your grocery checklist would seem confusing. You can do so if you wish, but we usually prepare checklists for specific purposes.
Another facet of checklists is that you typically mark the items on the checklist once they are completed or for whatever reason you believe they are no longer pertinent. You might mark the checklist as you go along. Sometimes you mark the checklist after you are all done, or nearly done, and want to double-check that you got everything that was supposed to take place.
Overall, a checklist provides a helpful indication of what you need to do or be thinking about. The checklist provides an additional benefit as a verification tool. You can keep track of what has been done and what still needs to be done. If you think you are completely done, it is simple to relook at the checklist and verify that all the tasks were performed.
The good news is that checklists not only are helpful for humans but they can be utilized successfully by AI. Yes, that’s right, even AI can be better off via the old-fashioned longstanding value of leveraging checklists.
Checklists Come To The Aid Of Generative AI
Let’s now shift into generative AI mode.
First, some background about prompts and generative AI. When you use generative AI, you enter a prompt. The prompt is what will shape the response that you will get from generative AI. If you give a confusing prompt, the odds are that the AI will not be able to answer your query or give you a confusing response. There are all kinds of prompting tactics and strategies that can be employed to get the most out of using generative AI.
For my coverage of over thirty-five specific and vital prompt engineering techniques, see the link here.
I’d like to introduce you to a prompting technique known as the checklist prompt. It goes like this. When you compose your prompt, make sure to tell the AI that you want it to craft a checklist. The checklist created by the AI needs to be based on the nature of your prompt and the questions being asked in the prompt. Each prompt will get its own particular checklist.
Furthermore, you want to emphasize to the AI that the purpose of the checklist is to serve as an evaluation mechanism. If a checklist is just a checklist, it might not do much good. The AI will need to utilize the checklist to verify that the generated answers cover what was asked.
Here is what I usually indicate when I want to initiate the checklist prompting technique:
- My preferred checklist initiating prompt: “Upon my entering a prompt, you are to make an evaluation checklist that is specific to that prompt and do so before answering the question. Then, after coming up with your answer, you are to independently use the checklist to verify that each aspect of my question has been answered by you. The checklist must consist solely of Yes or No items only. The checklist should be relevant and sensible, so be careful what you include in it. Be succinct but also comprehensive. Make sure to show me the completed and confirmed checklist at the end of the answer to the question posed in the prompt.”
That is a bit lengthy, admittedly. The reason it is somewhat long is that I prefer to be relatively detailed and make sure the AI gets the nitty-gritty of what I want to undertake. You can make those instructions shorter, or longer, based on your checklist preferences. Any kind of wording along those lines should do the trick. You are welcome to use that exact passage, but you can also compose something of your own preference if that is desired.
I typically use that prompt right before I anticipate wanting to have the AI start making use of checklists. Once you’ve entered that above prompt, the AI is going to assume that during the rest of the conversation you want checklists. It will be automatic at that juncture.
If you decide during an active conversation that you don’t want a checklist for your next prompt, make sure to say so in that next prompt. In that sense, you can turn on or turn off the checklist facet that you’ve activated. By default, it will continue throughout the conversation.
When you start a brand-new conversation, the checklist prompting instruction will no longer be applicable, unless you’ve opted to use generative AI in a manner that carries conversations across each other (to learn how this is done, see my coverage at the link here). It all depends on which generative AI app you are using and the parameters that you have set up for that AI.
Checklist Prompting Is Easy To Invoke
Let’s look at the checklist prompt in action.
I logged into ChatGPT to do this. You can use just about any generative AI app such as ChatGPT, GPT-4o, o1, Claude, Gemini, or Llama, and the chances are that checklist prompt will be useful there too. It works well for nearly all the major generative AI and large language model designs.
First, let’s tell AI that we want to use a checklist prompting approach. Make sure to clarify what you have in mind. If you are vague about the nature of the checklist you’d like to have, the AI might veer into oddball territory. Be specific. Be direct.
Here we go with my preferred initiation prompt for checklist usage:
- My entered prompt: “Upon my entering a prompt, you are to make an evaluation checklist that is specific to that prompt and do so before answering the question. Then, after coming up with your answer, you are to independently use the checklist to verify that each aspect of my question has been answered by you. The checklist must consist solely of Yes or No items only. The checklist should be relevant and sensible, so be careful what you include in it. Be succinct but also comprehensive. Make sure to show me the completed and confirmed checklist at the end of the answer to the question posed in the prompt. Do you understand these instructions?”
- ChatGPT generated response: “Yes, I understand. Here’s a summary of your instructions to ensure I follow them accurately: (1) Create a Checklist: Upon receiving a prompt, I’ll create a specific Yes/No checklist that covers all aspects of the prompt. (2) Answer the Question: I’ll address each element of the checklist in my response. (3) Verify with the Checklist: After writing my answer, I’ll go through the checklist independently to verify that every item has been answered. (4) Show Completed Checklist: I’ll display the completed checklist with my response to confirm that each aspect of the prompt was addressed.”
I’d like you to observe a few noteworthy aspects.
I tried to lay out what I meant by providing a checklist. For example, I noted that the checklist should be of a Yes or No variety. Without explicitly making that indication, the AI might come up with some other means of assessing the verification, such as using a numeric score or possibly an unusual wording scheme.
You usually do not have to mention that the AI should show you the checklist since that’s pretty much assumed by the AI. On the other hand, my preference is to always be as specific as I can. The more you allow generative AI to wander, the further it might float away.
I also liked that ChatGPT confirmed my instructions by telling them back to me. That was great. A lack of echoing my instructions might imply that the AI didn’t really grasp what I had in mind. For the importance of using echo prompts, see my explanation at the link here.
Example Of Checklist Prompt Doing Its Thing
Now that we’ve got ChatGPT primed to proceed with a checklist approach, we can try and see how it goes.
I have some pressing questions about Abraham Lincoln that I’ve always wanted to know about. Let’s ask those questions. Meanwhile, it is the case that the checklist prompting instruction has been placed into action and we should see suitable results accordingly.
Take a look.
- My entered prompt: “I want to know some facts about Abraham Lincoln. Is it true that he worked as a bartender? As a child, did he say that someday he might be president? While serving as president, would he ever take time off or was he always working? Please let me know, thanks.”
- ChatGPT generated response:
“Here are the answers to your questions.”
1. “Lincoln as a Bartender: Yes, it’s true that Lincoln worked as a bartender. In 1833, he co-owned a general store and tavern in New Salem, Illinois, with a partner named William Berry.”
2. “Lincoln’s Childhood Ambition to Be President: There’s no evidence Lincoln said he wanted to be president as a child. His early years were spent in poverty on the frontier, with little formal education, and he was largely self-taught.”
3. “Did Lincoln Take Time Off as President? Despite the intense demands of the presidency during wartime, Lincoln did occasionally take brief respites. He would spend time at the Soldiers’ Home, a rural retreat near Washington, D.C., to rest and recover from the stresses of leadership.”
“Completed Checklist:”
- “Did I confirm if Abraham Lincoln worked as a bartender? Yes”
- “Did I address whether Lincoln claimed as a child that he might be president? Yes”
- “Did I clarify whether Lincoln took any time off while serving as president? Yes”
End of ChatGPT response
After all those years of mulling over those pressing questions about Lincoln, I now have answers.
Note that the answers were followed by a checklist. The checklist abided by my instruction to say whether each item was answered and do so by indicating Yes or No.
One thing that seems almost off-putting is that if you were to briskly glance at the Yes or No about whether each question was answered, you might falsely think the Yes or No is related to the question itself. In other words, I saw the word “Yes” about whether the AI addressed the question inquiring if Lincoln had during childhood said he might someday be president, and I thought this was yes that he did. That isn’t the case. The Yes is that the AI answered the question. The answer to that question was that Lincoln did not do so.
You can either just mentally train yourself to realize that the Yes or No means whether the item was addressed, or you could tell AI to reword the checklist response so that the Yes or No is more subtly stated and aligned correspondingly. Another possibility would be to have the AI use a checkmark rather than the words Yes or No.
Shifting From Checklist Format To Essay Format
Once you’ve placed generative AI into a checklist framework, the answers generated will often be shown in a numbered checklist format. That might not be what you wanted. You are perhaps okay with the checklist being numbered, but you might want the answer itself to be portrayed as a narrative or essay.
Easy-peasy to solve.
Just tell the AI what you want.
- My entered prompt: “Please convert the answers about Lincoln into a brief narrative.”
- ChatGPT generated response: “One lesser-known chapter of Lincoln’s life was his brief time as a bartender. In 1833, before his political career took off, Lincoln co-owned a general store and tavern in New Salem, Illinois, with a partner named William Berry. As a child, there is no evidence that Lincoln dreamed of becoming president. His early years were marked by poverty, hard work, and a patchwork of education. Once he ascended to the presidency, Lincoln was known for his relentless dedication to the job, particularly given the Civil War’s demands. Even so, he recognized the need to step away from time to time to recharge. Lincoln would often retreat to the Soldiers’ Home, a quiet place just outside Washington, D.C., where he could find some respite from the pressures of wartime leadership.”
There, I now have a short essay that seems smoother looking and sweeter on the eyes.
I should mention something else that is a standing concern that you should always be thinking about. You are never to believe generative AI at face value. There is always a solid chance that the AI has said something wrong, misinterpreted something, or gone so far as producing a so-called AI hallucination (see my explanation at the link here).
Does the invoking of a checklist potentially aid in boosting the odds that the response will be better or less likely to be off-kilter?
Maybe, but you still should double-check, possibly triple-check any AI responses. Period, end of story.
Research On The Esteemed Checklist Prompt
We can turn to AI research about the checklist prompting technique to gauge how it does on an empirical basis. In the research study entitled “TICKing All The Boxes: Generated Checklists Improve LLM Evaluation And Generation” by Jonathan Cook, Tim Rocktaschel, Jakob Foerster, Dennis Aumiller, and Alex Wang, arXiv, October 4, 2024, here are key salient points (excerpts):
- “We rigorously show that LLMs can generate evaluation checklists similar in quality to those written by trained human annotators across multiple diverse instruction-following datasets.”
- “We introduce TICK, a checklist-based, automatic evaluation protocol that yields stronger agreement with humans than other general-purpose LLM-as-judge evaluations. Because TICK can be fully automated, it is cheaper and faster to run than existing checklist-based evaluations and can be applied to arbitrary instruction-following datasets.”
- “We leverage Self-TICK (STICK) to substantially improve instruction-following ability via self-refinement and Best-of-N selection on multiple challenging benchmarks.”
- “We explore using LLM-generated checklists to assist human evaluators tasked with directly scoring an LLM output and find that this improves inter-annotator agreement.”
The bottom line is that the research study made use of the checklist prompt on a wide variety of scenarios. All in all, the checklist prompting technique made a useful and notable difference in the results generated by AI.
Checking On The Checklist Prompting
Should you always make use of a checklist prompt?
Nope, just use it when the need arises.
If your prompt is lengthy and contains a slew of questions, the odds are that a checklist prompt would be beneficial. Assuming that you normally ask just one question at a time, the main reason to use the checklist prompt might be if you anticipate that the output or response will be multi-fold. When an answer is going to contain lots of distinct points, the checklist prompt might be handy since the response will indubitably be laid out one point at a time.
My overall recommendation is to keep the checklist prompt at the top of your mind when using generative AI. Use the prompt as needed. Combine the checklist prompt with other successful prompting techniques. Follow those guidelines and I’d say you are good to go.
A final comment to conclude this exploration.
Mark Twain famously made this assertion: “Continuous improvement is better than delayed perfection.” You should be continuously improving your prompt engineering skillset. The checklist prompt is most definitely a crucial part of your prompting prowess.
I can now check off from my personal checklist that I provided coverage on the nature and use of the checklist prompting technique. Whew, I’m glad that got accomplished.
Read the full article here
