Why Humor Is the Perfect Benchmark for Generative AI

🌈 Abstract

The article discusses using humor as a way to test the capabilities of large language models (LLMs) like ChatGPT and Google's Gemini. It argues that the ability to understand and generate humor is a good indicator of an LLM's ability to understand human intent, predict patterns of human understanding, and cleverly subvert those patterns - skills that are valuable across many domains.

🙋 Q&A

[01] Testing LLMs with Humor

1. Why does the author believe that testing an LLM's ability to be funny is a good way to evaluate its capabilities?

• Humor requires the LLM to understand human intent, predict patterns of human understanding, and then subvert those patterns in a clever way without being offensive or boring. This demonstrates a range of skills that are valuable across many domains, not just for performing rote tasks.
• The author argues that an LLM that can successfully write a genuinely funny stand-up comedy routine is demonstrating a deep understanding of people and language that would translate to other areas like creative writing, data analysis, etc.

2. How did the author's tests of ChatGPT and Google's Gemini model compare?

• ChatGPT was the clear winner, demonstrating a strong understanding of the quirks and dichotomies that make Bichon Frises a funny breed of dog. Its jokes played on recognizable patterns that Bichon owners would appreciate.
• In contrast, Gemini's routine came across as if it was just reciting facts about Bichons rather than crafting clever, humorous observations. It made basic errors that undermined its authenticity.

3. What do the results of the humor tests reveal about the underlying approaches and training of the two models?

• The author suggests the results indicate that ChatGPT was built with creativity and nuanced understanding of language in mind, while Gemini seems more focused on information retrieval and factual knowledge.
• The ability to craft authentic, surprising humor shows ChatGPT was trained to go beyond just reciting information, while Gemini appears limited in its ability to generate novel, creative content.

[02] The Value of Humor as a Benchmark for LLMs

1. Why does the author believe humor is a valuable benchmark for evaluating LLMs, even if most users don't care about their joke-writing abilities?

• Testing an LLM's humor reveals important insights about how it was built and trained - whether it was optimized more for information retrieval vs. creative expression, for example.
• The author argues that just as traditional computers were benchmarked by their ability to solve math problems, LLMs should be benchmarked by their ability to understand and generate human-like language, of which humor is a key component.

2. What other potential benefits does the author see in using humor as a benchmark for LLMs?

• Humor captures an LLM's ability to understand intent, predict patterns, and subvert expectations - skills that are valuable across many domains beyond just telling jokes.
• Evaluating an LLM's humor can provide insights into its strengths and weaknesses that may not be apparent from other types of tests or benchmarks.