Generative AI For The Rest Of US

Considering Perplexity

Perplexity is a concept in language modeling that helps us understand how well a model can predict or understand a piece of text. Imagine you're trying to guess the next word in a sentence: if you can often guess right, you're not very "perplexed" by the sentence. In language models, lower perplexity means the model finds the text more predictable, hence it's more confident in handling tasks with such text.

Researchers have observed that language models like GPT perform better with prompts - essentially instructions or inputs given to the model - that are more "natural" or similar to what they've seen during their training. However, determining what is "natural" is not always straightforward since these models have been trained on vast amounts of data, and we usually don't know exactly what's in that data. The concept of using perplexity as a measurement is introduced as a clever workaround. Perplexity measures how surprised or uncertain a model is with a given piece of text. The less surprised (or the lower the perplexity), the better the model understands the prompt, leading to improved performance on tasks.

In practice, if we're working with a language model and aiming for the best results, choosing prompts with lower perplexity could lead to better outcomes. But how can we reduce the perplexity of a prompt? What if we ask the model to create the prompt for us? This technique could be beneficial since the model is more familiar with the language it generates, the same it has seen during its training.

Suppose you want to ask the model the following question:

Write a blog post about aging gracefully.

You can use the following prompts to request the model to create a prompt for us:

Write a list of prompts that would lead to a blog post about aging gracefully.