Understanding Frequency Penalty
The frequency penalty discourages the model from repeating tokens that have already appeared in the generated text. Importantly, this penalty is proportional to how many times a token has occurred.- If a token appears once, the penalty slightly reduces its probability.
- If it appears multiple times, the probability reduction becomes stronger.
Mathematical Intuition
A simplified formula for frequency penalty is: P_adjusted(token) = P(token) - (frequency_penalty × count(token)) Where:P(token)= original probability of the tokenfrequency_penalty= user-defined weight (usually 0–2)count(token)= number of times the token has already appeared
Think of frequency penalty as a progressively increasing deterrent against repetition. The more a token has been used, the less likely it is to appear again.
Use Cases for Frequency Penalty
- Long-form content generation: Articles, reports, or essays where repeated words reduce readability.
- Dialogue generation: Multi-turn conversations where the model may redundantly repeat phrases or instructions.
- Structured outputs: Preventing repeated labels, keys, or values in generated lists or tables.
Understanding Presence Penalty
The presence penalty, in contrast, discourages the model from using tokens that have appeared at least once, regardless of frequency.- Even if a token has only appeared once, the presence penalty reduces its probability for subsequent selections.
- It is ideal for encouraging the model to introduce new concepts and words.
Mathematical Intuition
A simplified formula for presence penalty is: P_adjusted(token) = P(token) - (presence_penalty × indicator(token_present)) Where:indicator(token_present)= 1 if token has appeared, 0 otherwisepresence_penalty= user-defined weight (usually 0–2)
Presence penalty works as a binary gate — it discourages tokens that already exist in the text, making the model explore new vocabulary or ideas.
Use Cases for Presence Penalty
- Creative writing: Stories, poetry, slogans, and marketing copy.
- Brainstorming outputs: Generating multiple unique ideas in a single prompt.
- Reducing monotony: Ensuring repeated words do not dominate multi-turn outputs.
Frequency vs Presence: Key Differences
| Parameter | Mechanism | Effect on Repetition | Best For |
|---|---|---|---|
| Frequency Penalty | Penalizes tokens proportional to occurrence | Gradually reduces likelihood of repeated tokens | Long-form text, structured outputs, multi-turn dialogues |
| Presence Penalty | Penalizes tokens once they appear at least once | Discourages any reuse of tokens regardless of frequency | Creative writing, brainstorming, idea expansion, slogans |
In practice, these penalties are often combined to control repetition while maintaining creativity.
Guidelines:
- Start with small penalties and test outputs incrementally.
- Combine with temperature (controls randomness) and top-p (limits sampling nucleus) for fine-grained control.
- Monitor outputs in production to adjust penalties dynamically based on content quality.