LLM Settings

<< back

This guide explains key configuration settings for Language Learning Models (LLMs). These settings allow you to control how the model generates text. Each setting impacts how creative, focused, or repetitive the generated responses will be.

Summary Table

Temperature

• Description: Temperature controls how random or creative the model’s responses are. It adjusts how confidently the model chooses the next word.

• Range: 0 to 2

• How it works:

  • Lower values make the model more focused and predictable. It will stick to safer, more obvious word choices.

  • Higher values add more randomness, making the responses more creative or unexpected.

• When to use: If you want straightforward, reliable answers, use a lower temperature. For more creative or varied responses, increase the temperature.

Top P (Nucleus Sampling)

• Description: Top P controls the variety of words the model can choose from by looking at the top choices with the highest combined probabilities. It dynamically narrows down the list of words the model considers.

• Range: 0 to 1

• How it works:

  • Lower values make the model focus on the most likely words, resulting in more predictable and focused responses.

  • Higher values give the model more flexibility to pick less likely words, making responses more diverse or creative.

• When to use: Use a higher Top P for more varied text, or a lower Top P if you want to keep the response concise and on-topic.

Frequency Penalty

• Description: Frequency penalty discourages the model from repeating the same words too often within its response. This helps to keep the text varied and avoid repetition.

• Range: -2.0 to 2.0

• How it works:

  • Higher values make the model less likely to repeat words, promoting more diverse word choices.

  • Lower or no values allow the model to repeat words as often as it sees fit.

• When to use: Increase this setting if you’re seeing too much word repetition. Leave it low if some repetition is okay or expected.

Presence Penalty

• Description: Presence penalty affects whether the model repeats topics or ideas that it has already mentioned in the response. It encourages the model to introduce new concepts.

• Range: -2.0 to 2.0

• How it works:

  • Higher values push the model to avoid bringing up the same topics again.

  • Lower or no values allow the model to talk about the same things multiple times.

• When to use: Use this setting when you want the model to keep introducing new ideas or content. Keep it lower if repeating key points is acceptable.

Max Tokens

• Description: Max Tokens sets a limit on how long the model’s response can be. Tokens are basically pieces of words or characters, and this controls the total number generated.

• How it works: A lower value restricts the response length, while a higher value allows for longer, more detailed answers.

• When to use: Set this depending on the response length you need. For short responses, use fewer tokens. For more elaborate answers, use more tokens.

Stop Words

• Description: Stop Words are specific words or phrases that you define to make the model stop generating text. Once the model hits one of these words, it stops.

• How it works: When the model generates a stop word, it finishes the response immediately.

• When to use: Use Stop Words to control when the model should stop, like at the end of a sentence, a specific phrase, or a custom endpoint (e.g., in dialogue systems).

These settings allow you to fine-tune how the model behaves, making it more predictable or creative, depending on your needs. Experimenting with these parameters helps you find the right balance for your specific use case.