6 ms·
The llama models were trained with a context size of 2048. By default llama.cpp limits it to 512, but you can use -c 2048 -n 2048 to get the full context window
by ddren 4y ago
The llama models were trained with a context size of 2048. By default llama.cpp limits it to 512, but you can use -c 2048 -n 2048 to get the full context window.
- worldsayshi 4y ago2048 words?
- teaearlgraycold 4y agoTokens
- doctoboggan 4y ago2048 "tokens", where one token is roughly equivalent to ¾ of a word
- wongarsu 4y agoTokens. Short or common words tend to be one token, while less common words are composed of multiple tokens. For GPT OpenAI gives the rule of thumb that on average you need four tokens to encode three words, and LLaMA should be similar
- worldsayshi 4y agoWell that's for sure bigger than my context size.