5 ms·
Why is max_seq_len set to 2048 [1] when the model card says the context size is 8k [2]? [1] https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a9596f1dea0
by _pastel 2y ago
Why is max_seq_len set to 2048 [1] when the model card says the context size is 8k [2]?
[1] https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a9596f1dea06d9c564f9c0e35f/llama/model.py#L32 https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a959...
[2] https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a9596f1dea06d9c564f9c0e35f/MODEL_CARD.md https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a959...
- mkolodny 2y agoThat's just the default. You can set max_seq_len to 8k. From the readme [0]: > All models support sequence length up to 8192 tokens, but we pre-allocate the cache according to max_seq_len and max_batch_size values. So set those according to your hardware. [0] https://github.com/meta-llama/llama3/tree/14aab0428d3ec3a9596f1dea06d9c564f9c0e35f?tab=readme-ov-file#inference https://github.com/meta-llama/llama3/tree/14aab0428d3ec3a959...