That's just the default. You can set max_seq_len to 8k. From the readme [0]: > All models support sequence length up to 8192 tokens, but we pre-allocate the ca

That's just the default. You can set max_seq_len to 8k. From the readme [0]: > All models support sequence length up to 8192 tokens, but we pre-allocate the cache according to max_seq_len and max_batch_size values. So set those according to your hardware. [0] https://github.com/meta-llama/llama3/tree/14aab0428d3ec3a9596f1dea06d9c564f9c0e35f?tab=readme-ov-file#inference https://github.com/meta-llama/llama3/tree/14aab0428d3ec3a959...