You can increase the context window beyond its max trained context using RoPE scaling[0] which will require more VRAM. But you can increase your context window

You can increase the context window beyond its max trained context using RoPE scaling[0] which will require more VRAM. But you can increase your context window for the same VRAM by quantizing the KV cache with FP8 (double the context) or TurboQuant (more than double)[1]. 0: https://medium.com/@leannetan/extending-context-length-with-hugging-faces-transformers-6b04db05b39a https://medium.com/@leannetan/extending-context-length-with-... 1: https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/#example-quantize-llama-attention-kv-cache-to-fp8 https://docs.vllm.ai/en/latest/features/quantization/quantiz...