Is pre-training in FP8 new? Also, 10M input token context is insane! EDIT: https://huggingface.co/meta-llama/Llama-3.1-405B https://huggingface.co/meta-llama/

Is pre-training in FP8 new? Also, 10M input token context is insane! EDIT: https://huggingface.co/meta-llama/Llama-3.1-405B https://huggingface.co/meta-llama/Llama-3.1-405B is BF16 so yes, it seems training in FP8 is new.