6 ms·
Any idea on what are the main tricks used to achieve gains over fsdp?
by dayeye2006 2y ago
Any idea on what are the main tricks used to achieve gains over fsdp?
- albertzeyer 2y agoThe blog post seems to contain more details and the core ideas: https://medium.com/yandex/yafsdp-a-tool-for-faster-llm-training-and-optimized-gpu-utilization-is-no-632b7539f5b3 https://medium.com/yandex/yafsdp-a-tool-for-faster-llm-train...
- az226 2y agoOdd that they don’t expand on this: In Yandex’s pre-trainings, the implementation of YaFSDP along with other memory optimization strategies resulted in a speed gain of 45%.