8 ms·
Deterministic seed
by Ericson2314 2mo ago
Deterministic seed
- nl 2mo agoDeterministic seeds barely work on a single machine, small scale training run. They just don't work at all on a many month long, 100K+ GPU cluster training run.
- Ericson2314 2mo agoSkill issue
- fc417fc802 2mo agoWhile using floating point? Not happening. You'd have to switch to fixed point, not just for the models themselves but also _all_ the training code (ie backprop). Even then you'd still need to account for order of events when an entire cluster of GPUs is involved. Also don't forget to account for any synthetic data sources. Or even non-synthetic for that matter - does your pipeline do any image resizing on the fly? Better make sure that's fully deterministic between machines (it almost certainly won't be). It's theoretically possible but I don't expect it to materialize any time soon.
- nl 2mo ago> theoretically possible I mean I guess, but not in a performant way if there are ever any hardware failures. And with 100K GPUs there are multiple hardware failures per day.
- Ericson2314 2mo ago> While using floating point? Where is this meme coming from? IEEE floating point is deterministic