Just based on working at big companies and knowing how much goes on that isn’t visible outside, i would guess they definitely have tried this especially if it o

Just based on working at big companies and knowing how much goes on that isn’t visible outside, i would guess they definitely have tried this especially if it only takes a few days on the cluster. It’s very likely they have experiments with weird ideas and architectures running nonstop for years now and the grokking idea is pretty well known.