6 ms·

I think the conjecture is that each expert of GPT-4 has 220B parameters, for a total of 1.76T parameters.