6 ms·
Do you need to run all experts at once?
by throwaway48540 2y ago
Do you need to run all experts at once?
- jiggawatts 2y agoNo, but different ones are invoked for each word so typically they’d all be preloaded into VRAM. Also, because of batching, different users would use different experts concurrently. There’s no detailed information about how GPT4 is hosted, but we can guess from competing models and the leaked info.