13 ms·
Wow, setting the "ubatch" size to 2048 (default is 512), like the "batch" size roughly doubled prompt processing speed.
by aktenlage 19d ago
Wow, setting the "ubatch" size to 2048 (default is 512), like the "batch" size roughly doubled prompt processing speed.