4 ms·
Is there a catch? If not, this would be super useful.
by josefrichter 2mo ago
Is there a catch? If not, this would be super useful.
- stymaar 2mo agoThe catch is that the token generation speed is going to be limited by network latency, making it unbearably slow to run over the internet. It can be great on a local network though, especially if your workload is prefill-heavy (more text input to process than output tokens to emit).