ak_t

6 ms·

1.
by ak_t 21d ago
Many benchmarks now measure the total cost or energy usage per completed task.
2.
by ak_t 1mo ago
You don't have to send every single request twice, just the ones that are haven't returned in time. Wait until some threshold, such as your p95 latency, and send your backup request after that. Return whichever request comes back
3.
by ak_t 1mo ago
It's been pretty consistent, the smaller models (hundreds of billions of params) usually catch up in 6 months or so to their frontier counterparts, at least on benchmarks.
4.
by ak_t 1mo ago
I think it is more likely that a smaller model (<400B) with similar intelligence gets developed long before the hardware to serve a 2.4T model gets cheaper than 10k.
5.
by ak_t 1mo ago
It doesn't need extra quantization. The official weights are natively mixed precision FP4/FP8, so it fits in ~160GB. The API slowness is probably from being batched with other concurrent user requests. The provider's aggregat
6.
by ak_t 1mo ago
Note this is the 07/31 release of DSv4 flash and not the "preview" that they put out a couple months or so ago. I've been running this model locally for a week, and the preview version before that. This updated one feels
7.
by ak_t 1mo ago
I haven't live there since 2020, but before that it was all various SaaS and data pipeline related billboards.
8.
by ak_t 1mo ago
It's expensive to run it locally at full quality, but at least on my setup, its about 5 times faster than any API, and is completely private.
9.
When is NVLink worth it?(platform-fools.com)
61 points by ak_t 2mo ago|9 comments