5 ms·
I was only using a single DGX Spark, and this was earlier in the year, so I was running some pretty aggressively quantized models — probably in the 1–3 bit rang
by Alephinitesimal 27d ago
I was only using a single DGX Spark, and this was earlier in the year, so I was running some pretty aggressively quantized models — probably in the 1–3 bit range.
My main issue at the time was that my financial data had lots of messy notes, comments, and irregular annotations. The quantized models often failed to process all of that context consistently and would miss things. So I ended up generating a fake dataset with the same structure, asking Claude Code to work out the analysis on that, and then bringing the result back to the local model for the final pass.
I was mainly using llama.cpp at the time, before B12X support was integrated into vLLM, so I think I wasn't using it then.
- jacquesm 27d agoThank you, interesting info! I think the Sparks are an interesting platform, the power consumption / memory bandwidth / memory amount trade-off is completely different from the regular cards and it will take a while for the software to really take advantage of them.