5 ms·
In my benchmark Deepseek-v4-flash did much better than Qwen 3.8 27B at reverse engineering. https://alexander-hanel.github.io/StressingLLMs/ https://alexander-
by __alexander 26d ago
In my benchmark Deepseek-v4-flash did much better than Qwen 3.8 27B at reverse engineering.
https://alexander-hanel.github.io/StressingLLMs/ https://alexander-hanel.github.io/StressingLLMs/
- petu 25d ago> This project evaluates local language models running on a single NVIDIA DGX Spark. "Did much better" is a bit misleading w/o that context and 1 hour time limit -- your benchmark design heavily favors V4 Flash. From results on your page V4 Flash processed 1-1.5M tokens an hour, while Q3.8 27B was failed before even reaching 200K tokens. By the way, how are you running V4 Flash on single Spark? Was it quantized?
- __alexander 25d agoA single model was loaded at a time and each model was given a 90 minute timeout. This is how I’m running it (actually an older version because they deleted the repo and replaced it) https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark