7 ms·

This is the version we'll be testing on our rtx 6000 today! Thank you

• eblanshey 1mo ago
  Why not just run FP8 on vLLM with that much vRAM? It's plenty fast.
  • hadlock 1mo ago
    For high concurrency, using the blackwell's native native W4A4 MLP compute path, nvfp4 is something like a 1.2-1.5x performance increase over FP8. We're doing data enrichment (so, tasks completed successfully + tokens/second) so the performance bump shows up in the tasks/month number. I am just now getting the benchmarks running against 3.8 27b but I expect similar results from benching 3.6 27b at the same quant.
    • eblanshey 1mo ago
      I see. Did you see any intelligence degradation between FP8 and NVFP4 for 3.6 27B? You're using vLLM, right?
      • hadlock 1mo ago
        I didn't have time to run this as well, but we're getting a 99% agent completion rate across all tasks and 98% task decision that matches the human selected option(s) on Qwen 3.8 27B @ NVFP4. There may be a difference between FP8 and NVFP4 but it's inconsequential for our data enrichment purposes.