4 ms·

Most of my and my peers PortCos run their own eval and benchmark sets, simply because they know what they need best. The reality is, capabilities have largely

Most of my and my peers PortCos run their own eval and benchmark sets, simply because they know what they need best. The reality is, capabilities have largely converged across foundation models over the last 18 months, and much of the value add is coming from the harness layer itself now. This has been the operating assumption for me and my peers, and has largely played out that way. That said, this has always been an issue with benchmarking since the very beginning. DB Benchmarks, compute benchmarks, and others that were external facing were always inherently a content and product marketing tool. The actual internal benchmarking used to model, understand, and enhance your product was always a closely held secret. Most of these conversations are happening, but largely in person and not on HN.

matan0904 10d ago
[flagged]
NitpickLawyer 10d ago
> capabilities have largely converged across foundation models over the last 18 months For reference, in March '25 the models du jour were Sonnet 3.7, gpt o4 and gemini 2.5 pro. GPT5 was in august '25. It's been a while since we've heard the old "models have stagnated". Oh well.
- tancop 10d ago
  It's not "models have stagnated" but "models released at the same time are on the same level". Improvements are still real but the relative gaps between OpenAI, Anthropic, Meta, Grok, Gemini and open models are closer than ever. That doesn't mean progress is slowing down, it's just more widely distributed.
  - alephnerd 10d ago
    This, and depending on the workflow and usecase, you don't necessarily need the latest and greatest with the right kind of harness engineering. Like everything in engineering, it's about tradeoffs and what works best for your specific problem.
    - porridgeraisin 6d ago
      There is also the aspect of model generations being robust under a particular ctx mgmt strategy/tailored harness. Only since this year are most models robust in this way. In many open models, the problem is that successive generations are not that robust yet. i.e if you have a working setup, next generation of the same model family will need way too much reworking. So it's difficult to upgrade the model. Gemini is stellar in this respect (probably because, and I suspect, their flash models are distillations of the same larger model). OpenAI/Anthropic are too, but higher cost of newly released models is quite the blocker. Deepseek is especially hard to upgrade. Kimi doesnt target this segment, so people that can post train it do so. Qwen and llama are the stellar open ones, incredibly stable, although llama has stopped receiving upgrades in a while. It is still used a lot though.
  - NitpickLawyer 10d ago
    Ah, I see. I misunderstood then. The thing about "gains come from the harness" made me think about it in that way.
Imustaskforhelp 10d ago
> The reality is, capabilities have largely converged across foundation models over the last 18 months, and much of the value add is coming from the harness layer itself now. Can you please elaborate on what are your thoughts on open weights models (GLM 5.3, Kimi K3, deepseek etc.) and if the value add is coming from the harness layer itself, then thoughts on open source harnesses (there are so many harnesses but to name a few: opencode, pi [omp as well], maki, codex is OSS as well, fx.sh) and you can always combine them with skills (Obra/superpowers, matt pocock skills plus using these skills and others to create some other custom skills tailored to your use case as well) And what about the combination of both now with this cheap open weights models + open source harnesses and other things to compete over the closed garden ecosystems? How does that comparison follow in reality Could you in theory use these methods to save on the massively expensive $$$ token spending on Anthropic/OAI? (Personal anecdote but I have GLM 5.3 + maki [sometimes omp/opencode but mostly maki] and its good enough for most use cases out there that I have and I dont know of too many use cases outside of say recreation of games for examples maybe that I would prefer complete SOTA models. I would also love to know where you believe that SOTA models absolutely do still make the difference discounting the benefits provided by the harness.)

matan0904 10d ago

[flagged]

NitpickLawyer 10d ago

> capabilities have largely converged across foundation models over the last 18 months For reference, in March '25 the models du jour were Sonnet 3.7, gpt o4 and gemini 2.5 pro. GPT5 was in august '25. It's been a while since we've heard the old "models have stagnated". Oh well.

tancop 10d ago
It's not "models have stagnated" but "models released at the same time are on the same level". Improvements are still real but the relative gaps between OpenAI, Anthropic, Meta, Grok, Gemini and open models are closer than ever. That doesn't mean progress is slowing down, it's just more widely distributed.
- alephnerd 10d ago
  This, and depending on the workflow and usecase, you don't necessarily need the latest and greatest with the right kind of harness engineering. Like everything in engineering, it's about tradeoffs and what works best for your specific problem.
  - porridgeraisin 6d ago
    There is also the aspect of model generations being robust under a particular ctx mgmt strategy/tailored harness. Only since this year are most models robust in this way. In many open models, the problem is that successive generations are not that robust yet. i.e if you have a working setup, next generation of the same model family will need way too much reworking. So it's difficult to upgrade the model. Gemini is stellar in this respect (probably because, and I suspect, their flash models are distillations of the same larger model). OpenAI/Anthropic are too, but higher cost of newly released models is quite the blocker. Deepseek is especially hard to upgrade. Kimi doesnt target this segment, so people that can post train it do so. Qwen and llama are the stellar open ones, incredibly stable, although llama has stopped receiving upgrades in a while. It is still used a lot though.
- NitpickLawyer 10d ago
  Ah, I see. I misunderstood then. The thing about "gains come from the harness" made me think about it in that way.

Imustaskforhelp 10d ago

> The reality is, capabilities have largely converged across foundation models over the last 18 months, and much of the value add is coming from the harness layer itself now. Can you please elaborate on what are your thoughts on open weights models (GLM 5.3, Kimi K3, deepseek etc.) and if the value add is coming from the harness layer itself, then thoughts on open source harnesses (there are so many harnesses but to name a few: opencode, pi [omp as well], maki, codex is OSS as well, fx.sh) and you can always combine them with skills (Obra/superpowers, matt pocock skills plus using these skills and others to create some other custom skills tailored to your use case as well) And what about the combination of both now with this cheap open weights models + open source harnesses and other things to compete over the closed garden ecosystems? How does that comparison follow in reality Could you in theory use these methods to save on the massively expensive $$$ token spending on Anthropic/OAI? (Personal anecdote but I have GLM 5.3 + maki [sometimes omp/opencode but mostly maki] and its good enough for most use cases out there that I have and I dont know of too many use cases outside of say recreation of games for examples maybe that I would prefer complete SOTA models. I would also love to know where you believe that SOTA models absolutely do still make the difference discounting the benefits provided by the harness.)