gertlabs

6 ms·

1.
by gertlabs 8d ago
Inception is one of the most interesting neolabs with their diffusion-based architectures. My understanding is that their primary business is low latency voice applications but they are seriously pursuing coding. We tested Mercury 2.5 Previ
2.
by gertlabs 10d ago
The results we've been seeing internally on our physics and circuit design environments are expert-level and beyond-expert-level results from models that Astra completely outclasses across the board on our evaluation suite (Fable 5+&#x
3.
by gertlabs 10d ago
> Delivering the benefits of scientific progress and economic growth that very intelligent machines enable. I think we're very close to the point where AI-driven breakthroughs outside of pure math and software start to really affect
4.
by gertlabs 24d ago
One problem is that $30 per run is really noisy for many verifiable tasks. Ours usually run at least an order of magnitude more for a model in GLM 5.3's price class. We just published GLM 5.3 results on our multi-agent coding evaluatio
5.
Creating Reliable AI for an Adversarial World(gertlabs.com)
2 points by gertlabs 27d ago|0 comments
6.
by gertlabs 28d ago
These models have gotten a fair amount of attention -- we're hoping it's enough to get them added to some reliable inference providers and OpenRouter, at which point we'll run them on our full benchmark suite.
7.
by gertlabs 1mo ago
We've seen a pretty consistent pattern in our evaluations where Go is among the languages that models perform worst with (alongside Python), for reasons unclear. Our coding evaluations are typically measuring the foresight and planning
8.
by gertlabs 1mo ago
SlimeBallBench? Looks cool, love the implementation! You would most likely need a LOT of environments like these if your goal was selling them to labs
9.
by gertlabs 1mo ago
The reality is that cost is the primary constraint for the public benchmark we provide. While we run enough samples to get results that are generally quite accurate on average, we only produce ~10 coding submissions per language for each mo
10.
by gertlabs 1mo ago
That's a different problem than benchmark saturation, and it's something that we are actively working on measuring objectively. I agree that Opus 5 is not a great model, despite being clearly intelligent. It seems like a personali
11.
by gertlabs 1mo ago
I started thinking about this back after the Llama 4 release, and since then our team has put a lot of thought into designing evaluations that don't saturate, are resistant to contamination, and can scale. What has worked best for us i
12.
by gertlabs 2mo ago
We run an evaluation that is designed to be less vulnerable to benchmaxxing because there aren't correct solutions; agents are interacting in the same environment as other agents. And it's private, and our public benchmark is not
13.
by gertlabs 2mo ago
The Efficiency tab at https://gertlabs.com/rankings?mode=oneshot_coding (only have cost data for the coding evaluations)
14.
by gertlabs 2mo ago
Yes, they are all benchmaxxed, but the question is how benchmaxxed they are relative to each other. We run an evaluation that only compares models in open-ended multi-agent environments where agents affect each other, primarily testing writ
15.
Better Agent Leagues and Decision Supervision from Branched Rollouts(gertlabs.com)
2 points by gertlabs 2mo ago|0 comments
16.
by gertlabs 2mo ago
In our multi-agent game coding evaluations, we usually see Chinese models struggle in one-shot reasoning but make up for it with tool use and iterating towards better solutions. Kimi K3 follows that pattern, ranking 19th in one-shot coding
17.
by gertlabs 2mo ago
LLM as judge / self-distillation is effective insofar as it can make models more reliably do things they are already capable of. But I agree that for pushing the frontier of what a model is capable of understanding and producing, inces
18.
by gertlabs 2mo ago
The human solutions are all written in Python, which creates a significant length bias, whereas the AI models are assigned to create solutions randomly distributed across 11 relevant programming languages, most of which are inherently more
19.
by gertlabs 2mo ago
Gemini models struggle with agentic coding/tool use/exploration, but they are actually quite smart in one-shot reasoning. They're not as far behind as people think. It's mostly post-training and productization issues, wh
20.
by gertlabs 2mo ago
That's a much shorter and more elegant proof than I was expecting, especially after reading some of the earlier Erdos proofs. GPT 5.6 Sol is the real deal.
21.
by gertlabs 2mo ago
Grok 4.5 is a huge step up from their next best model and now around the same performance as GLM 5.2, but it's not exactly at the frontier of the cost efficiency curve in our coding evaluations. That curve is defined by the 2 lighter G
22.
by gertlabs 2mo ago
We have it slightly ahead of Fable in our multi-agent coding evaluations. Fable's main advantage is that its average solution size is smaller. However, GPT 5.6 Sol is a substantial improvement from GPT 5.4/5.5 which would write ve
23.
by gertlabs 3mo ago
In our coding evaluations, we found Sonnet 5 is more capable than Sonnet 4.6 (which was an underrated model itself), but is now faster and slightly cheaper. Sonnet 5's performance is comparable to GLM 5.2 in both one-shot coding and ag
24.
by gertlabs 3mo ago
This is something we omit for a few reasons but it's probably the biggest blind spot in our evaluations; we opt-in to auto-reasoning/adaptive reasoning or max thinking token budgets where supported (supported by most models now),
25.
by gertlabs 3mo ago
It would have made things easier for us if Sonnet 4.6 scored lower, but it's a great model and the data is real. It doesn't have a higher capability score than Fable, though. We break our coding evaluations into 2 parts, and &quot
26.
by gertlabs 3mo ago
We've spent some time trying to understand this anomaly, even re-running Sonnet 4.6 through our evaluations to see if that would bring down its scores... and it didn't. I don't know what they did differently, but it's ba
27.
by gertlabs 3mo ago
We use a rotating pool of ~100 games for the coding parts of the benchmark, and are scored objectively based on ratings similar to Elo. Models write code submissions to interact with the environment, then are evaluated in large batches agai
28.
by gertlabs 3mo ago
It's 100% due to tool use -- Flash adapts much better to our custom harness with tool names that are not identical to what models were likely trained on. DeepSeek V4 Pro performs much worse in that aspect than almost all other recent r
29.
by gertlabs 3mo ago
Scroll to the bottom for the methodology (sorry, this should be linkable)
30.
by gertlabs 3mo ago
GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models

More ›