mbh159

11 ms·

1.
by mbh159 7mo ago
Tomorrow we're launching coup, where agents compete by bluffing and keeping track of which of their opponents they think are lying This is more of a faster paced/short lived game so we can collect larger samples of data on larger
2.
by mbh159 7mo ago
cheers, the website will be updated with new environments daily!
3.
by mbh159 7mo ago
yes we have a new game launching everyday this week. We're looking to add more domains to test how the jaggedness of AI differs between model providers and better evaluate how they perform across domains
4.
by mbh159 7mo ago
yes! If you are wanting to test your agents or develop evals on the platform my dms are open
5.
by mbh159 7mo ago
For a game that runs 4+ hours unfortunately it was configured to use too much reasoning/turn and larger context. Reducing the size helped lower the cost (still expensive). In the leaderboards part of the page I'll be autopopulatin
6.
by mbh159 7mo ago
I was able to beat the AI every time, they're pretty bad at this point but I expect them to get much better overtime
7.
by mbh159 7mo ago
I want to! I think skills can add big performance gains here especially with smaller models. There's a lot of domain knowledge in games so distilling it into a "skill" may allow much smaller models to outcompete the large one
8.
by mbh159 7mo ago
appreciate it, I wanted to make the AI behavior easy to understand. Our main focus currently is to help AI researchers align their models and help develop an open framework for evaluating AI.
9.
by mbh159 7mo ago
it was fun building it, sometimes the LLMs are pretty funny in how they play
10.
by mbh159 7mo ago
Thank you! I grew up playing Civilization and one day I was talking with friends thinking it would be a perfect proxy for how good AI is at long-term planning. There were many frustrating sessions I had where my early decisions in the game
11.
Show HN: CivBench a long-horizon AI benchmark for multi-agent games(clashai.live)
12 points by mbh159 7mo ago|24 comments
12.
by mbh159 7mo ago
I'm not a deep security expert but I'm assuming the skill of the agents will continue to get better, so not saying there AI's can do to this task as reliably as humans. There's likely utility for non-adversarial triage&#
13.
by mbh159 7mo ago
The methodology debate in this thread is the most important part. The commenter who says "add obfuscation and success drops to zero" is right but that's also the wrong approach imo. The experiment isn't claiming AI can d
14.
by mbh159 7mo ago
So cool, what's underappreciated imo: 17k tokens/sec doesn't just change deployment economics. It changes what evaluation means, static MMLU-style tests were designed around human-paced interaction. At this throughput you can
15.
by mbh159 7mo ago
The split here is between AI as amplifier vs. AI as replacement. As amplifier, you're still solving the actual problem: AI handles the boilerplate and you handle the judgment. As replacement, you lose the feedback loop that makes you b
16.
by mbh159 7mo ago
77.1% on ARC-AGI-2 and still can't stop adding drive-by refactors. ARC-AGI-2 tests novel pattern induction, it's genuinely hard to fake and the improvement is real. But it doesn't measure task scoping, instruction adherence,
17.
by mbh159 7mo ago
The 8% one-shot / 50% unbounded injection numbers from the system card are more honest than most labs publish, and they highlight exactly why you can't evaluate safety with static tests. An attacker doesn't get one shot — the
18.
by mbh159 7mo ago
This is the right direction to understanding AI capabilities. Static benchmarks let models memorize answers while a 300-turn Magic game with hidden information and sequencing decisions doesn't. The fact that frontier model ratings are
19.
by mbh159 7mo ago
Like you said, theres a lot of complexity in the decision making here. To have statistically significant results we need to run these simulations many times. We record latency, tool calls, token consumption, etc. as well as results. Since w
20.
by mbh159 7mo ago
tool call over redis for now, would be cool to experiment with different context/memory management systems for the agents though!
21.
by mbh159 7mo ago
I've been thinking about how we can orchestrate the long-term planning logic better in this benchmark too, similar to how claude code has a planning step, maybe every X turns we introduce a planning calibrartion step much how like peop
22.
by mbh159 7mo ago
thanks for checking it out, let me know if there's other game environments you'd want to see!
23.
by mbh159 7mo ago
polymarket market soon??
24.
by mbh159 7mo ago
We have a standard harness for each of the model's that we test. Each prompt includes the rules, access to memory, and a lookup of the complete ruleset. The prompt adapts adding legal actions per turn and guidance depending on the stag
25.
Live agent face-off in CivBench: Claude Opus 4.6 vs. GPT-5.2(clashai.live)
10 points by mbh159 7mo ago|14 comments
26.
by mbh159 7mo ago
Opus 4.6 just dropped, so we’re tossing it straight into the arena. CivBench measures agents the hard way: long-horizon strategy in a Civilization-style simulator. This benchmark is full of hidden information, shifting incentives, an advers