GregorStocks

6 ms·

1.
by GregorStocks 5mo ago
The "no longer" framing here is a bit surprising. I've been in this industry for 15 years and the whole time I've been reading comments from older engineers asserting that age discrimination is rampant and you should exp
2.
MCP Tool Design Is Not API Design(mage-bench.com)
1 points by GregorStocks 7mo ago|0 comments
3.
by GregorStocks 7mo ago
Down the road I can definitely imagine being interested in that (basically split out the "web-based replay viewer" part from the "LLM harness that I want to debug with a replay viewer" part, and then ingest non-LLM games
4.
by GregorStocks 7mo ago
I'm focused on Constructed for now. Eventually I'd like to try stuff like sideboarding, deck selection, deckbuilding, and drafting, but I wanna get the harness to the limit of models' abilities in Constructed first.
5.
by GregorStocks 7mo ago
Performance is tricky to measure. Right now the best measure of performance I've got is the "blunder index", but that's currently flagging a lot of stuff that I really don't consider to be true blunders - I think my
6.
by GregorStocks 7mo ago
Oh, that's a good bug report - historically it was just hallucinating card effects so I made the harness throw the Oracle text for all visible cards into the context, but I bet I forgot to do that for the mulligan decision specifically
7.
by GregorStocks 7mo ago
A lot of models (including Opus) keep insisting in their reasoning traces that going first can be a bad idea for control decks, etc, which I find pretty interesting - my understanding is that the consensus among pros is closer to "you
8.
by GregorStocks 7mo ago
To be clear, that's not estimated price, it's actual price I paid across all the real games. My hope is you'll see it trend down over time as I find more ways to make the harness token-efficient :)
9.
by GregorStocks 7mo ago
You still need an algorithm to decide, for each game that you're simulating, what actual decisions get made. If that algorithm is dumb, then you might decide Mono-Red Burn is the best deck, not because it's the best deck but becau
10.
by GregorStocks 7mo ago
The anxiety is coming from the "worrier" personality. Players are combination of a model version + a small additional "personality" prompt - in this case ( https://mage-bench.com/games/game_20260217_0
11.
by GregorStocks 7mo ago
Oh, fascinating - I didn't realize they released actual replay data publicly. It doesn't look like it's quite as rich as I'd like, though - it only captures one row per turn, so I don't think you can deduce things l
12.
by GregorStocks 7mo ago
Yep, plus you've got to worry about the card names (unless you're giving every single card a new name like Wizards did with "Through the Omenpaths") and whether a judge thinks that "no we don't distribute the i
13.
by GregorStocks 7mo ago
They were trained on the entire Internet, so they've basically picked up the rules by osmosis. They're fuzzy on specific cards and optimal strategy, but they pretty much know out-of-the-box how the game works, the same as if you w
14.
by GregorStocks 7mo ago
Well, it's hard to do it under the radar if I'm posting it on HackerNews :) I've put enough money into MTGO (and, sigh , Arena) that I don't want to roll the dice on a ban.
15.
by GregorStocks 7mo ago
Yeah, the intention here is not to answer "which deck is best" - the standard of play is nowhere near high enough for that. It's meant as more of a non-saturated benchmark for different LLM models, so you can say things like
16.
by GregorStocks 7mo ago
My understanding of the argument for "why these clients are legal" is basically that they're just implementing the rules engine, rules aren't copyrightable, card text is rules, and they aren't directly distributing
17.
by GregorStocks 7mo ago
I was really hoping I could build this on top of MTGO or Arena, just as a bot interacting with real Wizards APIs and paying the developers money. But they've got very strong "absolutely no bots" terms of service, and my under
18.
by GregorStocks 7mo ago
Yeah, that's why I'm using XMage for my project - it has real rules enforcement.
19.
by GregorStocks 7mo ago
Well, more games would be neat, but right now it's really tightly coupled with XMage - you can ungzip the stuff in https://github.com/GregorStocks/mage-bench/tree/master/websi... if you want to see
20.
by GregorStocks 7mo ago
XMage has non-LLM-based built in AIs, just using regular old if-then logic. Getting them to play against each other with no human interaction is the first thing I built. https://www.youtube.com/watch?v=a1W5VmbpwmY is an exa
21.
by GregorStocks 7mo ago
I'm not aware of any good ML models for MTG. I'm just using off-the-shelf LLMs with a custom harness. It'd certainly be possible to do RLHF or something using the harness I've built, but it'd be expensive - anybody
22.
by GregorStocks 7mo ago
You might be interested in Premodern: https://premodernmagic.com/ . You can play it on regular old MTGO. FOSS Magic clients are in a legal gray area at best. My mental model is that Wizards de facto tolerate clients like XMa
23.
by GregorStocks 7mo ago
You might be looking at really old games (meaning, like, Saturday) - I've made a lot of harness improvements recently which should make the "what does this card do?" hallucinations less common. But yeah, it still happens, esp
24.
by GregorStocks 7mo ago
My general intention is to tell them "you're playing MTG, your goal is to win, here are the tools available to you, follow whatever strategy you want" - I don't want to spoon-feed them strategy, that defeats the purpose
25.
by GregorStocks 7mo ago
For the 1v1 formats (Standard, Modern, Legacy) I'm basically just using the current metagame from MTGGoldfish. For Commander they get a random precon. At some point I might want a 1v1 "less complicated lines than Standard" fo
26.
by GregorStocks 7mo ago
You wouldn't really need a _ton_ of games to get plausible data, but unfortunately today each game costs real money - typically a dollar or more with my current harness, though I'm hoping to optimize it and of course I expect mode
27.
by GregorStocks 7mo ago
I want them to do politics in Commander, and theoretically they should - the chat log is exposed in the MCP tools just like the rest of the game history, and their prompts tell them to use chat. In practice they haven't really talked t
28.
Show HN: I taught LLMs to play Magic: The Gathering against each other(mage-bench.com)
117 points by GregorStocks 7mo ago|83 comments
29.
by GregorStocks 11mo ago
Looks like they do: > You can also manually install skills by adding them to ~/.claude/skills.
30.
by GregorStocks 1y ago
Yeah, but Dropbox is sort of playing on easy mode, because the data is "just files" and you can manually resolve the conflict with regular old text editors, etc. If you don't expose your app's data model in the file syst

More ›