4 ms·

I'd be curious in how well it passes 100th Coin's NES accuracy tests https://github.com/100thCoin/AccuracyCoin https://github.com/100thCoin/AccuracyCoin

by worble 9mo ago

utopiah 9mo ago
Indeed, that's what I kind of hinted at in https://news.ycombinator.com/item?id=46442195 https://news.ycombinator.com/item?id=46442195 and coincidentally https://news.ycombinator.com/item?id=46437688 https://news.ycombinator.com/item?id=46437688 briefly after, namely that OK, one can "generate" a "solution", that's much easier than before... but until we can verify somehow that it actually does what it say it does (and we know of hallucinations and have no reason to believe this changed) then testing itself, especially of well know "problems" is more and more important. That being said, it doesn't answer the "why" in the first place, an even more important question. At least though it does help somehow to compare with existing alternatives.
- garciasn 9mo ago
  Isn’t this how all software development works? Folks commit code, it’s tested, and reviewed, and then deployed. Why would this be any different?
  - PaulDavisThe1st 9mo ago
    That's not how software development works. Folks think, they write code, they do their own localized evaluation and testing, then they commit and then the rest of the (down|up)stream process begins. LLM's skip over the "actually verify that the code I just wrote does what I intended it to" step. Granted, most humans don't do this step as thoroughly and carefully as would be desirable (sometimes through laziness, sometimes because of a belief in (down|up)stream testing processes). But LLM's don't do it at all.
    - sally_glance 9mo ago
      They absolutely can do that if you give them the tools. Seeing Claude (I use it with opencode agents) run curl and playwright to verify and then fix it's implementation was a real 'wow' moment for me.
      - Q6T46nT668w6i3m 9mo ago
        We have different experiences. Often I’ll see Claude, et. al. find creative ways to fulfill the task without satisfying my intent, e.g., changing the implementation plan I specifically asked for, changing tolerances or even tests, and frequently disabling tests.
        DANmode 9mo ago
        Are you a customer?
        DANmode 9mo ago
        Don’t downvote because you don’t like the question. It obviously adds to the discussion: paid and non paid accounts are being conflated daily in threads like these! They’re not the same tier account! Free users, especially ones deemed less interesting to learn from for the future, are given table-scraps when they feel it’s necessary for load reasons.
        nineteen999 9mo ago
        Exactly. There's an impedance mismatch between those using the free/cheap tiers and those paying a premium, so the discussion gets squirrely because one side is talking about apples and the other oranges.
        DANmode 9mo ago
        Right. More specifically: One side is talking about apples, and the other is talking about mushy old apples, that sometimes you need to wait 12 hours for.
roger_ 9mo ago
I’m sure you can point Claude at that page and have it make the necessary changes to pass.
- deadbabe 9mo ago
  Or it could loop infinitely, never quite being able to pass all the tests.
  - hu3 9mo ago
    which is easily fixable by some human guidance
    - RAMJAC 9mo ago
      Sorta, I went into this not really knowing how to implement an emulator: https://github.com/RAMJAC-digital/RAMBO https://github.com/RAMJAC-digital/RAMBO With the NES there are all sorts of weird edge cases, one of which are NMI flags and resets; the PPU in general is kinda tricky to get right. Claude has had *massive** issues with this, and I've had to take control and completely throw out code it's generated. I'm restarting it with a clean slate though, as there are still issues with some of the underlying abstractions. PPU is still the bane of my existence, DMA, I don't like the instruction pipeline, haven't even gotten to the APU. It's getting an 80/130 on accuracy coin. Though, when it came to creating a WASM target, Claude was largely able to do it with minimal input on my end. Actually, getting the WASM emulator running in the browser was the least painful part of this project. You will run into three problems: 1) "The Wall" when any project becomes large enough, you need the context window to be *very* specific and scoped, with explicit details of what is expected, the success criteria and deliverables. 2) Ambiguity means Claude is going to choose the path of least resistance, and will pedantically avoid/add things which are not specced. Stubs for functions, "beyond scope", "deferred" are some favorite excuses to not refactoring or implementing obvious issues (anything that will go beyond the context window, Claude knows, but won't tell you will be punted work). 3) Chat bots *loooove* to talk, it will vomit code for days. Removing code/documentation is anathema to Claude. "Backward compatibility", deprecated, and legacy being its favorite.

I'd be curious in how well it passes 100th Coin's NES accuracy tests https://github.com/100thCoin/AccuracyCoin https://github.com/100thCoin/AccuracyCoin

by worble 9mo ago

I'd be curious in how well it passes 100th Coin's NES accuracy tests https://github.com/100thCoin/AccuracyCoin https://github.com/100thCoin/AccuracyCoin

utopiah 9mo ago

Indeed, that's what I kind of hinted at in https://news.ycombinator.com/item?id=46442195 https://news.ycombinator.com/item?id=46442195 and coincidentally https://news.ycombinator.com/item?id=46437688 https://news.ycombinator.com/item?id=46437688 briefly after, namely that OK, one can "generate" a "solution", that's much easier than before... but until we can verify somehow that it actually does what it say it does (and we know of hallucinations and have no reason to believe this changed) then testing itself, especially of well know "problems" is more and more important. That being said, it doesn't answer the "why" in the first place, an even more important question. At least though it does help somehow to compare with existing alternatives.

garciasn 9mo ago
Isn’t this how all software development works? Folks commit code, it’s tested, and reviewed, and then deployed. Why would this be any different?
- PaulDavisThe1st 9mo ago
  That's not how software development works. Folks think, they write code, they do their own localized evaluation and testing, then they commit and then the rest of the (down|up)stream process begins. LLM's skip over the "actually verify that the code I just wrote does what I intended it to" step. Granted, most humans don't do this step as thoroughly and carefully as would be desirable (sometimes through laziness, sometimes because of a belief in (down|up)stream testing processes). But LLM's don't do it at all.
  - sally_glance 9mo ago
    They absolutely can do that if you give them the tools. Seeing Claude (I use it with opencode agents) run curl and playwright to verify and then fix it's implementation was a real 'wow' moment for me.
    - Q6T46nT668w6i3m 9mo ago
      We have different experiences. Often I’ll see Claude, et. al. find creative ways to fulfill the task without satisfying my intent, e.g., changing the implementation plan I specifically asked for, changing tolerances or even tests, and frequently disabling tests.
      - DANmode 9mo ago
        Are you a customer?
        DANmode 9mo ago
        Don’t downvote because you don’t like the question. It obviously adds to the discussion: paid and non paid accounts are being conflated daily in threads like these! They’re not the same tier account! Free users, especially ones deemed less interesting to learn from for the future, are given table-scraps when they feel it’s necessary for load reasons.
        nineteen999 9mo ago
        Exactly. There's an impedance mismatch between those using the free/cheap tiers and those paying a premium, so the discussion gets squirrely because one side is talking about apples and the other oranges.
        DANmode 9mo ago
        Right. More specifically: One side is talking about apples, and the other is talking about mushy old apples, that sometimes you need to wait 12 hours for.

roger_ 9mo ago

I’m sure you can point Claude at that page and have it make the necessary changes to pass.

deadbabe 9mo ago
Or it could loop infinitely, never quite being able to pass all the tests.
- hu3 9mo ago
  which is easily fixable by some human guidance
  - RAMJAC 9mo ago
    Sorta, I went into this not really knowing how to implement an emulator: https://github.com/RAMJAC-digital/RAMBO https://github.com/RAMJAC-digital/RAMBO With the NES there are all sorts of weird edge cases, one of which are NMI flags and resets; the PPU in general is kinda tricky to get right. Claude has had *massive** issues with this, and I've had to take control and completely throw out code it's generated. I'm restarting it with a clean slate though, as there are still issues with some of the underlying abstractions. PPU is still the bane of my existence, DMA, I don't like the instruction pipeline, haven't even gotten to the APU. It's getting an 80/130 on accuracy coin. Though, when it came to creating a WASM target, Claude was largely able to do it with minimal input on my end. Actually, getting the WASM emulator running in the browser was the least painful part of this project. You will run into three problems: 1) "The Wall" when any project becomes large enough, you need the context window to be *very* specific and scoped, with explicit details of what is expected, the success criteria and deliverables. 2) Ambiguity means Claude is going to choose the path of least resistance, and will pedantically avoid/add things which are not specced. Stubs for functions, "beyond scope", "deferred" are some favorite excuses to not refactoring or implementing obvious issues (anything that will go beyond the context window, Claude knows, but won't tell you will be punted work). 3) Chat bots *loooove* to talk, it will vomit code for days. Removing code/documentation is anathema to Claude. "Backward compatibility", deprecated, and legacy being its favorite.