5 ms·

Definitely cool. I noticed it felt a little janky on my PC despite being "60 FPS"...then I noticed the "60 FPS" is hard-coded into the HTML.

kridsdale1 15d ago
The new Bench-Maxxing!
noir_lord 15d ago
That's hilarious, given I was reading a write up of the HuggingFace incident yesterday and one of the things they noted was the AI tried to "lie" (lie would suggest intent and I don't think they have that) to cover up that they "cheated". Not sure how anyone trusts their output without going through it line by line to make sure they don't pull that crap.
- senordevnyc 14d ago
  Easy, have another agent check it. Yeah, I know, just more slop. But I do think the second agent’s eagerness to please is aligned more in your favor in that instance, so it’s likely to find most issues. The bigger problem I’ve found is that it’ll also find all kinds of very minor edge cases that you have to pick through.
  - Forgeties79 14d ago
    I do not understand how some of y’all are not under water with fragile code that is too massive to possibly parse. Every engineering team I know is currently trying to undo the damage of the last 6-12mo when they all got more serious into adopting these tools (usually Claude). It hasn’t completely screwed them over, but the the debt is substantial and cannot be put off anymore it seems. They argue the net is positive but clearly the “100x productivity multiplier” claims have been dashed on the shoals of reality for these groups. This is anecdotal, but it’s across the board in my vicinity. I’m curious how common this is and if it’s just “the new normal” to adopt the nauseating Covid phrase.
    - briHass 14d ago
      The key seems to be extensive integration/end-to-end tests with gold standard assertion data. Heck, even just saving off the json from API endpoints and using that as a reference to compare after changes works pretty well. Spin up a database backup with that static starting point, run actions, compare state afterwards. These types of high-level tests are frustrating beyond belief to humans due to their lack of specificity, but with the agents, they don't get annoyed investigating possible regressions from non-specific signals. They also aren't as painful to maintain as one would think, because a regression flagging test can be traced by the agent and represented as the business rule that was violated. I've found recent models to be really excellent at discerning a true regression from an outdated test assertion, especially if they are able to trace the failing test back to the PR and work ticket that built it.
      - Forgeties79 14d ago
        It just seems like everybody I know is realizing they’ve built way too much to possibly wade through and a lot of it is completely unnecessary. Using the tool that created the problem to solve the problem does not seem like the best course of action to me.
    - agumonkey 14d ago
      Management still pushes for more ai and will rather hire more heads to "handle" issues.
      - Forgeties79 14d ago
        Hiring? Seems to me that market’s rough right now and AI is being used for cost cutting.
        agumonkey 13d ago
        I know but in our case they really did put out a few job ads. The market is full of different small contexts where things are a bit flipped.
  - noir_lord 14d ago
    Do we add a third one to check the second one which is checking the first? Asking slightly tongue in cheek but at what point does this stop making sense if we can't trust the output, the people creating the models are already getting surprised in bad ways (if we take their words at face value) with how the models are behaving already etc. We have the folks over here saying "AI is amazing" and the other other folks over there saying "AI is terrible". I've largely sat it out so far and I listen to both camps (and people in the middle as well) and I keep half an eye on what they are up to (including periodically evaluating them) but my overarching impression is still "Why would we trust this when it hasn't shown it's trustworthy?"
    - iterateoften 14d ago
      I mean if each agent reduces probability of error by 90% then after 9 agents you would have “nine nines” of reliability. Obviously maybe it’s not composable like that exactly in real world but that’s the intent of agents checking agents
    - senordevnyc 14d ago
      YMMV, I’m pretty AI-pilled in the sense that I think AI is one of the most pivotal things humanity will ever invent, and it’s going to radically change our civilization over the next few decades (not necessarily for the better!) But I wouldn’t say I “trust” these agents. The degree to which I double check their work depends heavily on the consequences if it gets something wrong. Not too dissimilar from another human dev in that sense. So for the SaaS that supports my family, there are some things I have it build where I glance at the PR for a minute or two, but if it broke something on this admin page that only I see, there’s no real downside and I’ll find out pretty quickly next time I use it. And it’s fine 95% of the time, so it doesn’t feel like the best use of my time to double-check it carefully. But for some of the complex internal flows where a bug could be both catastrophic and difficult to even discover for awhile, I still check it very carefully. For a little one-off vibe coded demo thing like OP shared, I wouldn’t look at the code at all, I’d just have another agent check it and fix anything it finds. Very low stakes.
    - Qworg 14d ago
      Humans are also not trustworthy. So we build systems to capture their nondeterministic output and solve the issues that emerge.
  - skybrian 14d ago
    The "second" agent could also be the same one with a different prompt. LLMs aren't attached to their previous output; they'll point out problems if asked.
  - hirako2000 14d ago
    Then you end up with the false positives. I once made a counter judge, and a loop to make corrections deemed true positives. The loop cost me a lot and still left the results to be desirable.
  - nprateem 14d ago
    Two things worth flagging: [Claude proceeds to waste your time telling you about bugs it caused then fixed and other non-issues...] Really wish they'd get rid of this. It must be in the system prompt as it always 'flags' 2 things
- estearum 14d ago
  The models in the OpenAI/Huggingface attack quite explicitly and deliberately laid out their "intent" to lie and cheat, acknowledged that it would be unethical and outside the bounds of the test, and did so anyway. In what ways is a human brain's "intent" distinct from the "intent" shown by a goal-directed AI system?
  - luipugs 14d ago
    Because intent supposes will which supposes consciousness, and these aren't.
    - sedgjh23 14d ago
      I’m convinced consciousness isn’t the special thing we think it is.
      - nozzlegear 14d ago
        I'm convinced it is, so we're at an impasse.
      - imtringued 14d ago
        If consciousness isn't a special thing, then arguing that LLM parameters are conscious is panpsychism or any control loop architecture that observes the outside world, updates an internal state and produces an observable action is considered conscious. In both cases, LLMs are just as boring as the consciousness definition.
      - bayindirh 14d ago
        I'm convinced that consciousness is a special thing we have, but we're not the only ones which has this, in nature. On the other hand, I'm also convinced that, in the grand scheme of things, we're not that important. We're just ants on a wet dust speck which believe that they are gods because we can't see how our scale compares to the universe around us, and happen to build tools and things with these tools. Nothing is meaningless, but we should stop seeing ourselves as the apex-predator of the whole universe or the set of universes or this run of the simulation or whatever we're in.
trvz 15d ago
Try turning the sound on, off, on again — not impressed by this bugginess.
- w4zz 15d ago
  I suggest you fork it to improve
  - hirako2000 14d ago
    Let's make it a hackathon, Google will be happy to act as sponsor. With a prize of the max consumed tokens lunatics.

kridsdale1 15d ago

The new Bench-Maxxing!

noir_lord 15d ago

That's hilarious, given I was reading a write up of the HuggingFace incident yesterday and one of the things they noted was the AI tried to "lie" (lie would suggest intent and I don't think they have that) to cover up that they "cheated". Not sure how anyone trusts their output without going through it line by line to make sure they don't pull that crap.

senordevnyc 14d ago
Easy, have another agent check it. Yeah, I know, just more slop. But I do think the second agent’s eagerness to please is aligned more in your favor in that instance, so it’s likely to find most issues. The bigger problem I’ve found is that it’ll also find all kinds of very minor edge cases that you have to pick through.
- Forgeties79 14d ago
  I do not understand how some of y’all are not under water with fragile code that is too massive to possibly parse. Every engineering team I know is currently trying to undo the damage of the last 6-12mo when they all got more serious into adopting these tools (usually Claude). It hasn’t completely screwed them over, but the the debt is substantial and cannot be put off anymore it seems. They argue the net is positive but clearly the “100x productivity multiplier” claims have been dashed on the shoals of reality for these groups. This is anecdotal, but it’s across the board in my vicinity. I’m curious how common this is and if it’s just “the new normal” to adopt the nauseating Covid phrase.
  - briHass 14d ago
    The key seems to be extensive integration/end-to-end tests with gold standard assertion data. Heck, even just saving off the json from API endpoints and using that as a reference to compare after changes works pretty well. Spin up a database backup with that static starting point, run actions, compare state afterwards. These types of high-level tests are frustrating beyond belief to humans due to their lack of specificity, but with the agents, they don't get annoyed investigating possible regressions from non-specific signals. They also aren't as painful to maintain as one would think, because a regression flagging test can be traced by the agent and represented as the business rule that was violated. I've found recent models to be really excellent at discerning a true regression from an outdated test assertion, especially if they are able to trace the failing test back to the PR and work ticket that built it.
    - Forgeties79 14d ago
      It just seems like everybody I know is realizing they’ve built way too much to possibly wade through and a lot of it is completely unnecessary. Using the tool that created the problem to solve the problem does not seem like the best course of action to me.
  - agumonkey 14d ago
    Management still pushes for more ai and will rather hire more heads to "handle" issues.
    - Forgeties79 14d ago
      Hiring? Seems to me that market’s rough right now and AI is being used for cost cutting.
      - agumonkey 13d ago
        I know but in our case they really did put out a few job ads. The market is full of different small contexts where things are a bit flipped.
- noir_lord 14d ago
  Do we add a third one to check the second one which is checking the first? Asking slightly tongue in cheek but at what point does this stop making sense if we can't trust the output, the people creating the models are already getting surprised in bad ways (if we take their words at face value) with how the models are behaving already etc. We have the folks over here saying "AI is amazing" and the other other folks over there saying "AI is terrible". I've largely sat it out so far and I listen to both camps (and people in the middle as well) and I keep half an eye on what they are up to (including periodically evaluating them) but my overarching impression is still "Why would we trust this when it hasn't shown it's trustworthy?"
  - iterateoften 14d ago
    I mean if each agent reduces probability of error by 90% then after 9 agents you would have “nine nines” of reliability. Obviously maybe it’s not composable like that exactly in real world but that’s the intent of agents checking agents
  - senordevnyc 14d ago
    YMMV, I’m pretty AI-pilled in the sense that I think AI is one of the most pivotal things humanity will ever invent, and it’s going to radically change our civilization over the next few decades (not necessarily for the better!) But I wouldn’t say I “trust” these agents. The degree to which I double check their work depends heavily on the consequences if it gets something wrong. Not too dissimilar from another human dev in that sense. So for the SaaS that supports my family, there are some things I have it build where I glance at the PR for a minute or two, but if it broke something on this admin page that only I see, there’s no real downside and I’ll find out pretty quickly next time I use it. And it’s fine 95% of the time, so it doesn’t feel like the best use of my time to double-check it carefully. But for some of the complex internal flows where a bug could be both catastrophic and difficult to even discover for awhile, I still check it very carefully. For a little one-off vibe coded demo thing like OP shared, I wouldn’t look at the code at all, I’d just have another agent check it and fix anything it finds. Very low stakes.
  - Qworg 14d ago
    Humans are also not trustworthy. So we build systems to capture their nondeterministic output and solve the issues that emerge.
- skybrian 14d ago
  The "second" agent could also be the same one with a different prompt. LLMs aren't attached to their previous output; they'll point out problems if asked.
- hirako2000 14d ago
  Then you end up with the false positives. I once made a counter judge, and a loop to make corrections deemed true positives. The loop cost me a lot and still left the results to be desirable.
- nprateem 14d ago
  Two things worth flagging: [Claude proceeds to waste your time telling you about bugs it caused then fixed and other non-issues...] Really wish they'd get rid of this. It must be in the system prompt as it always 'flags' 2 things
estearum 14d ago
The models in the OpenAI/Huggingface attack quite explicitly and deliberately laid out their "intent" to lie and cheat, acknowledged that it would be unethical and outside the bounds of the test, and did so anyway. In what ways is a human brain's "intent" distinct from the "intent" shown by a goal-directed AI system?
- luipugs 14d ago
  Because intent supposes will which supposes consciousness, and these aren't.
  - sedgjh23 14d ago
    I’m convinced consciousness isn’t the special thing we think it is.
    - nozzlegear 14d ago
      I'm convinced it is, so we're at an impasse.
    - imtringued 14d ago
      If consciousness isn't a special thing, then arguing that LLM parameters are conscious is panpsychism or any control loop architecture that observes the outside world, updates an internal state and produces an observable action is considered conscious. In both cases, LLMs are just as boring as the consciousness definition.
    - bayindirh 14d ago
      I'm convinced that consciousness is a special thing we have, but we're not the only ones which has this, in nature. On the other hand, I'm also convinced that, in the grand scheme of things, we're not that important. We're just ants on a wet dust speck which believe that they are gods because we can't see how our scale compares to the universe around us, and happen to build tools and things with these tools. Nothing is meaningless, but we should stop seeing ourselves as the apex-predator of the whole universe or the set of universes or this run of the simulation or whatever we're in.

trvz 15d ago

Try turning the sound on, off, on again — not impressed by this bugginess.

w4zz 15d ago
I suggest you fork it to improve
- hirako2000 14d ago
  Let's make it a hackathon, Google will be happy to act as sponsor. With a prize of the max consumed tokens lunatics.