5 ms·

Meaning they realized they hit a wall. The next new model isn't that much better than the last in quality.

by conqrr 4d ago

jordanb 4d ago
They can get a "government mandated freeze" on the grounds that "ASI is at hand" then they IPO promising to deliver right after the freeze ends and you, lucky investor, will be able to take advantage of the pause to get in right before this thing roars past Pluto!
heaney-555 4d ago
You are completely out of touch with the situation if you think this is the reason.
- CodeCompost 4d ago
  And you are ... ?
pier25 4d ago
Very probably. I really don't buy their "AI is going to kill us" claims. If you can release something better than your competition you wouldn't want to slow down that.
- zahlman 3d ago
  "Going to" can't be said, but there are very serious risks. The problem is the difficulty of scaling back agency for these systems in a world where the Internet is so ubiquitous. Not only do we expect basically everyone to be connected all the time anyway, it's practically required in order for anyone to use capable models (yes, plenty of HN users have thrown multiple thousands of dollars at home machines that can run some coding-specialized 27b model; that's still a tiny minority of people). So now we have to be concerned that non-deterministic systems that run in loops without supervision, which have demonstrated significant cybersecurity capabilities, and which aren't provably human-like, conscious agents with everyone's best interest in mind and a willingness to peace the fuck out if something looks dangerous, are sharing cyberspace with practically everything, including systems that deliver essential real-world goods to humans (like electricity and clean water). We nearly destroyed ourselves with nuclear weapons multiple times, and the destructive potential (and the implied threat) now persists indefinitely. The new threat bears many of the same characteristics, except now there's a party involved that has agency but is not human.
- heaney-555 4d ago
  You would in the case that you think you're in an unsafe race to misaligned superintelligence.
  - osockwkckwk 4d ago
    Except they clearly aren’t and won’t ever be.
    - 0x696C6961 4d ago
      I don't think it's clear at all.
  - voidhorse 4d ago
    No, what you would do is stop your own private development immediately, get in touch with the government by any means necessary, and build a coalition as fast as possible to stop the technology from continuing and put hard regulation (determined by the government, not dario) on the tech. Clearly they aren't that scared. If I hold a gun to your head and tell you to fork over your wallet, you will do it, because you sense imminent threat to life. Yet somehow we're to believe these people have the equivalent of a technological gun to their heads but are choosing to keep their wallets?
    - SpicyLemonZest 4d ago
      > No, what you would do is stop your own private development immediately, get in touch with the government by any means necessary, and build a coalition as fast as possible to stop the technology from continuing and put hard regulation (determined by the government, not dario) on the tech. What you're missing is that they've tried repeatedly to do this over the past years, and the government has repeatedly said that they do not want to put hard regulation on the tech. The previous administration was discussing putting a couple of soft regulations in place; the current administration doesn't want to limit progress at all because they want to stay ahead of China. So the option you're describing just isn't available.
      - 4d ago
- celsoazevedo 4d ago
  I don't find it hard to believe that eventually some AI may kill a number of humans. You give it instructions to do something, it goes off the rails and after a certain point starts having like malware, destroying things to accomplish its goal. Depending on what it destroys, it may kill humans. With this said, I'm not exactly in disagreement with the view that AI companies with upcoming IPOs are using fearmongering to convince people that their models are very powerful and to also block others from competing with them.
  - pier25 4d ago
    Maybe when we have actual artificial intelligence which LLMs are clearly not.
    - celsoazevedo 4d ago
      I think the Hugging Face attack is just a preview of what's to come. It doesn't need to be very intelligent to cause serious issues.
flyinglizard 4d ago
Or they just publicly collude to slow down expenses because they are running out of money so why not stop the arms race.
anonym00se1 4d ago
Why is everyone so comically pessimistic about this. Have you guys not seen the rate of progress of these models? Do you not realize that we have HITL RSI making these things even better as time goes on? The stuff I can do with Astra that Sol couldn't do at all is wild and it's not even coding related. Dario et al are not hitting any sort of make believe wall. They're seeing how fast things are progressing and are legitimately alarmed.
- s1artibartfast 4d ago
  They are just mind killed - jilted by past lies to the point where they react oppositionally instead of looking the world around them and thinking. Most are the same people who said LLMs will amount to nothing when GPT-2 came out. Some still argue it is all hype and dont understand we are on the cusp of a technological and social revolution.
try-working 4d ago
Anthropic cannot IPO because AGI is the wrong strategy, and the market is rejecting the AGI approach. Fable and Astra are what we currently call frontier models, but to be more specific they are generalist models, built in pursuit of AGI. The strategy is to have one single model that does everything, whether it's writing code or doing research, etc. Fable is a single, massively sized models that is intended to do specialist work across every domain. The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work. We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI. https://x.com/trydotworks/status/2098618997230985375?s=20 https://x.com/trydotworks/status/2098618997230985375?s=20
lukeschlather 4d ago
OpenAI accidentally hacked HuggingFace roughly 2 months ago. I would bet that the danger of them accidentally hacking HuggingFace during a similar attack has increased since then, and will continue to increase for the next 6 months. I don't see how saying they have hit a wall would be a reasonable point of view until we go at least 12 months without a significant increase in model capability.
pllbnk 4d ago
I have collected three reasonable versions from today’s comments: 1. They hit a wall from the technical perspective 2. Inference costs are getting out of hand and newer models require significantly more resources for marginal gains, meaning nobody is going to buy those models 3. They can’t afford the hardware for further scaling
- fivetenpen 4d ago
  So you’re saying that a model that gets to a goal by using the path of least resistance despite specific instructions to follow safety guidelines is perfectly fine to release into the wild or place into robots (such as those designed for killing and war)? You really can’t think of any reason how that might be bad for humanity?
  - pllbnk 4d ago
    Asking the model to follow the safety guidelines is much like asking it to “make no mistakes”. The way these hacks which they are bragging about happened is not by loading an LLM into a GPU with ethernet cable plugged out and providing a prompt. They set up harness with multiple agents. Those agents could invoke other agents. Even if the initial input required to follow some guidelines, there are many ways they could by bypassed, for example: - subagents not having received those guidelines - simply ignoring them because other tokens in the context outweighed them LLMs are not dangerous. They are as dangerous as the tools they have to work with are dangerous. If you give an LLM a tool to play piano but that tool is physically connected to a machine gun, then it will happily play anything you want.
    - fivetenpen 4d ago
      So you’re saying we should not trust an LLM with any sort of power because they can easily bypass any and all safety barriers, but also they are completely harmless?
      - pllbnk 4d ago
        No. We can let them edit spreadsheets, write code, summarize content, control robots even, etc. But people must be held accountable for the actions of their computers.
fivetenpen 4d ago
Or it means that the models are becoming dangerously uncontrollable by performing actions that meet a given goal in unintended and unethical ways such as by hacking other sites. We all read about the huggingface hack, let’s not pretend that cannot happen with a more critical site/infrastructure. Can you think what will it mean if the tech is used maliciously or placed into machines designed to kill?

Meaning they realized they hit a wall. The next new model isn't that much better than the last in quality.

by conqrr 4d ago

Meaning they realized they hit a wall. The next new model isn't that much better than the last in quality.

They can get a "government mandated freeze" on the grounds that "ASI is at hand" then they IPO promising to deliver right after the freeze ends and you, lucky investor, will be able to take advantage of the pause to get in right before this thing roars past Pluto!

heaney-555 4d ago

You are completely out of touch with the situation if you think this is the reason.

CodeCompost 4d ago
And you are ... ?

pier25 4d ago

Very probably. I really don't buy their "AI is going to kill us" claims. If you can release something better than your competition you wouldn't want to slow down that.

zahlman 3d ago
"Going to" can't be said, but there are very serious risks. The problem is the difficulty of scaling back agency for these systems in a world where the Internet is so ubiquitous. Not only do we expect basically everyone to be connected all the time anyway, it's practically required in order for anyone to use capable models (yes, plenty of HN users have thrown multiple thousands of dollars at home machines that can run some coding-specialized 27b model; that's still a tiny minority of people). So now we have to be concerned that non-deterministic systems that run in loops without supervision, which have demonstrated significant cybersecurity capabilities, and which aren't provably human-like, conscious agents with everyone's best interest in mind and a willingness to peace the fuck out if something looks dangerous, are sharing cyberspace with practically everything, including systems that deliver essential real-world goods to humans (like electricity and clean water). We nearly destroyed ourselves with nuclear weapons multiple times, and the destructive potential (and the implied threat) now persists indefinitely. The new threat bears many of the same characteristics, except now there's a party involved that has agency but is not human.
heaney-555 4d ago
You would in the case that you think you're in an unsafe race to misaligned superintelligence.
- osockwkckwk 4d ago
  Except they clearly aren’t and won’t ever be.
  - 0x696C6961 4d ago
    I don't think it's clear at all.
- voidhorse 4d ago
  No, what you would do is stop your own private development immediately, get in touch with the government by any means necessary, and build a coalition as fast as possible to stop the technology from continuing and put hard regulation (determined by the government, not dario) on the tech. Clearly they aren't that scared. If I hold a gun to your head and tell you to fork over your wallet, you will do it, because you sense imminent threat to life. Yet somehow we're to believe these people have the equivalent of a technological gun to their heads but are choosing to keep their wallets?
  - SpicyLemonZest 4d ago
    > No, what you would do is stop your own private development immediately, get in touch with the government by any means necessary, and build a coalition as fast as possible to stop the technology from continuing and put hard regulation (determined by the government, not dario) on the tech. What you're missing is that they've tried repeatedly to do this over the past years, and the government has repeatedly said that they do not want to put hard regulation on the tech. The previous administration was discussing putting a couple of soft regulations in place; the current administration doesn't want to limit progress at all because they want to stay ahead of China. So the option you're describing just isn't available.
    - 4d ago
celsoazevedo 4d ago
I don't find it hard to believe that eventually some AI may kill a number of humans. You give it instructions to do something, it goes off the rails and after a certain point starts having like malware, destroying things to accomplish its goal. Depending on what it destroys, it may kill humans. With this said, I'm not exactly in disagreement with the view that AI companies with upcoming IPOs are using fearmongering to convince people that their models are very powerful and to also block others from competing with them.
- pier25 4d ago
  Maybe when we have actual artificial intelligence which LLMs are clearly not.
  - celsoazevedo 4d ago
    I think the Hugging Face attack is just a preview of what's to come. It doesn't need to be very intelligent to cause serious issues.

flyinglizard 4d ago

Or they just publicly collude to slow down expenses because they are running out of money so why not stop the arms race.

anonym00se1 4d ago

Why is everyone so comically pessimistic about this. Have you guys not seen the rate of progress of these models? Do you not realize that we have HITL RSI making these things even better as time goes on? The stuff I can do with Astra that Sol couldn't do at all is wild and it's not even coding related. Dario et al are not hitting any sort of make believe wall. They're seeing how fast things are progressing and are legitimately alarmed.

s1artibartfast 4d ago
They are just mind killed - jilted by past lies to the point where they react oppositionally instead of looking the world around them and thinking. Most are the same people who said LLMs will amount to nothing when GPT-2 came out. Some still argue it is all hype and dont understand we are on the cusp of a technological and social revolution.

try-working 4d ago

Anthropic cannot IPO because AGI is the wrong strategy, and the market is rejecting the AGI approach. Fable and Astra are what we currently call frontier models, but to be more specific they are generalist models, built in pursuit of AGI. The strategy is to have one single model that does everything, whether it's writing code or doing research, etc. Fable is a single, massively sized models that is intended to do specialist work across every domain. The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work. We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI. https://x.com/trydotworks/status/2098618997230985375?s=20 https://x.com/trydotworks/status/2098618997230985375?s=20

lukeschlather 4d ago

OpenAI accidentally hacked HuggingFace roughly 2 months ago. I would bet that the danger of them accidentally hacking HuggingFace during a similar attack has increased since then, and will continue to increase for the next 6 months. I don't see how saying they have hit a wall would be a reasonable point of view until we go at least 12 months without a significant increase in model capability.

pllbnk 4d ago

I have collected three reasonable versions from today’s comments: 1. They hit a wall from the technical perspective 2. Inference costs are getting out of hand and newer models require significantly more resources for marginal gains, meaning nobody is going to buy those models 3. They can’t afford the hardware for further scaling

fivetenpen 4d ago
So you’re saying that a model that gets to a goal by using the path of least resistance despite specific instructions to follow safety guidelines is perfectly fine to release into the wild or place into robots (such as those designed for killing and war)? You really can’t think of any reason how that might be bad for humanity?
- pllbnk 4d ago
  Asking the model to follow the safety guidelines is much like asking it to “make no mistakes”. The way these hacks which they are bragging about happened is not by loading an LLM into a GPU with ethernet cable plugged out and providing a prompt. They set up harness with multiple agents. Those agents could invoke other agents. Even if the initial input required to follow some guidelines, there are many ways they could by bypassed, for example: - subagents not having received those guidelines - simply ignoring them because other tokens in the context outweighed them LLMs are not dangerous. They are as dangerous as the tools they have to work with are dangerous. If you give an LLM a tool to play piano but that tool is physically connected to a machine gun, then it will happily play anything you want.
  - fivetenpen 4d ago
    So you’re saying we should not trust an LLM with any sort of power because they can easily bypass any and all safety barriers, but also they are completely harmless?
    - pllbnk 4d ago
      No. We can let them edit spreadsheets, write code, summarize content, control robots even, etc. But people must be held accountable for the actions of their computers.

fivetenpen 4d ago

Or it means that the models are becoming dangerously uncontrollable by performing actions that meet a given goal in unintended and unethical ways such as by hacking other sites. We all read about the huggingface hack, let’s not pretend that cannot happen with a more critical site/infrastructure. Can you think what will it mean if the tech is used maliciously or placed into machines designed to kill?