10 ms·
I think the big secret is that AI is just software. In the same way that a financial firm doesn't all of sudden make a bunch of money because Microsoft shipped
by boh 5mo ago
I think the big secret is that AI is just software. In the same way that a financial firm doesn't all of sudden make a bunch of money because Microsoft shipped an update to Excel, AI is inert without intention. If there's any major successes in AI output it's because a person got it to do that. Claude Code is great, but it will also wipe out a database even though it's instructed not to (I can confirm from experience). The idea that there's some secret innovation that will come out any minute doesn't change the fact that it's software that requires human interaction to work.
- codingdave 5mo agoYes, and it has been said since day one of LLMs that all we need to do is keep things that way - no action without human intervention. Just like it was said that you should never grant AI direct access to change your production systems. But the stories of people who have done exactly that and had their systems damaged and deleted show that people aren't trying to even keep such basic safety nets in place. AI is getting strong enough that if people give some general direction as well as access to production systems of any kind, things can go badly. It is not true that all implementations of agentic AI requires human intervention for all action.
- stuaxo 5mo agoEh, these same people will attach openclaw to production systems soon and destroy their own companies.
- flats 5mo agoOne does not even need OpenClaw to achieve this outcome: https://x.com/lifeof_jer/status/2048103471019434248 https://x.com/lifeof_jer/status/2048103471019434248
- ffsm8 5mo agoYeeeehaaaaa, the vibes shall never end! On a more serious note, they were mostly f*cked by their paas provider imo. Claude will always do dumb shit. Especially if you tell it to not do something... By doing so you generally increase the likelihood of it doing it. It's even obvious why if you think about it, the pattern of "you had one job, but you failed" or "only this can't happen, it happened!" And all it's other forms is all over literature, online content etc. But their PaaS provider not scoping permissions properly is the root cause, all things considered. While Claude did cause this issue there, something else would've happened eventually otherwise.
- flats 5mo agoI absolutely agree with you. Also, some folks seem to be forgetting the virtues of boring, time-tested platforms & technologies in their rush to embrace the new & shiny & vibe-***ed. & also forgetting to thoroughly read documentation. It’s not terribly surprising to me that an “AI-first” infrastructure company might make these sorts of questionable design decisions.
- AndrewKemendo 5mo agoSounds like a pretty efficient self correcting mechanism I’m not sure what the problem is there
- ben_w 5mo agoNormalisation of deviance is the problem: https://en.wikipedia.org/wiki/Normalization_of_deviance https://en.wikipedia.org/wiki/Normalization_of_deviance Remember that these models are getting better; this means they get trusted with increasingly more important things by the time an error explodes in someone's face. It would be very bad if the thing which explodes is something you value which was handed off to an AI by someone who incorrectly thought it safe. AI companies which don't openly report that their AI can make mistakes are being dishonest, and that dishonesty would make this normalization of deviance even more prevelant than it already is.
- AndrewKemendo 5mo agoThat’s not a technical/AI problem in any sense, that’s a social problem in organizing and coordinating control structures Further, it’s only a problem to the extent that the downsides or risks are not accounted for which again… is a social problem not a technological problem This isn’t a problem for organizations that have well aligned incentives across their workflows A well organized company that has solid incentives is not going to diminish their own capacity by prematurely deploying a technology that is not capable of actually improving The issue is that 99% of the organizations that people deal with have entirely orthogonal incentives to them. They are then attributing the pain in dealing with that organization to the technology rather than the misaligned incentives
- ben_w 5mo ago> That’s not a technical/AI problem in any sense, that’s a social problem in organizing and coordinating control structures As @TeMPOraL here likes to point out, it can be genuinely fruitful to anthropomorphise AI. I only agree with partially, that this is true for *some* of the failure modes. > A well organized company that has solid incentives is not going to diminish their own capacity by prematurely deploying a technology that is not capable of actually improving Sure, but society as a whole doesn't have the right solid incentives to make sure that companies have the right solid incentives to do this. We can tell this quite easily by all the stupid things that get done. > The issue is that 99% of the organizations that people deal with have entirely orthogonal incentives to them. This is also fundamentally the AI alignment problem, that all AI are trained on some fitness function which is a proxy for what the trainer wanted, which is a proxy for what incentives their boss gave them, which is a proxy that repeats up to the owners in a capitalist society, which is a proxy for economic growth, which is a proxy for votes in a democracy, which is a proxy for good in a democracy.
- CamperBob2 5mo agoThe problem is, out of ten companies who take this approach, nine will indeed destroy themselves and one will end up with a trillion-dollar market cap. It will outcompete hundreds of companies who stuck with more conservative approaches. Everybody will want to emulate company #10, because "it obviously works." I don't see any stabilizing influences on the horizon, given how much cash is sloshing around in the economy looking for a place to land. Things are going to get weird, stupid, and chaotic, not necessarily in that order.
- jrflowers 5mo agoIf you had made a tool that gave gpt-3 the ability to run arbitrary commands on your production systems you could have seen things go badly.
- Terr_ 5mo agoMy cynical rule of thumb: By default we should imagine LLMs like javascript logic offloaded into a stranger's web-browser. The risks are similar: No prompts/data that go in can reliably be kept secret; A sufficiently-motivated stranger can have it send back completely arbitrary results; Some of those results may trigger very bad things depending on how you use or even just display them on your own end. P.S. This conceptual shortcut doesn't quite capture the dangers of poison data, which could sabotage all instances even when they happen to be hosted by honorable strangers.
- ctoth 5mo ago> doesn't change the fact that it's software that requires human interaction to work. Have you ever seen Claude Code launch a subagent? You've used it, right? You've seen it launch a subagent to do work? You understand that that is, in fact, Claude Code running itself, right?
- fnoef 5mo agoMy Linux server runs a cron job, that can spin off a thread and even use other ~apps~ tools. Did I invent AGI?
- ctoth 5mo agoDoes your Linux server decide what processes it should launch at what time with a theory of what will happen next in order to complete a goal you specified in natural language? If so yes, I reckon you sure have!
- balls187 5mo agoClaude does not have a "theory" of anything, and I'd argue applying that mental model to LLM+Tools is a major reason why Claude can delete a production database.
- Jtarii 5mo agoWell, humans also routinely accidentely delete production databases. I think at this point arguing that LLMs are just clueless automatons that have no idea what they are doing is a losing battle.
- californical 5mo agoI mean maybe it’s a losing battle today, but it is correct. So in a few years when the dust settles, we’ll probably all be using LLMs as clueless automatons that still do useful work as tools
- 5mo ago
- cyanydeez 5mo agoI think the market isn't for anyone but other businesses. We're all ants trying to understand how AI is going to eradicate the lower levels of society.
- dataviz1000 5mo agoLLM models are a distribution. Unlike a python script or turning machine, a LLM model is capable of generating any series of tokens. Developers need stop reasoning about LLM agents as deterministic and to start to think about agents in terms of Monte Carlo and Las Vegas algorithms. It isn't enough to have an agents, it also requires a cheap verifier. If I was a Ph.D. student today, I'd probably do a thesis on cheap verifiers for LLM agents. Since LLM agents are not reliable and therefore not very useful without it, that is a trillion dollar problem. Once a developer groks that concept, the agents stop being scary and the potential is large.
- add-sub-mul-div 5mo agoIf you told a programmer 30 years ago that someday we'd switch from a deterministic to nondeterministic paradigm for programming computers, they'd ask if we'd put lead back in the drinking water.
- dg247 5mo agoBeen doing this 30 years now. I am asking that question. Everyone talks around it.
- 52-6F-62 5mo agoYou aren't alone. Not even a few years ago if you introduced a component to a system that would result in non-deterministic output... Hell, a single function... You would be named and shamed for it because it went against every principle you should be learning as a novice writer of software. I have used the LLM tools, and I see the real-world potential for these things. But how it's all being sold and applied now: it's upside down.
- georgemcbay 5mo ago> they'd ask if we'd put lead back in the drinking water. With Lee Zeldin heading the EPA is anyone sure we won't?
- 5mo ago
- trolleski 5mo agoJust call the errors 'consciousness' and keep selling those tokens! Let the Spineless Generation have their last bubble!
- bellBivDinesh 5mo agoThe specter of AGI helps them obfuscate this
- cmdrk 5mo agoMy observation is that the true believers really don't want to think of models as an inert pile of weights. There's some mysticism attached to imagining it's the ship's computer from Star Trek, HAL-9000 or C-3PO. A file loaded into memory and executed over is just so... _pedestrian_.
- ben_w 5mo agoCanonically, the Star Trek computers have pretty much always been just computers, not themselves sentient because the software running on them just isn't. I'm still not sure if HAL-9000 was supposed to be conscious or just an interesting plot device with a persona as superficial as LLMs are dismissed as today. LLMs could definitely play the part of all three of your examples, given the flaws they showed on-screen. Could even do a decent approximation of Data (though perhaps not Lore without some jailbreaking). Still weird that even the best of them isn't really ready to be KITT.