6 ms·
I like to dunk on Meta as much as the next guy, but I think this makes sense: deterministic verification like this is not, and should never be, the LLM’s job. T
by ludwik 3mo ago
I like to dunk on Meta as much as the next guy, but I think this makes sense: deterministic verification like this is not, and should never be, the LLM’s job. The tools it has access to should enforce the permissions layer, ensuring that the LLM can never perform actions the user themselves should not be allowed to perform. In this case, the tool failed to do that.
- jffyjcaa 3mo ago[dead]
- TZubiri 3mo ago>deterministic verification like this is not, and should never be, the LLM’s job. But when humans handled it, this was not as much as a problem. That is, the humans did the job, because they recognized the need to do that job. Sure sometimes accounts could get recovered if a human was tricked, but evidently it was easier to trick the LLM in masse than humans.
- ajross 3mo ago> But when humans handled it, this was not as much as a problem. In fact it's arguably a feature. The ability of support staff to short-circuit nitpicky rules when there's an obvious external validation happening (e.g. you're on the phone with a user who's presenting ID in real time and correlating it with previous use of the account, etc...) makes for better data quality and happier customers. Obviously, yes, you can then human-engineer an authentication breach. But that was very difficult, because people are "common-sense careful" in a way we haven't been able to tease out of AI yet.
- ludwik 3mo agoMaybe that’s because I work with agentic AI in my day job, but this seems utterly obvious to me: no reasonable person would ever claim that LLMs are better at keeping secrets or enforcing rules than human employees. This notice is not about comparing humans and LLMs. It seems that the system was designed in the only reasonable way: with a deterministic permissions layer separate from the agent. But that layer failed to work properly. So the notice is comparing the difference between how the system was supposed to work and how it actually worked in reality. Normal post-mortem stuff.
- gavmor 3mo agoThe overall system that allowed this implementation is accountable. So why put such a fine point on it so as to exculpate the LLM?
- im3w1l 3mo agoIt helps set expectations for the fix. "The bug was in an external system that has now been fixed" means we it's probably fine going forward. "The LLM got tricked but we are gonna train it super hard not to do that again" means it will break again and again as people find new angles to convince it.
- dbbk 3mo agoYes the LLM part is irrelevant here. It'd be just the same if it was a HTML form.