7 ms·

I thought this article was going to be a bunch of security theater nonsense - maybe the relatively bland title - but after reading I found it to be incredibly i

• throwawaymaths 1y ago
  i mean isnt all this stuff up to the mcp author to return a reasonable error to the agent and ask for it to repeat the call with amendments to the json?
  • dotancohen 1y ago
    Yes. And this is where culture comes in. The culture of discipline of the C++ and the JavaScript communities are at extreme odds of the spectrum. The concern here is that the culture of interfacing with AI tools, such as MCP, is far closer to the discipline of the JavaScript community than to the C++ community.
    • fidotron 1y ago
      The fundamental difference is the JS community believe in finding the happy path that results in something they can sell before they have filled in all those annoying problem areas around it. If an LLM can be shown to be useful 80% of the time to the JS mindset this is fine, and the remaining 20% can be resolved once we're being paid for the rest, Pareto principle be damned.
  • nativeit 1y ago
    What's your point? It's up to a ship's captain to keep it afloat, doesn't mean the hundreds of holes in the new ship's hull aren't relevant.
  • stouset 1y ago
    This is no different than the argument that C is totally great as long as you just don’t make mistakes with pointers or memory management or indexing arrays. At some point we have to decide as a community of engineers that we have to stop building tools that are little more than loaded shotguns pointed at our own feet.
    • andersa 1y ago
      It's clearly a much better design if the shotguns are pointed at someone else's feet.
    • throwawaymaths 1y ago
      no, it's not because the nature of llms mean that even if you fully validate your communications with the llm statistically anything can happen, so any usage/threat model must already take nasal demons into account.
      • stouset 1y ago
        Adding new and innovative layers of unsafety on top doesn’t invalidate the need for safety in lower layers. If anything it makes it more critical.
        • throwawaymaths 1y ago
          you're missing the point. the point is, there is no need to make two layers out of one (i could be persuaded that it makes the coding easier)
  • cwilkes 1y ago
    This implies that the input process did a check when it imported the data from somewhere else. GIEMGO garbage in even more garbage out
  • dragonwriter 1y ago
    > i mean isnt all this stuff up to the mcp author Mostly, no. Whether its the client sending (statically) bad data or the server returning (statically) bad data, schema validation on the other end (assuming somehow it is allowed by the toolchain on the sending end) should reject it before it gets to the custom code of the MCP server or MCP client. For arguments that are the right type but wrong because of the state of the universe, yes, the server receiving it should send a useful error message back to the client. But that's a different issue.
• tomrod 1y ago
  We already have PEBKAC - problem exists between chair and keyboard. LLMs are basically automating PEBKAC
• cle 1y ago
  I don't understand this criticism by the author. MCP supports JSON Schema, and server responses must conform to the schema. If the schema requires an ISO-8601 timestamp (ex by specifying a "date" format in the schema) but the server sends a Unix epoch timestamp, then it is violating the protocol. The author even later says that MCP supports JSON Schema, but also claims "you can't generate type-safe clients". Which is plainly untrue, there exist plenty of JSON Schema code generators.
  • ohdeargodno 1y ago
    Except that any properly written software will respond to protocol and schema violations by throwing an error. Claude will happily cast your int into a 2023 Toyota Yaris and keep on hallucinating things.
    • starkparker 1y ago
      For the curious: > Cast an integer into the type of a 2023 Toyota Yaris using Javascript (GPT-4o mini) > To cast an integer into the type of a 2023 Toyota Yaris in JavaScript, you would typically create a class or a constructor function that represents the Toyota Yaris. Then, you can create an instance of that class using the integer value. Here's an example of how you might do this: // Define a class for the 2023 Toyota Yaris class ToyotaYaris { constructor(modelYear, value) { this.modelYear = modelYear; this.value = value; } displayInfo() { console.log(`Model Year: ${this.modelYear}, Value: ${this.value}`); } } // Function to cast an integer into a Toyota Yaris object function castToYaris(integerValue) { const modelYear = 2023; // Set the model year return new ToyotaYaris(modelYear, integerValue); } // Example usage const integerValue = 20000; // Example integer value const yaris = castToYaris(integerValue); yaris.displayInfo(); // Output: Model Year: 2023, Value: $20000
      • mrits 1y ago
        Billy, it's becoming increasingly harder to believe you are writing this code yourself
      • 1y ago
• cookiengineer 1y ago
  Let's put it this way: Before 2023 I always thought that all the bugs and glitches of technology in Star Trek were totally made up and would never happen this way. Post-LLM I am absolutely certain that they will happen exactly that way. I am not sure what LLM integrations have to do with engineering anymore, or why it makes sense to essentially put all your company's infrastructure into external control. And that is not even scratching the surface with the lack of reproducibility at every single step of the way. It "somehow works" isn't engineering.
  • withinboredom 1y ago
    For someone who isn't a trek fan -- can you elaborate on this?
    • whoknowsidont 1y ago
      The computer, at least aboard the enterprise, is kind of portrayed as a singular monolithic AI that can access the majority of the ship's subsystems (different networks, other computer/control units, etc) and functions. It can control nearly every aspect of the ship while talking with its human crew / commanding officers. So very much like an LLM accessing multiple pieces of functionality across different tools and API endpoints (if you want to imagine it that way). While it is seemingly very knowledgeable, it is rather stupid. It gets duped by nefarious actors or has a class of bugs that are elementary that put the crew into awkward positions. Most professional software engineers might have previously looked as these scenarios as implausible, given the "failure model" of current software is quite blunt, and especially given how far into the future the series took place. Now we see that computational tasks are becoming less predictable, less straight-forward, with cascading failures instead of blunt, direct failures. Interacting with an LLM might be compared to talking with a person in psychosis when it starts to hallucinate. So you get things like this in the Star Trek universe: https://www.youtube.com/watch?v=kUJh7id0lK4 https://www.youtube.com/watch?v=kUJh7id0lK4 Which make a lot more sense, become a lot more plausible and a lot more relatable with our current implementations of AI/LLM's.
      • 1y ago
• hinkley 1y ago
  > In healthcare, patient data types get coerced incorrectly, potentially leading to wrong medication dosing recommendations. May have changed, but unlikely. I worked with medical telemetry as a young man and it was impressed upon me thoroughly how important parsing timestamps correctly was. I have a faint memory, possibly false, of this being the first time I wrote unit tests (and without the benefit of a test framework). We even accounted for lack of NTP by recalculating times off of the timestamps I. Their message headers. And the reasons I was given were incident review as well as malpractice cases. A drug administered three seconds before a heart attack starts is a very different situation than one administered eight seconds after the patient crashed. We saw recently with the British postal service how lives can be ruined by bad data, and in medical data a minute is a world of difference.
  • deathanatos 1y ago
    > May have changed, but unlikely. I worked with medical telemetry as a young man and it was impressed upon me thoroughly how important parsing timestamps correctly was. I also work in healthcare, and we've seen HL7v2 messages with impossible timestamps. (E.g., in the spring-forward gap.)
    • hinkley 1y ago
      Since we were getting low latency data inside HTTP responses we could work off of the response header clock skew to narrow origination time down to around one second, and that’s almost as good as NTP can manage anyway. As RPC mechanisms go, HTTP is notable for how few of the classic blunders they made in 1.0 of the spec. Clock skew correction is just my favorite. Technically it exists for cache directives, but it’s invaluable for coordination across machines. There are reasons HTTP 2.0 waited decades to happen. It just mostly worked.
• avereveard 1y ago
  MCP focuses on transport and managing context and doesn't absolve the user for sensibly implementing the interface (i.e. defining a schema and doing schema validation) this is like saying "HTTP doesn't do json validation", which, well, yeah.
• oblio 1y ago
  We keep repeating this. When desktop OSes came out, hardware resources were scarce so all the desktop OSes (DOS, Windows, MacOS) forgot all the lessons from Unix: multi user, cooperative multitasking, etc. 10 years later PC hardware was faster than workstations from the 90s yet we're still stuck with OSes riddled with limitations that stopped making sense in the 80s. When smartphones came out there was this gold rush and hardware resources were scarce so OSes (iOS, Android) again forgot all the lessons. 10 years later mobile hardware was faster than desktop hardware from the 00s. We're still stuck with mistakes from the 00s. AI basically does the same thing. It's all lead by very bright 20 and 30 year olds that weren't even born when Windows was first released. Our field is doomed under a Cascade of Attention-Deficit Teenagers: https://www.jwz.org/doc/cadt.html https://www.jwz.org/doc/cadt.html (copy paste the link). It's all gold rushes and nobody does Dutch urban infrastructure design over decades. Which makes sense as this is all driven by the US, where long term plan I is anathema.
  • lovich 1y ago
    Our economic system punishes you for being born later, unless you manage to flip the table in terms of the status quo in the economy. Of course this keeps happening
• lowbloodsugar 1y ago
  I’ve been successfully using AI for several months now, and there’s still no way I’d trust it to execute trades, or set the dose on an XRay machine. But startups gonna start. Let them.
• jongjong 1y ago
  To me, the article was just rambling about all sorts of made up issues which only exist in the minds of people who never spent any time outside of corporate environments... A lot of 'preventative' ideas which make sense in some contexts but are mis-applied in different contexts. The stuff about type validation is incorrect. You don't need client-side validation. You shouldn't be using APIs you don't trust as tools and you can always add instructions about the LLM's output format to convert to different formats. MCP is not the issue. The issue is that people are using the wrong tools or their prompts are bad. If you don't like the format of an MCP tool and don't want to give formatting instructions the LLMs, you can always create your own MCP service which outputs data in the correct format. You don't need the coercion to happen on the client side.
• Squakie 1y ago
  I can offer a hacking/penetrarion testing perspective to this as a security researcher at a security consultint firm: this type of hallucination and trust is one of the largest things we exploit in our new LLM testing service. Overly agentic systems (one of the top 10 OWASP LLM vulns) is the most profound and commonly exploited issue that we've been able to leverage. If we can get an internal, sensitive-data-handling agent to ingest a crafted prompt, either via direct prompt injection against a more abstract “parent” agent, or by tainting an input file/URL it’s told to process, we can plant what I have internally coined an “unfolding injection.” The injection works like a parasitic goal, it doesn’t just trick one agent, it rewrites the downstream intent. As the orchestrator routes tasks to other agents, each one treats the tainted instructions as legitimate and works toward fulfilling them. Because many orchestrations re-summarize, re-plan, or synthesize goals between steps, the malicious instructions can actually gain fidelity as they propagate. By the time they reach a sensitive action (exfiltration, privilege escalation, external calls), there’s no trace of the original “weird” wording, just a confidently stated, fully-integrated sub-goal. It’s essentially a supply-chain attack on the orchestration layer: you compromise one node in the agent network, and the rest “help” you without realizing it. Without explicit provenance tracking and policy enforcement between agents, this kind of unfolding injection is almost trivial to pull off, and we've been able to compromise entire environments based on the information the agentic system provided us, or just gave us either a bind or reverse shell in the case it has cli access and ability to figure out its own network constraints. SSRF has been making a HUGE return in agentic systems, and Im sad defcon and black hat didnt really have many talks on this subject this year, because it is a currently evolving security domain and entirely new method of exploitation. The entire point of agentic systems is non determinism, but it also makes it a security nightmare. As a researcher though, this is basically a gold mine of all sorts of new vulnerabilities we'll be seeing. If you work as a bugbounty hunter and see a new listing for an AI company I can almost assuredly say you can get a pretty massive payout just by exploiting the innate trust between agents and the internal tools they are leveraging. Even if you dont have the architecture docs of the agentic system you can likely prompt inject the initial task enough to taint the further agents to have them list out the orchestration flow by creatively adjusting your prompt for different types of orchestration and how the company might be doing prompt engineering on the agents persona and task its designed to work on and then submit report on to parent agent, and the limited input validation between them.
• pstoll 1y ago
  > I can practically see it playing out now: there is some huge incident of some kind, in some system or service with an MCP component somewhere, with some elaborate post-mortem revealing that some MCP server somewhere screwed up Already happening. https://www.infosecurity-magazine.com/news/atlassian-ai-agent-mcp-attack/ https://www.infosecurity-magazine.com/news/atlassian-ai-agen...
• beefnugs 1y ago
  Real security is nonsense to vibe coders