8 ms·

I don't understand this criticism by the author. MCP supports JSON Schema, and server responses must conform to the schema. If the schema requires an ISO-8601 t

by cle 1y ago

I don't understand this criticism by the author. MCP supports JSON Schema, and server responses must conform to the schema. If the schema requires an ISO-8601 timestamp (ex by specifying a "date" format in the schema) but the server sends a Unix epoch timestamp, then it is violating the protocol. The author even later says that MCP supports JSON Schema, but also claims "you can't generate type-safe clients". Which is plainly untrue, there exist plenty of JSON Schema code generators.

ohdeargodno 1y ago
Except that any properly written software will respond to protocol and schema violations by throwing an error. Claude will happily cast your int into a 2023 Toyota Yaris and keep on hallucinating things.
- starkparker 1y ago
  For the curious: > Cast an integer into the type of a 2023 Toyota Yaris using Javascript (GPT-4o mini) > To cast an integer into the type of a 2023 Toyota Yaris in JavaScript, you would typically create a class or a constructor function that represents the Toyota Yaris. Then, you can create an instance of that class using the integer value. Here's an example of how you might do this: // Define a class for the 2023 Toyota Yaris class ToyotaYaris { constructor(modelYear, value) { this.modelYear = modelYear; this.value = value; } displayInfo() { console.log(`Model Year: ${this.modelYear}, Value: ${this.value}`); } } // Function to cast an integer into a Toyota Yaris object function castToYaris(integerValue) { const modelYear = 2023; // Set the model year return new ToyotaYaris(modelYear, integerValue); } // Example usage const integerValue = 20000; // Example integer value const yaris = castToYaris(integerValue); yaris.displayInfo(); // Output: Model Year: 2023, Value: $20000
  - mrits 1y ago
    Billy, it's becoming increasingly harder to believe you are writing this code yourself
  - tempaccount420 1y ago
    You really tried to inflict the most amount of damage to the reader by choosing GPT-4o mini
- cle 1y ago
  I just tried this in Claude Code. I made an MCP server whose tool output is declared as an integer but it returns a string at runtime. Claude Code validated the response against the schema and did not pass the response to the LLM. test - test_tool (MCP)(input: "foo") ⎿ Error: Output validation error: 'bar' is not of type 'integer'
  - ohdeargodno 1y ago
    This time. Can you guarantee it will validate it every time ? Can you guarantee the way MCPs/tool calling are implemented (which is already an incredible joke that only python brained developers would inflict upon the world) will always go through the validation layer, are you even sure of what part of Claude handles this validation ? Sure, it didn't cast an int into a Toyota Yaris. Will it cast "70Y074" into one ? Maybe a 2022 one. What if there are embedded parsing rules into a string, will it respect it every time ? What if you use it outside of Claude Code, but just ask nicely through the API, can you guarantee this validation still works ? Or that they won't break it next week ? The whole point of it is, whichever LLM you're using is already too dumb to not trip when lacing its own shoes. Why you'd trust it to reliably and properly parse input badly described by a terrible format is beyond me.
    - cle 1y ago
      This is deterministic, it is validating the response using a JSON Schema validator and refusing to pass it to an LLM inference. I can't gaurantee that behavior will remain the same more than any other software. But all this happens before the LLM is even involved. > The whole point of it is, whichever LLM you're using is already too dumb to not trip when lacing its own shoes. Why you'd trust it to reliably and properly parse input badly described by a terrible format is beyond me. You are describing why MCP supports JSON Schema. It requires parsing & validating the input using deterministic software, not LLMs.
      - whoknowsidont 1y ago
        >This is deterministic, it is validating the response using a JSON Schema validator and refusing to pass it to an LLM inference. No. It is not. You are still misunderstanding how this works. It is "choosing" to pass this to a validator or some other tool, _for now_. As a matter of pure statistics, it will simply not do this at some point in the future on some run. It is inevitable.
- Groxx 1y ago
  This can even be a feature: https://github.com/Zorokee/ArtificialCast https://github.com/Zorokee/ArtificialCast
dboreham 1y ago
imho it's a fantasy to expect type safe protocols except in the case that both client and server are written in the same (type safe) language. Actually even that doesn't work. What language actually allows a type definition for "ISO-8601 timestamp" that's complete? Everything ends up being some construction of strings and numbers, and it's often not possible to completely describe the set of valid values except by run-time checking, certainly beyond trivial cases like "integer between 0 and 10".
- cle 1y ago
  Generally you'd use a time library to model ISO-8601 dates in a typesafe way. Some fancier languages might have syntactic support for it, but they ultimately serve the same purpose. Related but distinct from serialization.
- mgh95 1y ago
  > What language actually allows a type definition for "ISO-8601 timestamp" that's complete? It is absolutely possible to do this, and to generate client code which complies with ISO-8601 in JS/TS. Large amounts of financial services would not work if this was not the case. See the c# support for ISO-8601 strings: https://learn.microsoft.com/en-us/dotnet/standard/base-types/standard-date-and-time-format-strings https://learn.microsoft.com/en-us/dotnet/standard/base-types...
  - whoknowsidont 1y ago
    You've misunderstood his statement and proven his point. `DateTime` is not an ISO-8601 type. It can _parse_ an ISO-8601 formatted string. And even past that, there are Windows-specific idiosyncrasies with how the `DateTime` class implements the parsing of these strings and how it stores the resulting value.
    - mgh95 1y ago
      > `DateTime` is not an ISO-8601 type. It can _parse_ an ISO-8601 formatted string. This is exactly the point: a string is just a data interchange format in the context of a DateTime, and C# provides (as far as I can tell) a complete way of accessing the ISO-8601 specification on the language object. It also supports type-safe generation of clients and client object (or struct) generation from the ISO-8601 string format. > And even past that, there are Windows-specific idiosyncrasies with how the `DateTime` class implements the parsing of these strings and how it stores the resulting value. Not really. The windows statements on the article (and I use this on linux for financial services software) are related to automated settings of the preferences for generated strings. All of these may be set within the code itself.
jongjong 1y ago
At its core, the article was just ramblings from someone being upset that LLMs didn't make things more complicated so that they could charge more billable hours to solve invented corporate problems... Which some people built their career on. The merchants of complexity are disappointed. It turns out that even machines don't care for 'machine-readable' formats; even the machines prefer human-readable formats. The only entities on this planet who appreciate so-called 'machine-readability' are bureaucrats; and they like it for the same reason that they like enterprise acronyms... Literally the opposite of readability.
- nxobject 1y ago
  I look forward to waiting a decade and seeing what MCP ends up reinventing.

by cle 1y ago

ohdeargodno 1y ago

Except that any properly written software will respond to protocol and schema violations by throwing an error. Claude will happily cast your int into a 2023 Toyota Yaris and keep on hallucinating things.

starkparker 1y ago
For the curious: > Cast an integer into the type of a 2023 Toyota Yaris using Javascript (GPT-4o mini) > To cast an integer into the type of a 2023 Toyota Yaris in JavaScript, you would typically create a class or a constructor function that represents the Toyota Yaris. Then, you can create an instance of that class using the integer value. Here's an example of how you might do this: // Define a class for the 2023 Toyota Yaris class ToyotaYaris { constructor(modelYear, value) { this.modelYear = modelYear; this.value = value; } displayInfo() { console.log(`Model Year: ${this.modelYear}, Value: ${this.value}`); } } // Function to cast an integer into a Toyota Yaris object function castToYaris(integerValue) { const modelYear = 2023; // Set the model year return new ToyotaYaris(modelYear, integerValue); } // Example usage const integerValue = 20000; // Example integer value const yaris = castToYaris(integerValue); yaris.displayInfo(); // Output: Model Year: 2023, Value: $20000
- mrits 1y ago
  Billy, it's becoming increasingly harder to believe you are writing this code yourself
- tempaccount420 1y ago
  You really tried to inflict the most amount of damage to the reader by choosing GPT-4o mini
cle 1y ago
I just tried this in Claude Code. I made an MCP server whose tool output is declared as an integer but it returns a string at runtime. Claude Code validated the response against the schema and did not pass the response to the LLM. test - test_tool (MCP)(input: "foo") ⎿ Error: Output validation error: 'bar' is not of type 'integer'
- ohdeargodno 1y ago
  This time. Can you guarantee it will validate it every time ? Can you guarantee the way MCPs/tool calling are implemented (which is already an incredible joke that only python brained developers would inflict upon the world) will always go through the validation layer, are you even sure of what part of Claude handles this validation ? Sure, it didn't cast an int into a Toyota Yaris. Will it cast "70Y074" into one ? Maybe a 2022 one. What if there are embedded parsing rules into a string, will it respect it every time ? What if you use it outside of Claude Code, but just ask nicely through the API, can you guarantee this validation still works ? Or that they won't break it next week ? The whole point of it is, whichever LLM you're using is already too dumb to not trip when lacing its own shoes. Why you'd trust it to reliably and properly parse input badly described by a terrible format is beyond me.
  - cle 1y ago
    This is deterministic, it is validating the response using a JSON Schema validator and refusing to pass it to an LLM inference. I can't gaurantee that behavior will remain the same more than any other software. But all this happens before the LLM is even involved. > The whole point of it is, whichever LLM you're using is already too dumb to not trip when lacing its own shoes. Why you'd trust it to reliably and properly parse input badly described by a terrible format is beyond me. You are describing why MCP supports JSON Schema. It requires parsing & validating the input using deterministic software, not LLMs.
    - whoknowsidont 1y ago
      >This is deterministic, it is validating the response using a JSON Schema validator and refusing to pass it to an LLM inference. No. It is not. You are still misunderstanding how this works. It is "choosing" to pass this to a validator or some other tool, _for now_. As a matter of pure statistics, it will simply not do this at some point in the future on some run. It is inevitable.
Groxx 1y ago
This can even be a feature: https://github.com/Zorokee/ArtificialCast https://github.com/Zorokee/ArtificialCast

dboreham 1y ago

imho it's a fantasy to expect type safe protocols except in the case that both client and server are written in the same (type safe) language. Actually even that doesn't work. What language actually allows a type definition for "ISO-8601 timestamp" that's complete? Everything ends up being some construction of strings and numbers, and it's often not possible to completely describe the set of valid values except by run-time checking, certainly beyond trivial cases like "integer between 0 and 10".

cle 1y ago
Generally you'd use a time library to model ISO-8601 dates in a typesafe way. Some fancier languages might have syntactic support for it, but they ultimately serve the same purpose. Related but distinct from serialization.
mgh95 1y ago
> What language actually allows a type definition for "ISO-8601 timestamp" that's complete? It is absolutely possible to do this, and to generate client code which complies with ISO-8601 in JS/TS. Large amounts of financial services would not work if this was not the case. See the c# support for ISO-8601 strings: https://learn.microsoft.com/en-us/dotnet/standard/base-types/standard-date-and-time-format-strings https://learn.microsoft.com/en-us/dotnet/standard/base-types...
- whoknowsidont 1y ago
  You've misunderstood his statement and proven his point. `DateTime` is not an ISO-8601 type. It can _parse_ an ISO-8601 formatted string. And even past that, there are Windows-specific idiosyncrasies with how the `DateTime` class implements the parsing of these strings and how it stores the resulting value.
  - mgh95 1y ago
    > `DateTime` is not an ISO-8601 type. It can _parse_ an ISO-8601 formatted string. This is exactly the point: a string is just a data interchange format in the context of a DateTime, and C# provides (as far as I can tell) a complete way of accessing the ISO-8601 specification on the language object. It also supports type-safe generation of clients and client object (or struct) generation from the ISO-8601 string format. > And even past that, there are Windows-specific idiosyncrasies with how the `DateTime` class implements the parsing of these strings and how it stores the resulting value. Not really. The windows statements on the article (and I use this on linux for financial services software) are related to automated settings of the preferences for generated strings. All of these may be set within the code itself.

jongjong 1y ago

At its core, the article was just ramblings from someone being upset that LLMs didn't make things more complicated so that they could charge more billable hours to solve invented corporate problems... Which some people built their career on. The merchants of complexity are disappointed. It turns out that even machines don't care for 'machine-readable' formats; even the machines prefer human-readable formats. The only entities on this planet who appreciate so-called 'machine-readability' are bureaucrats; and they like it for the same reason that they like enterprise acronyms... Literally the opposite of readability.

nxobject 1y ago
I look forward to waiting a decade and seeing what MCP ends up reinventing.