9 ms·

Understanding Tool Calling in LLMs – Step-by-Step with REST and Spring AI

101 points by muthuishere 1y ago

upghost 1y ago
I think it's interesting and odd that tool calling took the form of this gnarly json blob. I much prefer the NexusRaven[1] style where you provide python function stubs with docstrings and get back python function invocations with the arguments populated. Of course I don't really understand why MCP is popular over REST or CLI, either. [1]: https://github.com/nexusflowai/NexusRaven-V2 https://github.com/nexusflowai/NexusRaven-V2
- max-privatevoid 1y ago
  The actual API call is still going to be JSON. How do you deal with that? Pack your Python function definitions into an array of huge opaque strings? And who would want to write a parser for that?
  - upghost 1y ago
    I'm sure you realize it gets reassembled into "huge opaque strings" when it is fed into the LLM as context. The arbitrary transport of the context as JSON is just a bit of protocol theater. You don't really have to parse the output, Python already has a parser in the form of the AST library[1]. But I get your drift. Depending on your workflow this could seem like more work. [1]: https://docs.python.org/3/library/ast.html#module-ast https://docs.python.org/3/library/ast.html#module-ast
    - max-privatevoid 1y ago
      The inference engine can do whatever it wants. This is already the case. The actual format of the tool call text varies by model, and the inference engine handles the translation from/to the JSON representation, that's the least of its concerns. What I don't want to happen is for some shitty webdev who writes an AI client in JavaScript to be forced to write a custom parser for some bespoke tool call language (call it "MLML", the Machine Learning Markup Language, to be superseded by YAMLML and then YAYAMLML, ...), or god forbid, somehow embed a WASM build of Python in their project to be able to `import ast`, instead of just parsing JSON and looking at the fields of the resulting object.
      - upghost 1y ago
        Yeah that's fair, I concede the point. I got a good snicker out of the YAYAMLMLOLOL :D Seems like it's tools calling tools all the way down heh
- deleted 1y ago
  [deleted]
nullorempty 1y ago
I don't think Spring is well regarded on HN.
- sorokod 1y ago
  "Just write this...." adds an annotation One of the many issues with Spring is that abstractions it provides are extremely leaky [1]. It leaks frequently and when it does, an engineer is faced with the need to comprehend a pile of technology[2] that was supposed to be abstracted away in the first place. - [1] https://en.wikipedia.org/wiki/Leaky_abstraction https://en.wikipedia.org/wiki/Leaky_abstraction - [2] https://github.com/spring-projects/spring-ai https://github.com/spring-projects/spring-ai
  - th0ma5 1y ago
    I think about this occasionally trying to rationalize it. I see similar patterns in other things like R and Julia where they design something in the environment to seem like a composable tool, and maybe it is but only within two or three specific compositions and then the way the environment is described sure seems to imply some kind of universality but it just doesn't work. Some even seem to keep patching every leak (maybe Spring means Spring a leak? Haha) and there's a sunk cost fallacy thing with an immense documentation page.
    - sorokod 1y ago
      There is similarity between Spring and "Buy now, pay later" schemes. You do often get a working feature quickly while having the price of evolving and maintaining that feature spread over some future. This is the best I can do for rationalizing Spring.
  - xienze 1y ago
    In what ways are the abstractions leaky? @Tool or @GetMapping make no demands on how to implement “this is a tool” or “this is a GET REST endpoint.” That they’re coupled with Spring (or rather, Spring is the only implementation for the semantics of these annotations) doesn’t constitute a leaky abstraction.
rapidaneurism 1y ago
How do you pass a user token to MCP calls? Do you hand the token to the LLM and expect it to fill an argument?
- theblazehen 1y ago
  Usually via environment variables in the MCP server definition, or a config file
- asabla 1y ago
  I know I'm a bit late. But for MCP servers running over HTTP/custom it should use OAuth 2.0. If it's served via stdout, it should use configuration files and/or environment variables. ref: https://modelcontextprotocol.io/specification/2025-03-26/basic/authorization https://modelcontextprotocol.io/specification/2025-03-26/bas...
- therealpygon 1y ago
  Easy. The LLM is never making MCP calls. The LLM simply identifies an endpoint it thinks would be useful and provides the required request parameters (like the text to be searched for or processed). As far as an LLM is concerned, MCP calls are handled “client-side” (from its perspective). This is why you configure MCP servers in your client and not on the server. (Yes, some providers allow you to configure MCP servers, but that is just a layer between you and the LLM and not a feature of the LLM itself. So back to the credentials, that means that the credentials are managed “client-side” and the LLM never needs to see any of that. Think of it like this, say you set up an MCP url (my-mcp.com); the LLM knows nothing of this url, or what MCP server you use. So if instead you called my-mcp.com/<some-long-string>/, the LLM still doesn’t know. Now, instead of a URL parameter, your tool calls the MCP with a header (Bearer: <token>), the LLM still doesn’t know and you’ve accessed an OAUTH endpoint.
_moog 1y ago
I recently started diving into LLMs a few weeks ago, and one thing that immediately caught me off guard was how little standardization there is across all the various pieces you would use to build a chat stack. Want to swap out your client for a different one? Good luck - it probably expects a completely different schema. Trying a new model? Hope you're ready to deal with a different chat template. It felt like every layer had its own way of doing things, which made understanding the flow pretty frustrating for a noobie. So I sketched out a diagram that maps out what (rough) schema is being used at each step of the process - from the initial request all the way through Ollama and an MCP server with OpenAI-compatible endpoints showing what transformations occur where. Figured I'd share it as it may help someone else. https://moog.sh/posts/openai_ollama_mcp_flow.html https://moog.sh/posts/openai_ollama_mcp_flow.html Somewhat ironically, Claude built the JS hooks for my SVG with about five minutes of prompting.
- youdont 1y ago
  Have you tried BAML? We use it to manage APIs and clients, as well as prompts and types. It gives great low level control over your prompts and logic, but acts as a nice standardisation later.
  - _moog 1y ago
    That's going to be super useful for some of the high-level prompt-testing work I'm doing. Thanks! I'm also getting more into the lower-level LLM fine-tuning, training on custom chat templates, etc. which is more of where the diagram was needed.
  - redhale 1y ago
    +1 for BAML. I find that the "prompts as typed functions" concept really simplifies the mental model here, making LLM apps easier to reason about.
- nimchimpsky 1y ago
  [dead]
- 1dom 1y ago

101 points by muthuishere 1y ago

I think it's interesting and odd that tool calling took the form of this gnarly json blob. I much prefer the NexusRaven[1] style where you provide python function stubs with docstrings and get back python function invocations with the arguments populated. Of course I don't really understand why MCP is popular over REST or CLI, either. [1]: https://github.com/nexusflowai/NexusRaven-V2 https://github.com/nexusflowai/NexusRaven-V2

max-privatevoid 1y ago
The actual API call is still going to be JSON. How do you deal with that? Pack your Python function definitions into an array of huge opaque strings? And who would want to write a parser for that?
- upghost 1y ago
  I'm sure you realize it gets reassembled into "huge opaque strings" when it is fed into the LLM as context. The arbitrary transport of the context as JSON is just a bit of protocol theater. You don't really have to parse the output, Python already has a parser in the form of the AST library[1]. But I get your drift. Depending on your workflow this could seem like more work. [1]: https://docs.python.org/3/library/ast.html#module-ast https://docs.python.org/3/library/ast.html#module-ast
  - max-privatevoid 1y ago
    The inference engine can do whatever it wants. This is already the case. The actual format of the tool call text varies by model, and the inference engine handles the translation from/to the JSON representation, that's the least of its concerns. What I don't want to happen is for some shitty webdev who writes an AI client in JavaScript to be forced to write a custom parser for some bespoke tool call language (call it "MLML", the Machine Learning Markup Language, to be superseded by YAMLML and then YAYAMLML, ...), or god forbid, somehow embed a WASM build of Python in their project to be able to `import ast`, instead of just parsing JSON and looking at the fields of the resulting object.
    - upghost 1y ago
      Yeah that's fair, I concede the point. I got a good snicker out of the YAYAMLMLOLOL :D Seems like it's tools calling tools all the way down heh
deleted 1y ago
[deleted]

nullorempty 1y ago

I don't think Spring is well regarded on HN.

sorokod 1y ago
"Just write this...." adds an annotation One of the many issues with Spring is that abstractions it provides are extremely leaky [1]. It leaks frequently and when it does, an engineer is faced with the need to comprehend a pile of technology[2] that was supposed to be abstracted away in the first place. - [1] https://en.wikipedia.org/wiki/Leaky_abstraction https://en.wikipedia.org/wiki/Leaky_abstraction - [2] https://github.com/spring-projects/spring-ai https://github.com/spring-projects/spring-ai
- th0ma5 1y ago
  I think about this occasionally trying to rationalize it. I see similar patterns in other things like R and Julia where they design something in the environment to seem like a composable tool, and maybe it is but only within two or three specific compositions and then the way the environment is described sure seems to imply some kind of universality but it just doesn't work. Some even seem to keep patching every leak (maybe Spring means Spring a leak? Haha) and there's a sunk cost fallacy thing with an immense documentation page.
  - sorokod 1y ago
    There is similarity between Spring and "Buy now, pay later" schemes. You do often get a working feature quickly while having the price of evolving and maintaining that feature spread over some future. This is the best I can do for rationalizing Spring.
- xienze 1y ago
  In what ways are the abstractions leaky? @Tool or @GetMapping make no demands on how to implement “this is a tool” or “this is a GET REST endpoint.” That they’re coupled with Spring (or rather, Spring is the only implementation for the semantics of these annotations) doesn’t constitute a leaky abstraction.

rapidaneurism 1y ago

How do you pass a user token to MCP calls? Do you hand the token to the LLM and expect it to fill an argument?

theblazehen 1y ago
Usually via environment variables in the MCP server definition, or a config file
asabla 1y ago
I know I'm a bit late. But for MCP servers running over HTTP/custom it should use OAuth 2.0. If it's served via stdout, it should use configuration files and/or environment variables. ref: https://modelcontextprotocol.io/specification/2025-03-26/basic/authorization https://modelcontextprotocol.io/specification/2025-03-26/bas...
therealpygon 1y ago
Easy. The LLM is never making MCP calls. The LLM simply identifies an endpoint it thinks would be useful and provides the required request parameters (like the text to be searched for or processed). As far as an LLM is concerned, MCP calls are handled “client-side” (from its perspective). This is why you configure MCP servers in your client and not on the server. (Yes, some providers allow you to configure MCP servers, but that is just a layer between you and the LLM and not a feature of the LLM itself. So back to the credentials, that means that the credentials are managed “client-side” and the LLM never needs to see any of that. Think of it like this, say you set up an MCP url (my-mcp.com); the LLM knows nothing of this url, or what MCP server you use. So if instead you called my-mcp.com/<some-long-string>/, the LLM still doesn’t know. Now, instead of a URL parameter, your tool calls the MCP with a header (Bearer: <token>), the LLM still doesn’t know and you’ve accessed an OAUTH endpoint.

_moog 1y ago

I recently started diving into LLMs a few weeks ago, and one thing that immediately caught me off guard was how little standardization there is across all the various pieces you would use to build a chat stack. Want to swap out your client for a different one? Good luck - it probably expects a completely different schema. Trying a new model? Hope you're ready to deal with a different chat template. It felt like every layer had its own way of doing things, which made understanding the flow pretty frustrating for a noobie. So I sketched out a diagram that maps out what (rough) schema is being used at each step of the process - from the initial request all the way through Ollama and an MCP server with OpenAI-compatible endpoints showing what transformations occur where. Figured I'd share it as it may help someone else. https://moog.sh/posts/openai_ollama_mcp_flow.html https://moog.sh/posts/openai_ollama_mcp_flow.html Somewhat ironically, Claude built the JS hooks for my SVG with about five minutes of prompting.

youdont 1y ago
Have you tried BAML? We use it to manage APIs and clients, as well as prompts and types. It gives great low level control over your prompts and logic, but acts as a nice standardisation later.
- _moog 1y ago
  That's going to be super useful for some of the high-level prompt-testing work I'm doing. Thanks! I'm also getting more into the lower-level LLM fine-tuning, training on custom chat templates, etc. which is more of where the diagram was needed.
- redhale 1y ago
  +1 for BAML. I find that the "prompts as typed functions" concept really simplifies the mental model here, making LLM apps easier to reason about.
nimchimpsky 1y ago
[dead]
1dom 1y ago