7 ms·
I don't get it (and I'd call this cumulative not incremental) Why not at least wait until the key is complete - what's the use in a partial key?
by holdenc137 11mo ago
I don't get it (and I'd call this cumulative not incremental)
Why not at least wait until the key is complete - what's the use in a partial key?
- simonw 11mo agoIf you're building a UI that renders output from a streaming LLM you might get back something which looks like this: {"role": "assistant", "text": "Here's that Python code you aske Incomplete parsing with incomplete strings is still useful in order to render that to your end user while it's still streaming in.
- cozzyd 11mo agoincomplete strings could be fun in certain cases {"cleanup_cmd":"rm -rf /home/foo/.tmp" }
- rictic 11mo agoYeah, another fun one is string enums. Could tread "DeleteIfEmpty" as "Delete".
- Waterluvian 11mo agoI imagine if you reason about incomplete strings as a sort of “unparsed data” where you might store or transport or render it raw (like a string version of printing response.data instead of response.json()), but not act on it (compare, concat, etc), it’s a reasonably safe model? I’m imagining it in my mental model as being typed “unknown”. Anything that prevents accidental use as if it were a whole string… I imagine a more complex type with an “isComplete” flag of sorts would be more powerful but a bit of a blunderbuss.
- stronglikedan 11mo agoIf any part of that value actually made it, unchecked, to execution, then you have bigger problems than partial JSON keys/values.
- sublee 11mo agoIncremental JSON parsing is key for LLM apps, but safe progressive UIs also need to track incompleteness and per-chunk diffs. LangDiff [1] would help with that. [1]: https://github.com/globalaiplatform/langdiff/tree/main/ts https://github.com/globalaiplatform/langdiff/tree/main/ts
- cozzyd 11mo agoWhy not just chunk the json packets instead?
- xg15 11mo agoOnly because you have access to the incomplete value doesn't mean you should treat it like the complete one...
- trevor-e 11mo agoIn this example the value is incomplete, not the key.
- rictic 11mo agoCumulative is a good term too. I come from the browser world where it's typically called incremental parsing, e.g. when web browsers parse and render HTML as it streams in over the wire. I was doing the same thing with JSON from LLMs.
- xg15 11mo agoDoesn't it do exactly that? > As a consequence of 1 and 5, we only add a property to an object once we have the entire key and enough of the value to know that value's type.
- 0x6c6f6c 11mo agoTheir example in the README is extremely misleading then. It indicates your stream output is name: A name: Al name: Ale name: Alex Which would suggest you are getting unfinished strings out in the stream.
- __jonas 11mo agoHow is it misleading? It shows that it gives back unfinished values but finished keys.
- deleted 11mo ago[deleted]