6 ms·

What if the converted version is not in the wanted syntax?

by cyptus 1y ago

NitpickLawyer 1y ago
Constrained generation guarantees syntax. It does not guarantee semantic correctness tho. Imagine you want a json object with "hp" and "damage". If you use a grammar, the model will be forced to output a json object with those two values. But it's not guaranteed to get sensible values. With a 2nd pass you basically "condition" it on the text right above, hoping to get better semantic understanding.
- lyu07282 1y ago
  I'm pretty sure the grammar is generated from the Json schema, it doesn't just constrain json syntax, it constraints on the schema (including enums and such). The schema is also given to the model (at least in openai) you can put instructions in the json schema as well that will be taken into account.
  - NitpickLawyer 1y ago
    Perhaps I worded that poorly. What I mean by semantic correctness is that the model could output nonsensical values for some things. Say in a game, "normal" health is ~100hp and the model creates a wizard with 50hp but then a mouse with 10000hp. So you're guaranteed to get a parsable json object (syntactically correct) but what the values are in that json is not guaranteed to make sense in the given context.
    - lubujackson 1y ago
      I find it does pretty well given a reasonable prompt and (especially) well-named keys/JSON structure. So if you had boss.mouse.hp you would get higher HP than random_enemies.mouse.hp, or better: enemies.level_1.mouse.hp.
    - viralpraxis 1y ago
      You can specify `minimum` and `maximum` property for these fields. So this schema { "$id": "https://example.com/test.schema.json", "$schema": "https://json-schema.org/draft/2020-12/schema", "title": "Person", "type": "object", "properties": { "hp": { "type": "integer", "description": "HP", "minimum": 1, "maximum": 15 } } } is converted to this BNF-like representation: hp ::= ([1-9] | "1" [0-5]) space hp-kv ::= "\"hp\"" space ":" space hp root ::= "{" space (hp-kv )? "}" space space ::= | " " | "\n"{1,2} [ \t]{0,20}
      - michaelgiba 1y ago
        For anyone curious here is an interactive write up about this http://michaelgiba.com/grammar-based/index.html http://michaelgiba.com/grammar-based/index.html

What if the converted version is not in the wanted syntax?

by cyptus 1y ago

What if the converted version is not in the wanted syntax?

Constrained generation guarantees syntax. It does not guarantee semantic correctness tho. Imagine you want a json object with "hp" and "damage". If you use a grammar, the model will be forced to output a json object with those two values. But it's not guaranteed to get sensible values. With a 2nd pass you basically "condition" it on the text right above, hoping to get better semantic understanding.

lyu07282 1y ago
I'm pretty sure the grammar is generated from the Json schema, it doesn't just constrain json syntax, it constraints on the schema (including enums and such). The schema is also given to the model (at least in openai) you can put instructions in the json schema as well that will be taken into account.
- NitpickLawyer 1y ago
  Perhaps I worded that poorly. What I mean by semantic correctness is that the model could output nonsensical values for some things. Say in a game, "normal" health is ~100hp and the model creates a wizard with 50hp but then a mouse with 10000hp. So you're guaranteed to get a parsable json object (syntactically correct) but what the values are in that json is not guaranteed to make sense in the given context.
  - lubujackson 1y ago
    I find it does pretty well given a reasonable prompt and (especially) well-named keys/JSON structure. So if you had boss.mouse.hp you would get higher HP than random_enemies.mouse.hp, or better: enemies.level_1.mouse.hp.
  - viralpraxis 1y ago
    You can specify `minimum` and `maximum` property for these fields. So this schema { "$id": "https://example.com/test.schema.json", "$schema": "https://json-schema.org/draft/2020-12/schema", "title": "Person", "type": "object", "properties": { "hp": { "type": "integer", "description": "HP", "minimum": 1, "maximum": 15 } } } is converted to this BNF-like representation: hp ::= ([1-9] | "1" [0-5]) space hp-kv ::= "\"hp\"" space ":" space hp root ::= "{" space (hp-kv )? "}" space space ::= | " " | "\n"{1,2} [ \t]{0,20}
    - michaelgiba 1y ago
      For anyone curious here is an interactive write up about this http://michaelgiba.com/grammar-based/index.html http://michaelgiba.com/grammar-based/index.html