4 ms·
TypeChat: let's try really hard to try to convince the model to make the highest-scoring tokens follow the grammar we want. Guidance (and this project?): Let's
by 2bitencryption 3y ago
TypeChat: let's try really hard to try to convince the model to make the highest-scoring tokens follow the grammar we want.
Guidance (and this project?): Let's not even bother with trying to convince the model; instead, we'll only sample from the set of tokens that are guaranteed to be correct for the grammar we want to emit.
- btwillard 3y agoYeah, and our addition to all that is to almost completely remove the cost of determining the next valid tokens on each step.