6 ms·
You'd need to put the input first for this approach to work, but in my testing models work better if you lead with a question.
by execveat 3y ago
You'd need to put the input first for this approach to work, but in my testing models work better if you lead with a question.
- kcorbitt 3y agoHmm. I admit that I haven't thought about this deeply, but I'm not sure that's true? It seems to me that you could extend the KV cache either backwards or forwards equally easily.
- Siira 3y agoYou can’t. The later values depend on the earlier ones, so changing the early tokens invalidates your whole cache. This is also probably why leading with a question works better in the first place. All later processing conditions on the question in this way. BTW, in my very limited testing, GPT4 doesn’t care about the order.
- tysam_and 3y agoI could be reading this wrong, but my assumption is/has been that the prompt goes up to the end of the JSON field name, and the LLM is only filling in the actual value, not the key. I could be wrong on this one, however.