7 ms·
Instead of having distinct tokens, you could have modifier vectors which would be added to other tokens. Think in terms of control, shift, meta etc.
by lambdaone 3mo ago
Instead of having distinct tokens, you could have modifier vectors which would be added to other tokens. Think in terms of control, shift, meta etc.
- certainforest 3mo agoHey, Jasmine here (one of the authors) -- that's an interesting idea! There's an interesting exploration of this here: https://www.lesswrong.com/posts/HEzNZ9gvgYwT3aZFS/role-embeddings-making-authorship-more-salient-to-llms https://www.lesswrong.com/posts/HEzNZ9gvgYwT3aZFS/role-embed.... Curious if you have additional thoughts, and thanks for reading!