6 ms·
And you know what the tokenizer is made of? Weights.
by bfung 4mo ago
And you know what the tokenizer is made of?
Weights.
- jrahmy 4mo agoA tokenizer is a deterministic string-matching program, it's not made out of weights in the same sense as a neural network itself.
- davrosthedalek 3mo agoBut it could be. It's just less efficient.
- jrahmy 3mo agoI don't see how. You could ask a neural network to do the tokenization I suppose, but in doing so you'd have to convert the prompt into tokens via the same deterministic process the network was trained on, essentially just moving the exact same process up one layer.
- deleted 3mo ago[deleted]
- bfung 3mo agoHow does one choose what sequence of bytes constitutes a token?