5 ms·
Having a public tokenizer is quite useful, specially for embeddings. It allows you to do the chunking locally without going to the internet.
by xfalcox 1y ago
Having a public tokenizer is quite useful, specially for embeddings. It allows you to do the chunking locally without going to the internet.
- jgalt212 1y agoand you don't have to re-embed everything if the provider sunsets a model.