5 ms·
Seems odd they don't reference this on the page. Instead they list: "If you need a programmatic interface for tokenizing text, check out the transformers pack
by lspears 3y ago
Seems odd they don't reference this on the page. Instead they list:
"If you need a programmatic interface for tokenizing text, check out the transformers package for python or the gpt-3-encoder package for node.js."
with the links:
https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast https://huggingface.co/docs/transformers/model_doc/gpt2#tran...
https://www.npmjs.com/package/gpt-3-encoder https://www.npmjs.com/package/gpt-3-encoder