5 ms·

For your first question: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer

• calibas 2y ago
  I saw that, but the language makes me think it's not quite the same as what's really being used? "how a piece of text might be tokenized by a language model" "It's important to note that the exact tokenization process varies between models."
  • yorwba 2y ago
    That's why they have buttons to choose which model's tokenizer to use.
    • calibas 2y ago
      Yes, thank you, I understand that part. It's the might condition in the description that makes me think the results might not be the exact same as what's used in the live models.
      • baobabKoodaa 2y ago
        The results are the same.