Have you tried just using a faster tokenization library? Github's BPE crate (https://crates.io/crates/bpe https://crates.io/crates/bpe) advertises >10x speedup

Have you tried just using a faster tokenization library? Github's BPE crate (https://crates.io/crates/bpe https://crates.io/crates/bpe) advertises >10x speedup relative to HF, as do others.