It's more efficient, both in time (optimized character parsing, no repeated work if parser backtracks) and memory usage (store identifiers as an interned byte s

It's more efficient, both in time (optimized character parsing, no repeated work if parser backtracks) and memory usage (store identifiers as an interned byte string instead of a tree of character tokens). It also makes it easier for some tools to only use the lexer, which makes them more likely to work even if code is not syntactically correct or written for a newer language revision than the tool.