4 ms·
Supporting 10 different languages in regex is a drop in the ocean. The regex can be generated programmatically and you can compress regexes easily. We used to h
by doublesocket 6mo ago
Supporting 10 different languages in regex is a drop in the ocean. The regex can be generated programmatically and you can compress regexes easily. We used to have a compressed regex that could match any placename or street name in the UK in a few MB of RAM. It was silly quick.
- cogman10 6mo agoI think it will depend on the language. There are a few non-latin languages where a simple word search likely won't be enough for a regex to properly apply.
- mycall 6mo agoExactly this. Unicode is a big beast to consider in regex concats.
- astrocat 6mo agowoah. This is a regex use I've never heard of. I'd absolutely love to see a writeup on this approach - how its done and when it's useful.
- benlivengood 6mo agoYou can literally | together every street address or other string you want to match in a giant disjunction, and then run a DFA/NFA minimization over that to get it down to a reasonable size. Maybe there are some fast regex simplification algorithms as well, but working directly with the finite automata has decades of research and probably can be more fully optimized.
- doublesocket 6mo agoThis was many moons ago, written in perl. From memory we used Regexp::Trie - https://metacpan.org/release/DANKOGAI/Regexp-Trie-0.02/view/lib/Regexp/Trie.pm https://metacpan.org/release/DANKOGAI/Regexp-Trie-0.02/view/... We used it to tokenize search input and combined it with a solr backend. Worked really remarkably well.