OP here. Use redis if you can. Use something like diskhash if you cannot afford the extra overhead of sqlite/redis. In my case, the alternative we were using

OP here. Use redis if you can. Use something like diskhash if you cannot afford the extra overhead of sqlite/redis. In my case, the alternative we were using before is an in-memory hash table built at startup from a text-based representation of the data. It was pretty good and worked very well up to millions of entries, but at the current scale we work with, it takes least 10 minutes at startup were necessary to build up the thing and it used ~200GB.