6 ms·
If you aren’t caching LLM functions during development, then you’re an even greater glutton for punishment than the normal engineer. My local file cache Python
by wildermuthn 2y ago
If you aren’t caching LLM functions during development, then you’re an even greater glutton for punishment than the normal engineer.
My local file cache Python decorator also allows the decorator to define the hash manually, either by the decorator’s parameter function call that plucks a value from the cached function params, or by calling a global function from anywhere with any arbitrary value.
What’s cool about caching results locally to files during development is the ease of invalidating caches — just delete the file named after the function and key you want.
- mpeg 2y agoThis is also why in my custom cache I back it with sqlite – much easier to delete one db file than thousands of pickle files.
- AlecSchueler 2y agoGlobs are a thing?
- mpeg 2y agoWeird comment... yes, they are, but what is faster for me to type `rm .cache_dir/function_name*.pickle` or just to delete the one sqlite file in my file manager / vscode file tree. Regardless, there are other reasons why sqlite is nice for this, you gain control over locking and thread safety without having to implement it all from scratch
- canadiantim 2y agoI'm sure this is a stupid question, but why is it much better to be caching LLM functions during development?
- p10_user 2y agoBecause they are generally incredibly computationally expensive operations that can take hours/days to complete (?more)
- williamzeng0 2y ago100%, invalidation needs to be fast or you're not really saving time. I'm curious about calling a global function, what's the use case for that?
- Karellen 2y agoI feel like adding an argument to the decorator that labels the "version" of the function would make deliberate cache invalidation more straightforward for cache users.
- williamzeng0 2y agoThe version input makes sense, I could also see some developers disliking that ux because of it's verbosity. But to deliberately invalidate you have to make a manual effort in either case.
- Karellen 2y agoTo me, it's more that invalidating the cache without it requires knowledge of implementation details (i.e. where the cache is stored, and how the cache files are named) that ideally shouldn't have to "leak" for the cache to be generally useful. (Buy hey, we are talking about what is famously one of the hard problems in comp sci, so (respectful) disagreements on how best to do it should be expected :-)
- williamzeng0 2y agoI appreciate the sentiment! Makes a ton of sense.