6 ms·
In-memory database Redis wants to dabble in disk
- andrelaszlo 3y agoIt's not mentioned in the article but this is Redis Enterprise which is more like a superset of open source Redis.
- reconditerose 3y agoIt is part of the drive to make Redis "more like your classic database," he said. In the future, support for natural language queries and enhanced vector and feature store capabilities will be added. Seems way off the mark for why people use Redis. Developers use it precisely because it's not like their classic databases.
- deleted 3y ago[deleted]
- dimgl 3y agoI don’t think that’s necessarily always true. I was recently looking into using Redis as a regular database. Unfortunately data is still kept in memory even if it’s persisted on disk, so it was a no-go. You’d be surprised what people want!
- ReflectedImage 3y agoPerhaps you are looking for LMDB?
- tayo42 3y agoWhat were you trying to accomplish? Curious about your use if you can share it
- super_scooper 3y ago>You’d be surprised what people want! If what you want is persistent k/v, then there are already tons databases that can do that. Redis has a different use case, so what are you trying to say? That you want to misuse tools?
- lmm 3y ago> If what you want is persistent k/v, then there are already tons databases that can do that. Are there? What's a good, mainstream, persistent k/v for cheap single-node instances? PostgreSQL's KV support isn't good enough, Cassandra and MongoDB expect to be set up as a cluster and have all the overhead of that, BikeshedDB is alpha and I won't touch their upstream for obvious reasons, CouchDB is pretty much unmaintained, ....
- chokolad 3y agoHow about LMDB ? It’s single writer/multiple reader, very fast and reliable.
- lmm 3y agoThis thread is my first time hearing of it, it doesn't seem particularly established/mature (e.g. I can't find cloud services offering managed instances, whereas there are plenty of those for Redis)
- reconditerose 3y agoI'll bite. I think with most relational and document databases, my view of the "classical world", you are often storing all of your data in tables or documents in a structured way. Normalizing the data where possible. You're then building indexes on that data to pull it out in a useful operational, optimizing for the most common operations. If a new use case comes up, you add some more tables, add some constraints, and create some new indexes. I think with Redis, you are starting knowing that your use case needs to be fast. You start with the access patterns and fit that into key/value lookups (or search with modules). You're typically denormalizing the data, duplicating it or aggregating it from other places, so that you can serve operational data fast. Otherwise, just use in-memory SQL.
- willsmith72 3y agoThe lines between data "stores" are blurring. I've seen it before where x team needs a fast key value store, so the obvious choice is redis. A year down the line, the product is evolving and the devs want to do some "classic" queries. Maybe it makes sense to keep it in redis based on their support for the queries you need, rather than migrating/replicating to a whole other system.
- karmakaze 3y agoThey mention both flash and disk, which finally gets explained as tiers 2 and 3. Also less clear though I assume this applies to their hosted service offering and not the core software. > "We can take the lesser-used data that hasn't been touched in a while and shuttle it off to flash where it can sit for a while. When the user comes back eventually, it's very easy for us to seamlessly move it from flash back into memory. And that allows the company to save costs," he said. > Redis is now planning to extend the concept to disk-based memory to offer support for a three-tiered architecture.
- hivacruz 3y agoFor those interested in an alternative to Redis on disk, compatible with the Redis protocol, have a look on Kvrocks[1] which was recently accepted in the Apache foundation. It is based on RocksDB and works quite nicely for us. [1]: https://github.com/apache/kvrocks https://github.com/apache/kvrocks
- chocolatkey 3y agoThanks for this, it seems to even support redis' lua functionality with is essential for us. I wish I hadn't decided to use redis stack since even kvrocks isn't compatible with its extensions
- 9dev 3y agoWhat I actually need is a better way to bootstrap a cluster using Docker containers …
- wutwutwat 3y agohttps://redis.io/commands/restore/ https://redis.io/commands/restore/
- mrinterweb 3y agoI was confused because I know Redis already offers persistence. https://redis.io/docs/management/persistence/ https://redis.io/docs/management/persistence/ The main difference here is how its runtime will shift from runtime data only in RAM to not storing as much data in RAM and looking up on disk. Current Redis persistence looks like the runtime is stored in RAM and changes are periodically or append only persisted to disk. I hope I got that right.
- deleted 3y ago[deleted]
- whalesalad 3y ago> One main criticism of Redis had been its lack of support for SQL, the ubiquitous query language. Trollope said that was fixed now. A module RediSQL is available on GitHub. I think redisql has been supplanted by zeesql - https://zeesql.com/ https://zeesql.com/
- Xeoncross 3y agoI don't understand this. How can you want SQL for something that you are using because it's faster than SQL? If your dataset fits in memory and you want relations then just use PostgreSQL/MySQL. Why would you pick the unique data structures and performance of redis sets/lists/pubsub/HLL and want SQL in front of it?
- thrixton 3y agoSQL the language of choice for data manipulation, to me at least, although I wonder if I’m one of the dinosaurs after 20+ years. After working with MongoDb, CosmosDb and DynamoDb, I’m very happy to be back in a proper SQL world with Postgres.
- rezonant 3y agobecause it's web scale
- MenhirMike 3y agoNo, that's MongoDB. (Not as webscale as /dev/null though)
- Xeoncross 3y agoFor those that missed the joke: https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
- MenhirMike 3y ago
- ohnoesjmr 3y agoI've tried Redis-on-flash (enterprise version), and performance was very poor. Its just lsmt underneath and suffers from all the usual lsmt issues. In the end we went with Aerospike which actually while backed by NVMe, outperforms redis backed by memory. Aerospike in memory is completely absurd speeds and throughput. Sadly the setup is a bit weird and query model is a bit wonky but we worked around it. A bit sad on how expensive the enterprise version is.
- otterley 3y agoAnyone know how their plans differ from the Virtual Memory plans that antirez had about 13 years ago, implemented in Redis 2.6, and then (wisely) abandoned? https://redis.io/docs/reference/internals/internals-vm/ https://redis.io/docs/reference/internals/internals-vm/
- rjh29 3y agoNVMe is a thing now.
- infomaniac 3y agomemcached has recently gained the ability to spill to disk: https://github.com/memcached/memcached/wiki/Extstore https://github.com/memcached/memcached/wiki/Extstore we recently implemented this to grow our caches to >50TB
- hifycycyv 3y ago[dead]
- donatj 3y ago> It is part of the drive to make Redis "more like your classic database," he said. In the future, support for natural language queries and enhanced vector and feature store capabilities will be added. This initiative aligns with Redis's ambition to be seen as more than just a fast, albeit expensive, cache This sounds to me like not understanding your place in the market or why people use your product. I can name half a dozen classic SQL databases off hand. I can’t name a tool that competes with Redis at their niche. Why aim for an already oversaturated market when you already have a good profitable niche.
- deleted 3y ago[deleted]
- vidarh 3y ago> Why aim for an already oversaturated market when you already have a good profitable niche. Presumably because there are a lot of things that straddles the two, and where you risk losing people to databases or hybrid setups where the Redis use will be simple enough that people won't want to pay for support. E.g. cases where you can't afford to keep it all in memory where you today might end up resorting to using Redis as just a dumb-ish in front of a database, or even avoiding Redis entirely in favour of in-memory tables or aggressive cache settings for your database even in cases where you might prefer Redis if it could intelligently handle a larger working set more cheaply. And also to create a buffer given that the level of oversaturation in the classic database market may make their niche a tempting target to try to expand into for some of those to set themselves apart.
- jstummbillig 3y agoAssumedly, because they do know their place in the market and decided it's not great and that the outlook is worse. Recently DHH of Rails fame gave a talk about Redis and how at their company it's being replaced in favor of long term storage on the now much faster and cheaper SSDs. The takeaway was that the long caching it enables yields far better real world results in their products. He even had some charts.
- geenat 3y ago
- tehbeard 3y agoHow would the example of a scoreboard work with tiered memory? Surely you need the sorted set in memory to derive rank/manipulate it etc? Or would you have to shard it and add other elements on top in app code to focus on only a subsection of the data?
- pzmarzly 3y ago> Surely you need the sorted set in memory to derive rank/manipulate it etc? If you store it as an augmented balanced tree, then you only need to load the path from root to leaf to get or change rank. For example, if each node stored the number of leafs of its subtree, then you can avoid visiting most branches by simple logic: if you want to find element with rank=14, and left subtree only has 10 elements in total, ignore it and go straight to the right subtree, searching for element with rank=4 there.
- fumeux_fume 3y agoRedisk... Sounds like a good April fools joke
- iot_devs 3y agoHow bad is Redis with a swap file on SSD? I would expect it to be a very reasonable middle ground that requires almost no work and it is already enough and ready for most use cases.
- reconditerose 3y agoIt's really bad. The problem is that the swapping of pages is constantly stalling the main Redis thread, which is not built for parallelism. Redis is built around a single highly efficient thread, so stalling it for 250us to pull a page from disk instantly reduces the throughput by an order of magnitude.
- remram 3y agoI think I would rather they added a write-ahead-log to make writes durable, than disk lookup to support bigger-than-RAM datasets (slowly).