6 ms·

While I love Redis as a versatile tool for external data structures, it's still lacking in two areas IMHO: One, it would be cool to be able to embed it, simila

• amtamt 4mo ago
  Genuinely interested why we need HA in redis, just not read round robin from multiple non-HA instances? Redis (and memcache) are memory caches and should be treated like that, not like highly consistent distributed session store.
  • 9dev 4mo ago
    Redis doesn't necessarily have to be used as a cache. Streams, for example, make it a great message queue; but a single-node message queue is a single point of failure and thus not viable for many setups.
    • acejam 4mo ago
      That's why you run Redis Sentinel in production
      • 9dev 4mo ago
        That you do. Until you realise that there is only a single writer in that scenario, it doesn’t address any sharding concerns, you need to use compatible clients that opt into the sentinel protocol, during failover you’ll see client errors… there’s lots of room for improvement on redis HA.
      • lukaslalinsky 4mo ago
        With the amount of problems I had using Redis Sentinel, I really wish there was another way. On multiple occasions, with completely different deployments, it got itself into a non-repairable state where the only option was to drop it and setup the replicas manually. I was hoping someone would do a Patroni-like project for Redis, but I've not found it yet. I've moved all persistent data to PostgreSQL and use a number of Valkeys behind Envoy proxy as a cache.
        • dvkashapov 4mo ago
          I suggest you to take a look at rdsync (https://github.com/yandex/rdsync https://github.com/yandex/rdsync), exactly what you want: Patroni-like high-availability tool for Valkey/Redis. Uses ZooKeeper for external coordination. We use it in our large deployment and with a couple patches you will forged about the need to take manual actions to resolve broken states.
          • 9dev 4mo ago
            At which scale would you recommend this over a Sentinel setup?
            • dvkashapov 4mo ago
              To be honest - at any scale, this really does help me not wake up at night to fix broken states by hand as sometimes on-call engineer. Although note that rdsync is mainly for Valkey up to 9.1, there were Redis patches for 7.2 (last BSD version).
  • n_e 4mo ago
    Redis is used for plenty of things, not just memory caches. For example if you use it for session storage, you can't have your application read from a random instance that may or may not contain the session.
    • tossandthrow 4mo ago
      This case is exactly what he talks about. To get HA just setup more than one redis cache - or rebuild the session if it was lost in the redis cache.
      • 9dev 4mo ago
        It’s not. Imagine a web app that stores your user information in a session store, mapped by your cookie-provided session ID. Your web app searches redis 1 for the session id, but since that key is on redis 2, the lookup fails and the application thinks there is no such session, and rejects the request. Now you could solve this specific case by sharding by prefix, or by querying all instances, but then you still do not have high availability: if the instance a specific session is on is down, these users cannot authenticate. At that point you’re better off with a single instance.
        • olavgg 4mo ago
          But that is his point. If you cannot find the session id in redis, you login again. If your Redis server crash, you start a new one and everyone just login again. No data is lost.
          • 9dev 4mo ago
            Sure the data is lost. A session commonly holds arbitrary state, and even if it’s just the login information. This is ridiculous.
            • trumpdong 4mo ago
              If you consider it important, you have to store it in a real database. No buts. If you don't consider it important, sharded redis works fine.
  • __s 4mo ago
    Years ago I enabled durability on redis & used it as database for an online card game
  • compumike 4mo ago
    > Redis (and memcache) are memory caches and should be treated like that If you haven't come across Kvrocks yet, it may be worth a look: https://github.com/apache/kvrocks https://github.com/apache/kvrocks https://kvrocks.apache.org/ https://kvrocks.apache.org/ . It's a database with a Redis-compatible wire protocol, but the database is stored on disk. This means your working set is not limited by RAM and can be a few orders of magnitude larger! On modern SSDs this is still very fast. I think it improves the durability story as well. But the big win is the orders of magnitude larger database space. As I've been improving my side project https://totalrealreturns.com/ https://totalrealreturns.com/ recently I've ended up using both Redis and Kvrocks together. Redis is great for small global state that needs to be super fast. Kvrocks is great for larger bulk data storage (large precomputed datasets), but also supports a lot of the Redis data structures as well as Lua scripts.
  • marklubi 4mo ago
    For the project I've been working on for more than 15 years, we make extensive use of the pub/sub functionality for distributing live data. Pub/sub scales well across the cluster. Publish to one, and it goes out to subscribers on any of the nodes that they've connected to. Will millions of users, high availability is critical for this functionality.
  • yxhuvud 4mo ago
    Redis have many use cases, and acting as a cache is only one of them. One very common usage is as a backend for background worker jobs. That can need HA.
• karelpeeters 4mo ago
  What would be the point of embedding Redis into an application? What's the advantage of using Redis over using the builtin (or third party) data structures of the language the application is developed in? I'm asking as a non-webdev who never quite got what Redis actually does, but would love to learn.
  • jaapz 3mo ago
    I have 15 processes handling API requests over HTTP. I want rate limiting, so users don't DoS my API. I need shared state across these processes to store information about rate limits for each client key. I store those in redis. Or queues, caching, pub/sub... Redis can do a lot, it's really (and i mean really) fast, very easy to get up and running and the protocol is pretty simple.
  • mystifyingpoi 4mo ago
    For simple cases, it is probably a total overkill to even consider it, but for something heavier, embedding the database gives you a chance to trivially migrate later to a separate database server.
    • thefreeman 4mo ago
      Redis is not a database. It’s a key / value store.
      • theultdev 4mo ago
        that's still a database. it's not a relational database.
      • rytis 4mo ago
        It kind of is a database: A key-value database, or key-value store, is a data storage paradigm designed for storing, retrieving, and managing associative arrays, a data structure more commonly known today as a dictionary. https://en.wikipedia.org/wiki/Key–value_database https://en.wikipedia.org/wiki/Key–value_database
      • 4mo ago
• deleted 4mo ago
  [deleted]
• adamcharnock 4mo ago
  > One, it would be cool to be able to embed it, similar to sqlite, directly into applications. I've found myself wanting this on several occasions too. I.e. wanting all my rust backend processes (k8s pods) to have some minimal shared state, without having to spin up a Redis cluster. I've talked to Claude about it a couple of times, and it descends into something like, "you gotta use Raft or CRDTs, and pick 2 out of 3 from CAP". Which honestly seems pretty fair, and indicates to me that I'm dreaming for something magical. Nonetheless, it is nice to hear someone else asking for this. If this is indeed feasible (even if simple/limited), then I'd be interested to try it.
  • williamdclt 4mo ago
    I don't know if that'll make you feel any better but yeah, you're indeed asking for the impossible! You need consensus between your nodes that store state _somehow_, either these nodes are Redis and it does that for you, or these nodes are your pods and you need to do consensus yourself (zookeeper might help, but you're definitely in "complicated stuff" territory). Spinning up an in-memory (no persistence) Redis cluster in your k8s should be easy enough, hopefully?
    • adamcharnock 4mo ago
      Yeah, fair enough. And yes, adding a Redis cluster is fine, it is just another moving part to manage. But given that the alternative is made out of unobtainium, I guess that is just the way of it :-)
• echelon 4mo ago
  > it's still lacking in two areas This is entirely different than what Redis is and tries to solve. Sqlite is embedded. It's not a distributed SQL. Redis is a distributed data structure store and concurrency primitive. These are worlds apart. > HA story is so much more complicated than it should be It is precisely as complicated as it needs to be. You don't want data loss. If you're in the business of high available fault tolerance, you read the manual and learn how to Redis.
  • 9dev 4mo ago
    What kind of an answer is that? This software is perfect the way it is, you’re just to inept to hold it right? A high availability protocol should not leak into the client. It should be able to discover other nodes. It should not land in broken states so easily. It should not limit the number of writers. It should not error during failover. Are these hard problems? Yes. Should we just accept that things are hard because that’s how the gods have given them to us? No.
    • echelon 4mo ago
      High availability and abstraction complexity are orthogonal. Redis is a low-level concurrency primitive, and it made certain choices in dealing with CAP. It might be single-threaded, but it can easily absorb 100,000+ requests per second. I've built systems that handle billions of dollars of online payments flow, active-active, with six nines of uptime reliability on top of Redis. It does what it says on the tin, and it doesn't need to be everything for everybody. This is a hard domain and you're going to have to deal with different problems and tradeoffs. If you want something higher level, there are other systems to reach for.
• atmosx 4mo ago
  > Two, the HA story is so much more complicated than it should be. Really? I am curious, how would simplify it? It’s a very well defined problem and all the “solutions” are very complicated and with many strings attached. I have managed one or two systems that came in different modalities and you had to pick your poison and had to make sure the other engineering teams understand the trade offs. Some were more successful than others, but “easy” never crossed my mind. Redis is single threaded and doesn’t concern itself with these things, directly, exactly because Antitez understood the trade offs and made all the right choices. How would you improve the HA story without sacrificing ease of use and performance on a single thread?
• ncruces 4mo ago
  Redka can be used embed, in Go: https://github.com/nalgeon/redka https://github.com/nalgeon/redka
• anapeksha 3mo ago
  [dead]