5 ms·

Genuinely interested why we need HA in redis, just not read round robin from multiple non-HA instances? Redis (and memcache) are memory caches and should be tre

by amtamt 3mo ago

Genuinely interested why we need HA in redis, just not read round robin from multiple non-HA instances? Redis (and memcache) are memory caches and should be treated like that, not like highly consistent distributed session store.

9dev 3mo ago
Redis doesn't necessarily have to be used as a cache. Streams, for example, make it a great message queue; but a single-node message queue is a single point of failure and thus not viable for many setups.
- acejam 3mo ago
  That's why you run Redis Sentinel in production
  - 9dev 3mo ago
    That you do. Until you realise that there is only a single writer in that scenario, it doesn’t address any sharding concerns, you need to use compatible clients that opt into the sentinel protocol, during failover you’ll see client errors… there’s lots of room for improvement on redis HA.
  - lukaslalinsky 3mo ago
    With the amount of problems I had using Redis Sentinel, I really wish there was another way. On multiple occasions, with completely different deployments, it got itself into a non-repairable state where the only option was to drop it and setup the replicas manually. I was hoping someone would do a Patroni-like project for Redis, but I've not found it yet. I've moved all persistent data to PostgreSQL and use a number of Valkeys behind Envoy proxy as a cache.
    - dvkashapov 3mo ago
      I suggest you to take a look at rdsync (https://github.com/yandex/rdsync https://github.com/yandex/rdsync), exactly what you want: Patroni-like high-availability tool for Valkey/Redis. Uses ZooKeeper for external coordination. We use it in our large deployment and with a couple patches you will forged about the need to take manual actions to resolve broken states.
      - 9dev 3mo ago
        At which scale would you recommend this over a Sentinel setup?
        dvkashapov 3mo ago
        To be honest - at any scale, this really does help me not wake up at night to fix broken states by hand as sometimes on-call engineer. Although note that rdsync is mainly for Valkey up to 9.1, there were Redis patches for 7.2 (last BSD version).
n_e 3mo ago
Redis is used for plenty of things, not just memory caches. For example if you use it for session storage, you can't have your application read from a random instance that may or may not contain the session.
- tossandthrow 3mo ago
  This case is exactly what he talks about. To get HA just setup more than one redis cache - or rebuild the session if it was lost in the redis cache.
  - 9dev 3mo ago
    It’s not. Imagine a web app that stores your user information in a session store, mapped by your cookie-provided session ID. Your web app searches redis 1 for the session id, but since that key is on redis 2, the lookup fails and the application thinks there is no such session, and rejects the request. Now you could solve this specific case by sharding by prefix, or by querying all instances, but then you still do not have high availability: if the instance a specific session is on is down, these users cannot authenticate. At that point you’re better off with a single instance.
    - olavgg 3mo ago
      But that is his point. If you cannot find the session id in redis, you login again. If your Redis server crash, you start a new one and everyone just login again. No data is lost.
      - 9dev 3mo ago
        Sure the data is lost. A session commonly holds arbitrary state, and even if it’s just the login information. This is ridiculous.
        trumpdong 3mo ago
        If you consider it important, you have to store it in a real database. No buts. If you don't consider it important, sharded redis works fine.
__s 3mo ago
Years ago I enabled durability on redis & used it as database for an online card game
compumike 3mo ago
> Redis (and memcache) are memory caches and should be treated like that If you haven't come across Kvrocks yet, it may be worth a look: https://github.com/apache/kvrocks https://github.com/apache/kvrocks https://kvrocks.apache.org/ https://kvrocks.apache.org/ . It's a database with a Redis-compatible wire protocol, but the database is stored on disk. This means your working set is not limited by RAM and can be a few orders of magnitude larger! On modern SSDs this is still very fast. I think it improves the durability story as well. But the big win is the orders of magnitude larger database space. As I've been improving my side project https://totalrealreturns.com/ https://totalrealreturns.com/ recently I've ended up using both Redis and Kvrocks together. Redis is great for small global state that needs to be super fast. Kvrocks is great for larger bulk data storage (large precomputed datasets), but also supports a lot of the Redis data structures as well as Lua scripts.
marklubi 3mo ago
For the project I've been working on for more than 15 years, we make extensive use of the pub/sub functionality for distributing live data. Pub/sub scales well across the cluster. Publish to one, and it goes out to subscribers on any of the nodes that they've connected to. Will millions of users, high availability is critical for this functionality.
yxhuvud 3mo ago
Redis have many use cases, and acting as a cache is only one of them. One very common usage is as a backend for background worker jobs. That can need HA.

by amtamt 3mo ago

9dev 3mo ago

Redis doesn't necessarily have to be used as a cache. Streams, for example, make it a great message queue; but a single-node message queue is a single point of failure and thus not viable for many setups.

acejam 3mo ago
That's why you run Redis Sentinel in production
- 9dev 3mo ago
  That you do. Until you realise that there is only a single writer in that scenario, it doesn’t address any sharding concerns, you need to use compatible clients that opt into the sentinel protocol, during failover you’ll see client errors… there’s lots of room for improvement on redis HA.
- lukaslalinsky 3mo ago
  With the amount of problems I had using Redis Sentinel, I really wish there was another way. On multiple occasions, with completely different deployments, it got itself into a non-repairable state where the only option was to drop it and setup the replicas manually. I was hoping someone would do a Patroni-like project for Redis, but I've not found it yet. I've moved all persistent data to PostgreSQL and use a number of Valkeys behind Envoy proxy as a cache.
  - dvkashapov 3mo ago
    I suggest you to take a look at rdsync (https://github.com/yandex/rdsync https://github.com/yandex/rdsync), exactly what you want: Patroni-like high-availability tool for Valkey/Redis. Uses ZooKeeper for external coordination. We use it in our large deployment and with a couple patches you will forged about the need to take manual actions to resolve broken states.
    - 9dev 3mo ago
      At which scale would you recommend this over a Sentinel setup?
      - dvkashapov 3mo ago
        To be honest - at any scale, this really does help me not wake up at night to fix broken states by hand as sometimes on-call engineer. Although note that rdsync is mainly for Valkey up to 9.1, there were Redis patches for 7.2 (last BSD version).

n_e 3mo ago

Redis is used for plenty of things, not just memory caches. For example if you use it for session storage, you can't have your application read from a random instance that may or may not contain the session.

tossandthrow 3mo ago
This case is exactly what he talks about. To get HA just setup more than one redis cache - or rebuild the session if it was lost in the redis cache.
- 9dev 3mo ago
  It’s not. Imagine a web app that stores your user information in a session store, mapped by your cookie-provided session ID. Your web app searches redis 1 for the session id, but since that key is on redis 2, the lookup fails and the application thinks there is no such session, and rejects the request. Now you could solve this specific case by sharding by prefix, or by querying all instances, but then you still do not have high availability: if the instance a specific session is on is down, these users cannot authenticate. At that point you’re better off with a single instance.
  - olavgg 3mo ago
    But that is his point. If you cannot find the session id in redis, you login again. If your Redis server crash, you start a new one and everyone just login again. No data is lost.
    - 9dev 3mo ago
      Sure the data is lost. A session commonly holds arbitrary state, and even if it’s just the login information. This is ridiculous.
      - trumpdong 3mo ago
        If you consider it important, you have to store it in a real database. No buts. If you don't consider it important, sharded redis works fine.

__s 3mo ago

Years ago I enabled durability on redis & used it as database for an online card game

compumike 3mo ago

> Redis (and memcache) are memory caches and should be treated like that If you haven't come across Kvrocks yet, it may be worth a look: https://github.com/apache/kvrocks https://github.com/apache/kvrocks https://kvrocks.apache.org/ https://kvrocks.apache.org/ . It's a database with a Redis-compatible wire protocol, but the database is stored on disk. This means your working set is not limited by RAM and can be a few orders of magnitude larger! On modern SSDs this is still very fast. I think it improves the durability story as well. But the big win is the orders of magnitude larger database space. As I've been improving my side project https://totalrealreturns.com/ https://totalrealreturns.com/ recently I've ended up using both Redis and Kvrocks together. Redis is great for small global state that needs to be super fast. Kvrocks is great for larger bulk data storage (large precomputed datasets), but also supports a lot of the Redis data structures as well as Lua scripts.

marklubi 3mo ago

For the project I've been working on for more than 15 years, we make extensive use of the pub/sub functionality for distributing live data. Pub/sub scales well across the cluster. Publish to one, and it goes out to subscribers on any of the nodes that they've connected to. Will millions of users, high availability is critical for this functionality.

yxhuvud 3mo ago

Redis have many use cases, and acting as a cache is only one of them. One very common usage is as a backend for background worker jobs. That can need HA.