8 ms·

What would be the point of embedding Redis into an application? What's the advantage of using Redis over using the builtin (or third party) data structures of t

What would be the point of embedding Redis into an application? What's the advantage of using Redis over using the builtin (or third party) data structures of the language the application is developed in? I'm asking as a non-webdev who never quite got what Redis actually does, but would love to learn.

jaapz 3mo ago
I have 15 processes handling API requests over HTTP. I want rate limiting, so users don't DoS my API. I need shared state across these processes to store information about rate limits for each client key. I store those in redis. Or queues, caching, pub/sub... Redis can do a lot, it's really (and i mean really) fast, very easy to get up and running and the protocol is pretty simple.
mystifyingpoi 3mo ago
For simple cases, it is probably a total overkill to even consider it, but for something heavier, embedding the database gives you a chance to trivially migrate later to a separate database server.
- thefreeman 3mo ago
  Redis is not a database. It’s a key / value store.
  - theultdev 3mo ago
    that's still a database. it's not a relational database.
  - rytis 3mo ago
    It kind of is a database: A key-value database, or key-value store, is a data storage paradigm designed for storing, retrieving, and managing associative arrays, a data structure more commonly known today as a dictionary. https://en.wikipedia.org/wiki/Key–value_database https://en.wikipedia.org/wiki/Key–value_database
  - bijowo1676 3mo ago
    you are confusing redis with memcached
freakynit 3mo ago
Probably because Redis gives you a very well-defined/understood set of rich data structures with built-in behavior like TTL, atomic operations, eviction, and persistence. These things are otherwise usually scattered across native types, helper classes, or entirely separate libraries.
- stingraycharles 3mo ago
  It doesn’t seem like the right tool for the job, though. Aren’t your own programming language’s constructs much more well-defined / understood ?
  - lpapez 3mo ago
    I use PHP. None of the language tools or constructs available to me are adequate. https://blog.codinghorror.com/the-php-singularity/ https://blog.codinghorror.com/the-php-singularity/
    - stingraycharles 3mo ago
      And you want to embed Redis inside PHP as a solution?? That’s nuts.
      - sinpif 3mo ago
        Where else could they store their serialized PHP data structures? (just kidding)
  - freakynit 3mo ago
    Language's own native data-structures are generally much more capable and vast. 99%+ developers use only a very limited set of those capabilities. This approach packages those most used ones into a nice, consistent DSL. It's similar in effect to what busybox does to shell utilities, though the motives are different.
    - 3mo ago
razighter777 3mo ago
In practice, mostly scaling sessions and ephemeral data (caching) across multiple intances of a microservice on multiple machines. Seperating the kv store and the application allows upgrading each application while retaining availability and avoiding loss of session data.
noodletheworld 3mo ago
Why would you embed SQLite? It’s the same use case with a different api. A typical (meaningful) example might be communication between threads or actors in a single process, or idempotent tests. As with SQLite, an external xxx that does this for you is certainly better, etc. but it’s convenient sometimes, to have an application that doesn’t go “now before you run this install Postgres…”. It’s seldom useful for a web app where you control everything.
jchw 3mo ago
To me the thing I like about Redis is that it gives you a storage engine very suitable for caches; it handles TTLs and memory pressure, as well as built-in serialization with the ability to get better performance by allowing for some data loss. At the same time, many users will be deploying small programs to individual machines. If you could just have Redis be embedded this would make it very operationally simple: no additional daemons and a single file to backup if you want to. It would also be useful because of the ability to switch modalities. When running a multi node service, you can use Redis to share data between nodes and use Redis pubsub as a communication bus. If you wanted to support a simple single node configuration too, then it wouldn't need to be a special case, it could just go through the same mechanism but with an embedded Redis instance. It's pretty similar to SQLite: being able to embed more or less a complete storage engine into your app can be very convenient and powerful.
- 0x457 3mo ago
  Well, if you have a single instance than using language libraries and structures will be better in most cases. If you use multiple nodes, then you probably want your redis lifecycle not be tied to application lifecycle.
  - jchw 3mo ago
    I am not aware of an in-process alternative similar to what Redis offers.
    - s_trumpet 3mo ago
      Mnesia, if you’re using Erlang or Elixir.
      - jchw 3mo ago
        Unfortunately I have never really used Erlang outside of deploying RabbitMQ. I mostly use Go, Rust, Python, sometimes C/C++. However, Mnesia seems like it is quite a bit more of a complete distributed database engine than Redis. To me the nicest thing about Redis is just the convenience of what it offers: very fast data structures, serialized, optimized (at least by default) for cases where speed is more important than durability. It is simple on many levels and somewhat constrained in scope. Mnesia seems to be aiming more generally in the distributed database category. So how do you feel they compare?
zbentley 3mo ago
A few nice things about doing this in no particular order: Embedding would make local dev/CI integration testing convenient. Embedding replicated Redis with each application instance would give you HA benefits while infra-management complexity. Embedded redis (even via local RPC) is still going to be faster than a lot of languages or frameworks’ built-in data structures. Large array operations in, say, Python are gonna slower than RPCing to Redis (assuming that the data structures are built gradually and not built all at once); to beat Redis you’d have to use numpy or something—-which is definitely preferable, but is extra work if your app already uses Redis for other things. Just like choosing SQLite over e.g. LMDB or RocksDB, embedded Redis would be a nice future proofing option for small apps during the prototype phase; less would have to be changed to move Redis out of the app than if a different cache or persistence service were chosen.
nilamo 3mo ago
I mostly use redis for pub/sub communication between services. If the app wasn't a collection of knative functions, and instead a monolith, it would be cool to also use redis for event based communication.
jiggawatts 3mo ago
Locality and latency. Network hops are not free! Those milliseconds are an eternity compared to local function calls. The optimal architecture is something like what Service Fabric or Orleans can do with their distributed dictionary types: reads are generally in-process and take only nanoseconds (but writes require a synchronous replica copy to a remote host.) Obviously this requires load balancers to steer traffic consistently, but that’s a common feature… outside of the public clouds where they forgot latency exists.

jaapz 3mo ago

I have 15 processes handling API requests over HTTP. I want rate limiting, so users don't DoS my API. I need shared state across these processes to store information about rate limits for each client key. I store those in redis. Or queues, caching, pub/sub... Redis can do a lot, it's really (and i mean really) fast, very easy to get up and running and the protocol is pretty simple.

mystifyingpoi 3mo ago

For simple cases, it is probably a total overkill to even consider it, but for something heavier, embedding the database gives you a chance to trivially migrate later to a separate database server.

thefreeman 3mo ago
Redis is not a database. It’s a key / value store.
- theultdev 3mo ago
  that's still a database. it's not a relational database.
- rytis 3mo ago
  It kind of is a database: A key-value database, or key-value store, is a data storage paradigm designed for storing, retrieving, and managing associative arrays, a data structure more commonly known today as a dictionary. https://en.wikipedia.org/wiki/Key–value_database https://en.wikipedia.org/wiki/Key–value_database
- bijowo1676 3mo ago
  you are confusing redis with memcached

freakynit 3mo ago

Probably because Redis gives you a very well-defined/understood set of rich data structures with built-in behavior like TTL, atomic operations, eviction, and persistence. These things are otherwise usually scattered across native types, helper classes, or entirely separate libraries.

stingraycharles 3mo ago
It doesn’t seem like the right tool for the job, though. Aren’t your own programming language’s constructs much more well-defined / understood ?
- lpapez 3mo ago
  I use PHP. None of the language tools or constructs available to me are adequate. https://blog.codinghorror.com/the-php-singularity/ https://blog.codinghorror.com/the-php-singularity/
  - stingraycharles 3mo ago
    And you want to embed Redis inside PHP as a solution?? That’s nuts.
    - sinpif 3mo ago
      Where else could they store their serialized PHP data structures? (just kidding)
- freakynit 3mo ago
  Language's own native data-structures are generally much more capable and vast. 99%+ developers use only a very limited set of those capabilities. This approach packages those most used ones into a nice, consistent DSL. It's similar in effect to what busybox does to shell utilities, though the motives are different.
  - 3mo ago

razighter777 3mo ago

In practice, mostly scaling sessions and ephemeral data (caching) across multiple intances of a microservice on multiple machines. Seperating the kv store and the application allows upgrading each application while retaining availability and avoiding loss of session data.

noodletheworld 3mo ago

Why would you embed SQLite? It’s the same use case with a different api. A typical (meaningful) example might be communication between threads or actors in a single process, or idempotent tests. As with SQLite, an external xxx that does this for you is certainly better, etc. but it’s convenient sometimes, to have an application that doesn’t go “now before you run this install Postgres…”. It’s seldom useful for a web app where you control everything.

jchw 3mo ago

To me the thing I like about Redis is that it gives you a storage engine very suitable for caches; it handles TTLs and memory pressure, as well as built-in serialization with the ability to get better performance by allowing for some data loss. At the same time, many users will be deploying small programs to individual machines. If you could just have Redis be embedded this would make it very operationally simple: no additional daemons and a single file to backup if you want to. It would also be useful because of the ability to switch modalities. When running a multi node service, you can use Redis to share data between nodes and use Redis pubsub as a communication bus. If you wanted to support a simple single node configuration too, then it wouldn't need to be a special case, it could just go through the same mechanism but with an embedded Redis instance. It's pretty similar to SQLite: being able to embed more or less a complete storage engine into your app can be very convenient and powerful.

0x457 3mo ago
Well, if you have a single instance than using language libraries and structures will be better in most cases. If you use multiple nodes, then you probably want your redis lifecycle not be tied to application lifecycle.
- jchw 3mo ago
  I am not aware of an in-process alternative similar to what Redis offers.
  - s_trumpet 3mo ago
    Mnesia, if you’re using Erlang or Elixir.
    - jchw 3mo ago
      Unfortunately I have never really used Erlang outside of deploying RabbitMQ. I mostly use Go, Rust, Python, sometimes C/C++. However, Mnesia seems like it is quite a bit more of a complete distributed database engine than Redis. To me the nicest thing about Redis is just the convenience of what it offers: very fast data structures, serialized, optimized (at least by default) for cases where speed is more important than durability. It is simple on many levels and somewhat constrained in scope. Mnesia seems to be aiming more generally in the distributed database category. So how do you feel they compare?

zbentley 3mo ago

A few nice things about doing this in no particular order: Embedding would make local dev/CI integration testing convenient. Embedding replicated Redis with each application instance would give you HA benefits while infra-management complexity. Embedded redis (even via local RPC) is still going to be faster than a lot of languages or frameworks’ built-in data structures. Large array operations in, say, Python are gonna slower than RPCing to Redis (assuming that the data structures are built gradually and not built all at once); to beat Redis you’d have to use numpy or something—-which is definitely preferable, but is extra work if your app already uses Redis for other things. Just like choosing SQLite over e.g. LMDB or RocksDB, embedded Redis would be a nice future proofing option for small apps during the prototype phase; less would have to be changed to move Redis out of the app than if a different cache or persistence service were chosen.

nilamo 3mo ago

I mostly use redis for pub/sub communication between services. If the app wasn't a collection of knative functions, and instead a monolith, it would be cool to also use redis for event based communication.

jiggawatts 3mo ago

Locality and latency. Network hops are not free! Those milliseconds are an eternity compared to local function calls. The optimal architecture is something like what Service Fabric or Orleans can do with their distributed dictionary types: reads are generally in-process and take only nanoseconds (but writes require a synchronous replica copy to a remote host.) Obviously this requires load balancers to steer traffic consistently, but that’s a common feature… outside of the public clouds where they forgot latency exists.