7 ms·
Redis explained
- dtertman 4y agoAt desktop resolution, the floating table of contents menu blocks out two of the (excellent) illustrations (second and second-last). Deleting aside.toc was very helpful.
- dsmmcken 4y agoYes, I would suggest increasing z-index on images so they pass above the toc. Adding a large dropshadow to the images the same color as the background would make it look like it fades out as it passes by. That's what I did for our blog that has a similar floating TOC + images that escape the text width.
- its_bbq 4y agoI've been looking into tech stacks to make a collaborative editor and Redis CRDTs come up a lot. IIUC this requires a Redis db running in each users machine and they connect P2P with each other. Do I understand right? Anyone have good resources for this? I've also seen Riak come up as an alternative. Do they work similarly?
- witnesser 4y agoThe only person stand out to witness a use case is a adserverer, I read the 1st 100lines of comments. It is like california highway system particularly when I witnessed, the billboard is very outstanding. The jams an pits, people are very nice to them.
- witnesser 4y agoThe above is just random comment. So I have a long time question, how cache miss is handled.
- rfrey 4y agoA question so noob I'm almost shy to ask it: The simplest scenario in the article is a single Redis instance residing on the same machine as the application. What's the benefit to this versus just storing data directly within the application?
- louissm_it 4y agoStoring the data directly inside the application still means you need to store it somewhere, likely a SQL database (such as PostgreSQL). These databases are insanely well engineered and very very fast, but compared to a key value store such as Redis and Memchached they are comparatively slow and resource hungry (because they are optimized for different things). So if you can fetch some cached data from a Redis key, even if on the same machine, it will cost you significantly less than querying a relational database.
- halukakin 4y agoNot all applications can store data out of the box. For instance some ways of PHP have embedded caches, some others don't have cache by default and you would need to install cache software (for instance apcu). Also, redis has many different types of data. For instance coding something similar to its "hash" data type is not trivial.
- ok123456 4y agoShort lived processes/workers.
- radicalriddler 4y agoRedis persists on disk (well, it's optional), if you restart your server I'd assume that it'd be able to restore the disk data into memory, versus your applications memory, which would just be lost. I'm not a Redis user, but that's based on what I've read
- piaste 4y agoYour application and runtime are probably tuned to act as servers, with short-lived requests and little or no persistent stage, and they may not play well with keeping a bunch of persistent data around forever. I personally first reached for Redis when I needed to asynchronously process a bunch of JSON uploaded by clients via POST. I initially just stuck them in a ConcurrentQueue in memory, but no matter how much I fiddled with HostedServices and BackgroundWorkers and whatever the MS documentation recommended, the ASP.NET Core app would occasionally 'lose' that queue before it could be consumed (or the consuming loop would get stuck, with the same result). You are also probably running your app on a pretty high-level language, with bytecode and reflection and all that nice stuff - if not even an interpreted language - while Redis is raw C code and will outperform your homebrew double-linked list or hash set.
- googletron 4y agoI wrote a little post on how Redis works and its various setups! How does everyone setup Redis? Elasticache is a good answer too :P
- tpmx 4y ago[Potentially inaccurate content removed by author]
- _gmnw 4y agoThe saltiness isn't a good look here. Especially seeing as he's not the poster. It's the HN algorithm which is probably due to the fact that other posts from his domain have done relatively well, plus the actual poster here has quite a bit of karma.
- tpmx 4y ago
- xnorswap 4y agoIt's a new article so it's relatively easy to explain: HN automatically combines submissions so that subsequent submissions count as upvotes for the first submission. If a popular source posts a new article, users will "rush" to post it to HN to reap that sweet karma and the winner will "catch" the upvotes of the others.
- tpmx 4y agoThat could explain it. Thanks! Is this behavior documented anywhere on news.ycombinator.com?
- mindcrime 4y agoThere isn't a ton of documentation per-se about HN behavior. There is: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html and a handful of posts by dang, sama, pg, etc. over the course of the years. most of the rest is what long-time users have just figured out through observation. There's a Git repo[1] out there that aggregates a lot of that stuff, but keep in mind that it's technically unofficial. That said, I think most of what's there is widely considered to be correct. [1]: https://github.com/minimaxir/hacker-news-undocumented https://github.com/minimaxir/hacker-news-undocumented
- _gmnw 4y agoReally love this style of writing. Pairing the diagrams/illustrations with the easy to grok copy is really helpful for folks like myself who have been mainly focused on the front-end. What tool do you use for your diagramming, is it all hand-drawn?
- dsmmcken 4y agoFont for the handwriting is Skippy Sharp, incase anyone else was wondering.
- googletron 4y agoIts hand drawn with some fonts for the titles.
- stjohnswarts 4y agoThis is what I do for my presentations (on a wacom of course). I have gotten grief over it, but I work faster and get less distracted by the eccentricacies of powerpoint and figma. My handwriting is abysmal so I will do that in a "handwriting font" to sort of look hand drawn. Even if I have to convert them later for some big wig, at least I have my "rough draft". Plus it all feels a little more human.
- dsmmcken 4y agoBeautifully presented, worth reading just for the illustrations.
- xnorswap 4y agoI'm not too familiar with redis and this may well help, so thank you. I see some data-types on the right. It surprises me that redis doesn't have a numeric data type. I understand that at its heart it is just a key-value store and doesn't ever need to do range-based lookup but it still surprises me. One consequence of "everything is a string" I've run into (although probably a sign I'm "doing it wrong"), is serialisation overhead in the client. If redis is expecting strings then it's left to the client to choose an appropriate serialisation which can have either performance or other pitfalls.
- voxic11 4y agoNumbers in redis can be natively represented using BITFIELDS. > BITFIELD player:1:stats SET u32 #0 1000 1) (integer) 0 > BITFIELD player:1:stats INCRBY u32 #0 -900 1) (integer) 100 > BITFIELD player:1:stats GET u32 #0 1) (integer) 100
- xnorswap 4y agoOK, that's helpful thank you. That said, all the keys themselves are still strings and therefore you can't have a SET of numbers or bitfields.
- morelisp 4y agoHow would a native number type avoid some serialization overhead that using e.g. 4 byte BE keys yourself must pay?
- deleted 4y ago[deleted]
- googletron 4y agoMy personal nightmare happened and accidentally published a rough draft! It has since been updated! Apologies!
- lalwanivikas 4y agoI haven't read it, but the presentation is really beautiful! Is it possible to make it printer friendly by any chance? Default browser PDF is a mess. Long form content like this is much easier to read printed.
- theden 4y agoThis is great, the visual explanations work really well One thing that threw me off is that it says for an SSD a random read is 150μs, but 1MB sequential read is 1ms? Shouldn't sequential reads be faster, or are two different read sizes being compared or something? If so, the ambiguity may confuse some people to think random reads are faster
- sharikous 4y agoMy interpretation is that 150us is the minimum latency no matter what for any size, since the seek time is provided for comparison for HDs
- darkcha0s 4y agoWell I'm guessing that it's referring to the fact that 1MB sequential is essentially a bunch of random reads? AFAIK, on SSD's there is no concept/guarantee that blocks are adjacent, so a sequential read is just a bunch of random reads.
- jasonwatkinspdx 4y agoThe way the Flash Translation Layer works is complicated, but long story short, there's still an advantage to sequential reads and writes on SSDs. The difference in latency and throughput isn't as dramatic as with spinning disks, but is still there. Random vs sequential writes have big implications for the long term health and performance of the SSD.
- ptbg 4y agoI love this style of post. You cover a wide range of topics in an easy to understand way. Keep up the great work!
- groffee 4y agoIt's a good article, but a couple of hopefully constructive points 1, .toc-wrap covers the image on desktop 2, the image is way too busy, there's too much going on
- xnorswap 4y ago> Send 1KB over a 1GBps network This is said to have a 10μs latency in the chart. But I'm fairly sure that is a calculation of bandwidth based on 1KB / 1GBps 10μs is about 3Km, so at most a 1.5Km round-trip. For a chart labelled latency, I'm surprised to see bandwidth calculations included. Any network hop would actually have far greater latency, if nothing else because communication typically involves more than a single round-trip for acknowledgement, etc. It might be worth making it clear some of the numbers are about bandwidth not latency.
- foota 4y agoDistance is of course a factor, but at fixed distance size matters a lot, and most applications are at more or less a fixed latency.
- googletron 4y agoFair point! Will update! I think the focus was on pure line latency. Check out the more detailed post here. https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832
- btilly 4y agoI think a few more concrete use cases would help. First, a key limitation that every architect should pay attention. Redis reaches the limits of what you can do in well-written single-threaded C. One of those limits is that you really, really, *really* don't want to go outside of RAM. Think about what is stored, and be sure not to waste space. (It is surprisingly easy to leak memory.) Second, another use case. Replication in Redis is cheap. If your data is small and latency is a concern (eg happened to me with an adserver), then you can locate read-only Redis replicas everywhere. The speed of querying off of your local machine is not to be underestimated. And third, it is worth spending time mastering Redis data structures. For example suppose you have a dynamic leaderboard for an active game. A Redis sorted set will happily let you instantly display any page of that leaderboard, live, with 10 million players and tens of thousands of updates per second. There are a lot of features like that which will be just perfect for the right scenario.
- koolba 4y ago> One of those limits is that you really, really, really don't want to go outside of RAM. Think about what is stored, and be sure not to waste space. (It is surprisingly easy to leak memory.) You can have massive amounts of RAM these days. You’re sooner to hit big-O limits from bad architectural decisions than run out of memory. If you do get to that point you likely have enough value in your usage to justify scaling out further and sharding. > And third, it is worth spending time mastering Redis data structures. Bingo. The true secret to properly using Redis: understanding the big-O complexity of each operation (…and ensuring that none of your interactions are more than logarithmic).
- googletron 4y ago> understanding the big-O complexity of each operation (…and ensuring that none of your interactions are more than logarithmic). This is a good idea, maybe a prompt for another post.
- btilly 4y agoYou can have massive amounts of RAM these days. You’re sooner to hit big-O limits from bad architectural decisions than run out of memory. If you do get to that point you likely have enough value in your usage to justify scaling out further and sharding. Absolute disagreement. It is very easily to accidentally leak a few hundred MB per week in a busy Redis system. The code will look and work fine...at first. It is correspondingly hard to track down and clean up the leak a few months later. (Particularly if there are multiple such to track down.) Yes, you can go for years just buying larger and larger EC2 instances. But that will also come with a shocking price tag. I know of a number of organizations that this happened to. And pretty much every bad Redis story I hear about had this as a root cause. That is why I brought it up as an important consideration.
- anton96 4y agoVery interesting. This is leading me to think, using redis as the sole database is very tempting but the Ram requirement is making me think twice. Wouldn’t there be a database like redis that only stores the latest data into memory and keeps the rest in an AOF file ?
- remote-dev 4y agoNot to make this an ad, but you can actually do better with Redis Enterprise using Redis on flash (part of the flexible and annual plans). It stores hot data in RAM and "warm" data in flash. Here is a good 68s video on the subject: https://www.youtube.com/watch?v=hFQnhPstqLM https://www.youtube.com/watch?v=hFQnhPstqLM
- omarshammas 4y agoVery informative and love the illustrations. I'm building a new website and am using sidekiq for background job processing which relies on redis behind the scenes to store all the job data. I configured a high availability redis instance with `maxmemory-policy noeviction` to ensure no data is lost. The website is still in its infancy so not thinking about scale for the next little while but curious if you have any tips or gotchas to keep an eye out for. Thanks!
- googletron 4y agoI would ensure that the data size is managed if you hit the limits due to your policy Redis will stop responding to ensure the data it has isn’t lost. I would also turn on some sort of persistence for data recovery in case of catastrophic failure. Early on this is totally fine and I would setup some monitors in redis data size relative to memory and try to keep 20% overhead weird things start to happen when systems are memory constrained.
- topspin 4y agoI am thinking of using Redis as a lightweight queuing mechanism. An event source will MULTI a small amount of metadata as a hash and append a list. Event sinks will BLPOP the list and retrieve and delete the metadata key. One requirement is the events survive power loss. Is there anything inherently wrong with this? Gotchas? A mockup I've done works great so far.
- renonce 4y agoIn case the event sink crashes or the connection to Redis is lost, you could lose events. Redis Streams are better designed for use cases where more reliable delivery is needed and have a ton more features, though it comes with more complexity.
- topspin 4y agoI hadn't looked at streams yet. Thanks. This page from AWS about Redis streams goes exactly to my use case: Redis Streams and Message Queues: https://aws.amazon.com/redis/Redis_Streams_MQ/ https://aws.amazon.com/redis/Redis_Streams_MQ/
- CSDude 4y agoThere used to be disque by antirez, which died. https://github.com/antirez/disque https://github.com/antirez/disque
- topspin 4y agoSee my other reply where the disque author talks about exactly that.
- zo1 4y agoRabbitMQ. It's so cheap and easy to startup a super performant queuing broker with docker these days. And the libraries are all there, async ready and with established patterns. Closest to zero code you can get for this. You'll likely end up reimplementing all those patterns and support around them using redis. If you want something quick and easy and dirty, go with Redis. But switch to Rabbit when you start having to write a lot of handling and other code.
- jrm4 4y agoAs someone who doesn't code for a living but teaches it to mostly novices, this helps (because before this I had no clue what it was except that it had something to do with databases.) Typically for my courses we just use some flavor of SQL and call it a day (and that kind of spoils us because of how declarative it tends to be) -- roughly, what's the "explain like I'm 10" use case for Redis over something else? From what I'm seeing, it's mostly an "efficiency" thing?
- notyourday 4y agoBlazingly fast serialized access to the shared data structures over the network by multiple writers and readers.
- TheBlight 4y agoIf you're not super concerned about reliability but really need speed. That's when Redis really makes the most sense, IMO.
- hobs 4y agoI have worked at places where every page load hits the database, and we've scaled ok, mainly because it was b2b stuff. However a simple redis instance in front of the database serving as a readable cache changes the rules of the game significantly - depending on the complexity of your calculation and your end result subsequent "page loads" or whatever you are doing can be tens of thousands (or more) times as efficient, and if you decide to use an expensive database or a cloud database this can help you a lot. Eventually the hard part is you might have bugs in synchronizing the state of redis and your database, look to existing implementations for your stack instead of reinventing the wheel.
- avmich 4y agoRelational databases are optimized for typical operations over data structured in tables. So, joins and records. However sometime you want something simpler - like LIFO queue - and wouldn't mind to have is faster. Redis allows to have this - the variety of data structures it has is much bigger than with relational databases. They (Redis and RDBs) both have their uses, of course. Ideally you would structure your system to use one of them where appropriate according to data requests.
- Havoc 4y agoThe white cube in the traditional usage example - what does that represent? App code? Or so that cache miss to db implemented in some standardized way?
- thecosmicfrog 4y agoExcellently written and illustrated! We've just added Redis to our platform so guides like this are a fantastic resource. Thanks!
- didip 4y agoBy the way, if you want a painless multi master Redis, simply install KeyDB. I am not affiliated, just a happy user.
- tunesmith 4y agoIn what cases would one want to pick memcached instead of redis?
- redis_telesign 4y ago
- claytn 4y agoIt's not clear to me why it makes sense to use both RDB Files and AOF on the same Redis instance. Seems like AOF would always be the more accurate source of truth here. What am I missing? Great article though!