5 ms·

Has anyone seen max (p100) client latencies of 300 to 400ms but totally normal p99? We see this across almost all our redis clusters on elasticache and have no

Has anyone seen max (p100) client latencies of 300 to 400ms but totally normal p99? We see this across almost all our redis clusters on elasticache and have no idea why. CPU usage is tiny. Slowlog shows nothing.

secondcoming 3y ago
Is it doing backups?
- jontonsoup 3y ago
  My understanding is elasticache does not let you turn them off.
  - secondcoming 3y ago
    That would be surprising, have you tried with CONFIG SET xyz ?
    - jontonsoup 3y ago
      Running from the client side says that the config command doesn’t exist. Not sure how to run from the server side on elasticache.
GauntletWizard 3y ago
I would guess your problem is probably scheduler based. The default(ish) Linux scheduler operates in 100ms increments, the first use of a client takes 3-4 round-trips. TCP opens, block, request is sent, the client blocks on write, the client attempts to read and blocks on read. If CPU usage is high momentarily, each of these yields to another process and your client isn't scheduled for another 100ms
- jontonsoup 3y ago
  Hmm. We have super low CPU utilization- something like 9%. This is also across 10+ different clusters.
  - jontonsoup 3y ago
    We also pool our clients heavily. Maybe we could reduce the new connections to zero to test.
tayo42 3y ago
Is the memory full and evicting? Or do you have a large db with lots of keys with ttls? Redis does a bunch of maintenance stuff on the same thread iirc in the background but not really
- jontonsoup 3y ago
  Memory is maybe 50% full. We are totally over provisioned. We actually just downsized and it didn’t impact anything. We do expire but we don’t think we have a thundering herd problem with them all happening at the same time.
nicwolff 3y ago
Are you evicting or deleting large sets (or lists or sorted sets)? We use a Django ORM caching library that adds each resultset's cache key to a set of keys to invalidate when that table is updated – at which point it issues `DEL <set key>` and if that set has grown to hundreds of thousands – or millions! – of keys the main Redis process will block completely for as long as it takes to loop through and evict them.
- jontonsoup 3y ago
  nope!

secondcoming 3y ago

Is it doing backups?

jontonsoup 3y ago
My understanding is elasticache does not let you turn them off.
- secondcoming 3y ago
  That would be surprising, have you tried with CONFIG SET xyz ?
  - jontonsoup 3y ago
    Running from the client side says that the config command doesn’t exist. Not sure how to run from the server side on elasticache.

GauntletWizard 3y ago

I would guess your problem is probably scheduler based. The default(ish) Linux scheduler operates in 100ms increments, the first use of a client takes 3-4 round-trips. TCP opens, block, request is sent, the client blocks on write, the client attempts to read and blocks on read. If CPU usage is high momentarily, each of these yields to another process and your client isn't scheduled for another 100ms

jontonsoup 3y ago
Hmm. We have super low CPU utilization- something like 9%. This is also across 10+ different clusters.
- jontonsoup 3y ago
  We also pool our clients heavily. Maybe we could reduce the new connections to zero to test.

tayo42 3y ago

Is the memory full and evicting? Or do you have a large db with lots of keys with ttls? Redis does a bunch of maintenance stuff on the same thread iirc in the background but not really

jontonsoup 3y ago
Memory is maybe 50% full. We are totally over provisioned. We actually just downsized and it didn’t impact anything. We do expire but we don’t think we have a thundering herd problem with them all happening at the same time.

nicwolff 3y ago

Are you evicting or deleting large sets (or lists or sorted sets)? We use a Django ORM caching library that adds each resultset's cache key to a set of keys to invalidate when that table is updated – at which point it issues `DEL <set key>` and if that set has grown to hundreds of thousands – or millions! – of keys the main Redis process will block completely for as long as it takes to loop through and evict them.

jontonsoup 3y ago
nope!