13 ms·

the 600ns figure represents our optimized write path and not a full fsync operation. we achieve it -among other things- through: 1- as mentioned, we are not us

the 600ns figure represents our optimized write path and not a full fsync operation. we achieve it -among other things- through: 1- as mentioned, we are not using any traditional filesystem and we're bypassing several VFS layers. 2- free space management is a combination of two RB trees, providing O(log n) for slice and O(log n + k) - k being the number of adjacent free spaces for merge. 3- majority of the write path employs a lock free design and where needed we're using per cpu write buffers the transactional guarantees we provide is via: 1- atomic individual operations with retries 2- various conflict resolution strategies (timestamp, etc.) 3- durability through controlled persistence cycles with configurable commit intervals depending on the plan, we provide persistence guarantee between 30 sec to 5 minutes

dangoodmanUT 2y ago
I didn't necessarily mean exactly fsync. I guess I'll ask: Is it actually flushed to persistent disk in 600ns such that if the node crashes, the data can always be read again? Or does that not fully flush?
- mehrant 2y ago
  yes, in that case data can potentially be lost. 30 sec in a worse case scenario without HA.
  - gkbrk 2y ago
    There's a tiny 50,000,000x difference between the now admitted 30 seconds and the previously claimed 600 nanoseconds.
  - dangoodmanUT 2y ago
    So it's not actually persistence then. That's extremely deceptive, and (IANAL) I think false advertisement. I'd clarify it. That's also not HA, that's durability. Concerning.
    - pclmulqdq 2y ago
      I have a product to sell you with a postgres interface but p99 write latency of 100 nanoseconds. It's postgres but our driver says "write done" before a write completes. It's revolutionary!
dangoodmanUT 2y ago
And hold on, 600ns can't possibly be right... A memory copy plus updating what ever internal memory structures you have is definitely going to be over 1us. Even a non-fsync NVMe write is still >=1us, so this is grossy misleading.
- mehrant 2y ago
  our p50 is indeed 600ns for write, the way I explained it. I understand that at this point, this can be read as "trust me bro" kind of statement, but I can offer you something. we can have a quick call and I provide you access to a temp server with HPKV installed on it, with access to our test suit and you'll have a chance to run your own tests. this can be a good learning opportunity for both of us (potentially more for us) :) if you're interested, please send us an email to support@hpkv.io and we can arrange that
  - mehrant 2y ago
    for the time being, have a look at this please: http://hpkv.io/videos/performance_local.webm http://hpkv.io/videos/performance_local.webm this is 1M records, 3M operations on a single node, single thread, recorded in real time (1x). I understand that without access to the source of test program it's hard to trust, but we can arrange that if you decided to take on that call :)
    - pclmulqdq 2y ago
      The question from most of us isn't "did you get that number," it's "what does that number actually mean?" Writes don't need to return any data, so you can sort of set that latency number arbitrarily by changing the meaning of "write done." I can make "redis with 0 write latency" by returning a "write done" immediately after the packet lands, but then the meaning of "write done" is effectively nil. In every persistent database, that number indicates that an entry was written to a persistent write-ahead log and that the written value will stay around if the machine crashes immediately after the write. Clearly you don't do this because it's impossible to do in 600 ns. For most of the non-persistent databases (eg redis, memcached), write latency is about how long it takes for something to enter the main data structure and become globally readable. Usually, "write done" also means that the key is globally readable with no extra performance cost (ie it was not just dumped into a write-ahead log in memory and then returned). In a world where you spoke about the product more credulously or where code was open-source, I might accept that this was the case. As it stands, it looks like: 1. This was your "marketing gimmick" number that you are trying to sell (every database that isn't postgres has one). 2. You got it primarily by compromising on the meaning of "write done," and not on the basis of good engineering.
pclmulqdq 2y ago
Wait, "depending on the plan"? You're already monetizing your non-persistent non-database?

dangoodmanUT 2y ago

I didn't necessarily mean exactly fsync. I guess I'll ask: Is it actually flushed to persistent disk in 600ns such that if the node crashes, the data can always be read again? Or does that not fully flush?

mehrant 2y ago
yes, in that case data can potentially be lost. 30 sec in a worse case scenario without HA.
- gkbrk 2y ago
  There's a tiny 50,000,000x difference between the now admitted 30 seconds and the previously claimed 600 nanoseconds.
- dangoodmanUT 2y ago
  So it's not actually persistence then. That's extremely deceptive, and (IANAL) I think false advertisement. I'd clarify it. That's also not HA, that's durability. Concerning.
  - pclmulqdq 2y ago
    I have a product to sell you with a postgres interface but p99 write latency of 100 nanoseconds. It's postgres but our driver says "write done" before a write completes. It's revolutionary!

dangoodmanUT 2y ago

And hold on, 600ns can't possibly be right... A memory copy plus updating what ever internal memory structures you have is definitely going to be over 1us. Even a non-fsync NVMe write is still >=1us, so this is grossy misleading.

mehrant 2y ago
our p50 is indeed 600ns for write, the way I explained it. I understand that at this point, this can be read as "trust me bro" kind of statement, but I can offer you something. we can have a quick call and I provide you access to a temp server with HPKV installed on it, with access to our test suit and you'll have a chance to run your own tests. this can be a good learning opportunity for both of us (potentially more for us) :) if you're interested, please send us an email to support@hpkv.io and we can arrange that
- mehrant 2y ago
  for the time being, have a look at this please: http://hpkv.io/videos/performance_local.webm http://hpkv.io/videos/performance_local.webm this is 1M records, 3M operations on a single node, single thread, recorded in real time (1x). I understand that without access to the source of test program it's hard to trust, but we can arrange that if you decided to take on that call :)
  - pclmulqdq 2y ago
    The question from most of us isn't "did you get that number," it's "what does that number actually mean?" Writes don't need to return any data, so you can sort of set that latency number arbitrarily by changing the meaning of "write done." I can make "redis with 0 write latency" by returning a "write done" immediately after the packet lands, but then the meaning of "write done" is effectively nil. In every persistent database, that number indicates that an entry was written to a persistent write-ahead log and that the written value will stay around if the machine crashes immediately after the write. Clearly you don't do this because it's impossible to do in 600 ns. For most of the non-persistent databases (eg redis, memcached), write latency is about how long it takes for something to enter the main data structure and become globally readable. Usually, "write done" also means that the key is globally readable with no extra performance cost (ie it was not just dumped into a write-ahead log in memory and then returned). In a world where you spoke about the product more credulously or where code was open-source, I might accept that this was the case. As it stands, it looks like: 1. This was your "marketing gimmick" number that you are trying to sell (every database that isn't postgres has one). 2. You got it primarily by compromising on the meaning of "write done," and not on the basis of good engineering.

pclmulqdq 2y ago

Wait, "depending on the plan"? You're already monetizing your non-persistent non-database?