chaotic-good

10 ms·

1.
by chaotic-good 10mo ago
I don't see any mentions of p99 latency in the benchmark results. Pushing gigabytes per second is not that difficult on modern hardware. Doing so with reasonable latency is what's challenging. Also, instead of using custom benchma
2.
by chaotic-good 2y ago
I tend to agree with this. The code tend to be loaded with primitives that express ownership semantics or error handling. Every time something changes (for instance, you want not just read but also modify values referenced by the iterator)
3.
by chaotic-good 3y ago
I'm not asking anything on behalf of any company and just genuinely curious (and I don't think that we're competitors, both systems are designed for totally different niches). I'm working on tiered-storage implementation
4.
by chaotic-good 3y ago
> Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? That's easy. EBS and similar solutions comes with the price. They're very expensive. Especially, when you need a
5.
by chaotic-good 3y ago
It could be easy to operate when everything is fine but what's about incidents? If I understand correctly, there is a metadata database (BTW, is it multi-AZ as well?). But what if there is a data loss incident and some metadata was lo
6.
by chaotic-good 3y ago
The disk thrashes because of fsyncs (Kafka doesn't perform any fsync's). But you can provision more disk space to mitigate this problem. And it looks like the test was set up this way to make Redpanda look worse.
7.
by chaotic-good 3y ago
You have to provision you disk space accordingly. NVMe needs some free space to have good performance. In this case I think that in Redpanda benchmarks the disk space was available and in case of benchmarks done by Confluent guy the system
8.
Analyzing 500B Rows Using Akumuli TSDB(akumuli.org)
1 points by chaotic-good 7y ago|0 comments
9.
Runtime Cost of Write-Ahead Logging(akumuli.org)
3 points by chaotic-good 8y ago|0 comments
10.
by chaotic-good 8y ago
Modern TSDB is expected to support tags. This means that every series will have a unique set of tag-value pairs associated with it. E.g. 'host=Foo OS=CentOS arch=amd64 ssdmodel=Intel545 ...'. And in the query you could pinpoint re
11.
by chaotic-good 8y ago
90% compression on time series is not viable unless you have some very specific dataset.
12.
by chaotic-good 8y ago
'metric_name text' is actually a tag-value list. Many TSDB's allows you to match data by tag. Each tag should be represented by a column in your example. Single table design will be prone to high read/write amplification
13.
by chaotic-good 8y ago
They can't handle high cardinality. Imagine having millions of columns in the column-oriented database (70% of those columns are updated every second). Imagine that you have to add new columns all the time. The main misconception about
14.
Using vector I/O optimize memory usage(akumuli.org)
7 points by chaotic-good 8y ago|0 comments
15.
Scaling TSDB-specific operations(akumuli.org)
2 points by chaotic-good 8y ago|0 comments
16.
Designing Inverted Index for the Time-Series Database(akumuli.org)
2 points by chaotic-good 9y ago|0 comments
17.
Storage engine design notes (LSM vs. B+tree)(akumuli.org)
2 points by chaotic-good 9y ago|0 comments
18.
by chaotic-good 9y ago
> Successful, popular concurrent platforms like Erlang/OTP, Nginx, and node.js, eschew threads in favor a single-threaded, async/non-blocking code model. All these platforms have some problems. E.g. Nginx is PITA if your proces
19.
by chaotic-good 9y ago
Yes, it was possible for a long time.
20.
by chaotic-good 9y ago
It's not a TSDB. It doesn't support ingestion without some fancy batching. The insert rate will be too limited without that. MergeTree was designed for batch processing, not for real-time processing.
21.
by chaotic-good 9y ago
JFK that the link doesn't work: the Russian government is going to create national big data operator. All internet companies must store their information there. They can use this information to generate ad-revenue but they should pay 2
22.
Silicon Valley's nightmares comes true in Russia(translate.google.com)
1 points by chaotic-good 9y ago|1 comments
23.
by chaotic-good 9y ago
What do you mean by gloss?
24.
by chaotic-good 9y ago
If I understand this right, with Intel's Optane you will eventually need to write everything to HDD because data collection happens at steady pace and the cache size is limited.
25.
by chaotic-good 9y ago
>> Maybe I'm getting this all wrong, but aren't the leaves also representing chunked data, which is compressed. Leaf nodes contain data from one series (this data should be read together) and SSTable with time-series data co
26.
by chaotic-good 9y ago
It's not that easy, actually. The simplest method that can utilize the full throughput of the drive is to use large writes (1MB or larger). This is the fastest possible way to write data to the SSD, period. This method also creates the
27.
by chaotic-good 9y ago
> I understand that was a micro-benchmark of one part of the system. The whole system is looking to be roughly in line with the Gorilla numbers. This makes sense now. I've found out that the compression algorithm performance numbers
28.
by chaotic-good 9y ago
Akumuli is desinged for an SSD and NVMe drives so I chose to have a lot of random reads and writes. My laptop's NVMe drive have a random write througphut around 400MB/s (AFAIR) and my most havy performance test wrote data at rate
29.
by chaotic-good 9y ago
1. Each leaf node is a fixed size block that contains compressed values and timestamps. 1000 values is just an example, number of values in one leaf node is variable. 2. Because there is a lot of data-structures. I'm using tree per ser
30.
by chaotic-good 9y ago
This is a one more example of the design in which one file holds many series and everything is chunked by time: - "there is no longer a single file per series but instead a handful of files holds chunks for many of them" - "W

More ›