chenyang

7 ms·

1.
by chenyang 2y ago
Please checkout automq: https://github.com/AutoMQ/automq AutoMQ is a cloud-first alternative to Kafka by decoupling durability to S3 and EBS. 10x cost-effective. Autoscale in seconds. Single-digit ms latency.
2.
Vortex: A Stream-Oriented Storage Engine for Big Data Analytics [pdf](storage.googleapis.com)
1 points by chenyang 2y ago|0 comments
3.
by chenyang 2y ago
Hello, I am the author of this article, any questions are welcome about AutoMQ( https://github.com/AutoMQ/automq )
4.
Show HN: Apache Amoro is a Lakehouse management system built on iceberg(github.com)
7 points by chenyang 2y ago|1 comments
5.
by chenyang 2y ago
Just like warpstream or confluent freight cluster, but AutoMQ is source available, you may find the key s3 wal implementation here: https://github.com/AutoMQ/automq/blob/main/s3stream/src/main..
6.
by chenyang 2y ago
Here is a benchmark report for your reference: https://docs.automq.com/docs/automq-opensource/IJLQwnVROiS5c... Compared to Apache Kafka, here are some highlighted conclusions: * a 300-fold efficiency in partition
7.
by chenyang 2y ago
S3Stream is a shared streaming storage library that provides a unified interface for reading and writing streaming data to cloud object storage services like Amazon S3, Google Cloud Storage, and Azure Blob Storage. EBS is utilized here for
8.
EBS != Local Disk, EBS == Shared Storage(twitter.com)
22 points by chenyang 2y ago|0 comments
9.
Deep dive into the shared-storage of AutoMQ, vs. Kafka/Tiered Storage(twitter.com)
1 points by chenyang 2y ago|0 comments
10.
by chenyang 2y ago
In fact, EBS is entirely a cloud storage solution and operates as shared storage. It is not a local disk system.
11.
by chenyang 2y ago
In addition to the high cost of S3Express, utilizing warpstream to write three replicas to S3Express and later compacting them to S3Standard could result in quadruple network/outbound traffic costs. With two consumer groups involved, t
12.
The cloud-native technologies employed by AutoMQ(twitter.com)
2 points by chenyang 2y ago|0 comments
13.
by chenyang 2y ago
Can't agree more! S3 will be the modern data storage primitive. Also, the move towards shared storage and separating compute from storage is a key trend in cloud-native architecture, enhancing scalability and cost-efficiency.
14.
by chenyang 2y ago
You might be interested in our strategies for managing different types of failures: - In case of an EC2 instance failure, we take advantage of EBS's ability to be attached to multiple instances( https://docs.aws.amazon.com&#x
15.
by chenyang 2y ago
Yes, acknowledgments for writes occur once the data is committed to the EBS WAL, with each write operation bypassing the cache via Direct IO. Data is then asynchronously uploaded to S3. Given that EBS already ensures various levels of data
16.
by chenyang 2y ago
Indeed, operating Kafka can be challenging and complex due to its nature as a stateful and distributed system. However, AutoMQ has adopted a cloud-native architecture, offloading storage to EBS and S3( https://docs.automq.com&#x2F
17.
by chenyang 2y ago
Hi, thank you for your interest in AutoMQ. If you have any questions regarding cost-related data, here is an analysis report for your reference: https://docs.automq.com/docs/automq-s3kafka/EJBvwM3dNic6uYkZ...
18.
by chenyang 2y ago
Yes, thank you for the clarification. AutoMQ has replaced the topic-partition storage with cloud-native S3Stream ( https://github.com/AutoMQ/automq/tree/main/s3stream ) library, thereby harnessing the bene
19.
by chenyang 2y ago
Agree that Kafka is best suited for stream scenarios. However, we also see Kafka being extensively used in online business scenarios.
20.
by chenyang 2y ago
Indeed, an increasing number of open-source (or more accurately, source-available) software projects are either transitioning to the Business Source License (BSL) or adopting BSL from the outset. Examples include: 1. CockroachDB 2. Redpanda