ryanworl

7 ms·

1.
The Hitchhiker's Guide to Disaster Recovery and Multi-Region Kafka(warpstream.com)
5 points by ryanworl 1y ago|0 comments
2.
by ryanworl 1y ago
Does this trick preclude the ability to sort your data within a partition? You wouldn’t be able to rely on the row IDs being sequential anymore to be able to just refer to a prefix of them within a newly created file.
3.
by ryanworl 1y ago
This is a well-known class of optimization and the literature term is “late materialization”. It is a large set of strategies including this one. Late materialization is about as old as column stores themselves.
4.
Taking Out the Trash: Garbage Collection of Object Storage at Scale(warpstream.com)
7 points by ryanworl 1y ago|0 comments
5.
by ryanworl 1y ago
The equivalent to this feature is one of my favorite parts of Husky, Datadog’s storage and query system for event data. https://youtu.be/mNneCaZewTg?si=N68fsBlYS3tuvLe3 begins at 34:32
6.
Multiple Regions, Single Pane of Glass(warpstream.com)
20 points by ryanworl 2y ago|3 comments
7.
by ryanworl 2y ago
We're still drafting our next post in this series, but the answer is actually very simple: two tiers of object storage do not have the same drawbacks as a combination of object storage and local disk. We wanted to explain that in this
8.
by ryanworl 2y ago
This strategy will not work well for Apache Kafka because it is extremely IOPS hungry if you have more than a few partitions, and a replay of a large topic will require lots of IO bandwidth. It would work well e.g. a columnar database where
9.
by ryanworl 2y ago
(WarpStream co-founder here) We're not talking about no disks as in no storage, just nothing other than object storage. This does have a latency trade-off, but with the advent of S3 Express One Zone and Azure's equivalent high-per
10.
The Original Sin of Cloud Infrastructure(warpstream.com)
28 points by ryanworl 3y ago|2 comments
11.
Kafka as a KV Store: deduplicating keys with just 128 MiB of RAM(warpstream.com)
16 points by ryanworl 3y ago|0 comments
12.
by ryanworl 3y ago
Co-founder of WarpStream here. We wrote a short article explaining how the economics of this new storage class work for a system like ours: https://www.warpstream.com/blog/s3-express-is-all-you-need
13.
S3 Express Is All You Need(warpstream.com)
163 points by ryanworl 3y ago|85 comments
14.
by ryanworl 3y ago
WarpStream co-founder here. Implementing the Idempotent Producer feature for the Apache Kafka protocol was definitely a fun challenge for us. Please let me know if you have any questions!
15.
by ryanworl 3y ago
[WarpStream co-founder] WarpStream doesn't implement compacted topics today. It is on our roadmap, though. Compacted topics are typically not used in high-throughput workloads, so our plan is to delay compactions for longer than a disk
16.
by ryanworl 3y ago
[WarpStream CTO here] WarpStream flushes after 4MiB of data or a configurable amount of time. Flushes can also happen concurrently. In general, we'd prefer to not introduce many knobs. We're running a realistic throughput testing
17.
by ryanworl 3y ago
[WarpStream CTO here] I'm not going to respond to your comment directly (we've already solved all the problems you've mentioned), but I thought I should mention for the sake of the other readers of this thread that you work f
18.
by ryanworl 3y ago
We're aiming for per-GB usage-based pricing that is significantly cheaper than the alternatives, but the BYOC model combined with our extremely efficient cloud control plane gives us a lot of flexibility here. That's why we can of
19.
by ryanworl 3y ago
[WarpStream co-founder here] Please sign up for our mailing list! We'll have some interesting things to announce related to this soon.
20.
by ryanworl 3y ago
With an appropriately configured client (i.e. one that retries and waits for requests to be acknowledged), another Agent would receive the retry and the event would be written to the topic at that time.
21.
by ryanworl 3y ago
[WarpStream co-founder here] We've done lots of customer research here and, combined with the experience my co-founder and I have, we can confidently say most Kafka users (especially high-throughput users) would happily make a trade of
22.
by ryanworl 3y ago
[WarpStream co-founder here] My co-founder and I worked at Datadog for over 3 years where we built Husky, an event storage and query system built directly on top of S3 as well. We know what we're doing here, I promise ;) https:/&
23.
by ryanworl 3y ago
[WarpStream CTO here] WarpStream is Kafka protocol compatible, so we do support topic-partitions and consumer groups. We do not expose support for transactions or idempotent producing today, but the internals of the system support that and
24.
by ryanworl 3y ago
(WarpStream CTO here) This is a bit subtle I will admit, but we view the Kafka protocol as a successor here because it will outlive Kafka the implementation.
25.
by ryanworl 3y ago
[WarpStream co-founder here] That is correct about flushing. RE: consuming. The TLDR; is that the agents in an availability zone cluster with each other to form a distributed file cache such that no matter how many consumers you attach to a
26.
by ryanworl 3y ago
[WarpStream co-founder here] What do you think would be a good limit for the free plan? This isn't actually an architectural constraint for us. We just didn't want to promise unlimited usage forever so we picked a somewhat arbitra
27.
by ryanworl 3y ago
[WarpStream co-founder and CTO here] 1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the num
28.
by ryanworl 3y ago
I'm Ryan Worl, co-founder and CTO of WarpStream. We're super excited to announce our Developer Preview of our Kafka protocol compatible streaming system built directly on top of S3 with no stateful disks/nodes to run, no reba
29.
by ryanworl 3y ago
FoundationDB is used at Datadog as the metadata store for Husky, the storage and query engine powering a significant number of Datadog products, such as logs, network performance monitoring, and trace analytics. 1. https://www.da
30.
Husky: Exactly-Once Event Ingestion and Multi-Tenancy at Scale at Datadog(datadoghq.com)
22 points by ryanworl 4y ago|0 comments

More ›