sagiba

14 ms·

1.
by sagiba 2mo ago
Hi HN, I'm Sagi, founder of Varve ( https://varve.io ). For the past few months I've been working with S3 buckets in the Registry of Open Data on AWS. Some of them are enormous, with many billions of objects, and don&#x2
2.
Show HN: Swath – parallel S3 listing without knowing the key distribution(github.com)
1 points by sagiba 2mo ago|1 comments
3.
by sagiba 4mo ago
Looks like a big deal. Is this basically paving the way to free cross-cloud connectivity? I wonder why they would only do this for private networks and not for public traffic. For example, AWS is still not part of the Bandwidth Alliance [1]
4.
AWS Interconnect – multicloud now offers a free 500 Mbps tier(aws.amazon.com)
3 points by sagiba 4mo ago|2 comments
5.
How intelligent is S3 Intelligent-Tiering?(sagi.org)
3 points by sagiba 4mo ago|0 comments
6.
by sagiba 5mo ago
That's a clean architecture and the dataset abstraction isn't really needed when every file has a DB row and clear lifecycle. The post is more about the pipeline / ML / log / export world where ownership isn't
7.
by sagiba 5mo ago
Ha. The fact that teams reach for iceberg to organize things that aren't really tables is itself a symptom of needing better management tools for other types of data.
8.
by sagiba 5mo ago
Curious what your use cases look like. If you're storing data where you always know what's there, who created it, and whether it's still in use without needing to query for it, that's actually a great place to be. The po
9.
by sagiba 5mo ago
They solve it, partially, for tabular data. Delta, Iceberg, DuckLake are all table formats. And yeah, they do more than dataset abstraction (transactions, time travel, schema evolution). But that's just one slice of storage. Most teams
10.
by sagiba 5mo ago
You can have lots of buckets, but each one typically still contains many datasets. Think of a team doing ML, for example. They work with data all day across many different tools, each reading some inputs from S3 and writing outputs to S3. T
11.
by sagiba 5mo ago
Good point, prefixes are performance boundaries too, per-prefix rate scaling means you can spread load across prefixes to get aggregate throughput well above 3.5k RPS [1]. But that's a different thing than what the post is about. Even
12.
by sagiba 5mo ago
POSIX isn't the ask. Datasets are. The need to keep track of what data you have stored is universal, not my specific requirement.
13.
by sagiba 5mo ago
Agree it doesn't have to be part of S3 itself. My point is that there is a missing semantic layer. In practice, many teams use S3 directly without any layer on top. So without better organizational capabilities, they can't keep tr
14.
Buckets and objects are not enough(sagi.org)
19 points by sagiba 5mo ago|21 comments