danthelion

23 ms·

1.
FOMO-Driven Development(estuary.dev)
2 points by danthelion 2mo ago|0 comments
2.
Exactly-Once Delivery Is a Spectrum, Not a Checkbox: Part 1(medium.com)
3 points by danthelion 3mo ago|0 comments
3.
Why Is Everyone Buying Change Data Capture?(estuary.dev)
2 points by danthelion 1y ago|0 comments
4.
I Changed My Mind: AI Will Replace Us(medium.com)
14 points by danthelion 1y ago|20 comments
5.
HTAP: Still the Dream, a Decade Later(medium.com)
3 points by danthelion 1y ago|0 comments
6.
A New Reference Architecture for Change Data Capture (CDC)(estuary.dev)
4 points by danthelion 1y ago|0 comments
7.
Show HN: Interactive Change Data Capture (CDC) Playground(change-data-capture.com)
2 points by danthelion 1y ago|0 comments
8.
Fast JSON Processing in Real-Time Systems: Simdjson and Zero-Copy Design(estuary.dev)
22 points by danthelion 2y ago|2 comments
9.
by danthelion 2y ago
At Estuary, we’re creating a real-time data streaming platform that doesn’t rely on Kafka and uses JSON as the primary data format stored in an object storage. Many people are interested in how we achieve millisecond-level latency in our da
10.
Welcome to the Data Engineering Dream Factory(arecadata.com)
3 points by danthelion 2y ago|0 comments
11.
Apache Iceberg: The Hadoop of the Modern Data Stack?(medium.com)
2 points by danthelion 2y ago|0 comments
12.
Data Is Only Useful Outside of the Database(medium.com)
11 points by danthelion 2y ago|1 comments
13.
by danthelion 2y ago
Imagine a system monitoring payment transactions. Each transaction stream (e.g., purchase events) could be joined with customer account data (e.g., past purchasing patterns or blacklist flags). Streaming joins enable flagging potentially fr
14.
by danthelion 2y ago
Materialize is powered by timely dataflow and differential dataflow, Estuary Flow uses a streaming mapreduce architecture: https://estuary.dev/why-mapreduce-is-making-a-comeback/
15.
Streaming joins are hard(estuary.dev)
117 points by danthelion 2y ago|57 comments
16.
by danthelion 2y ago
Gazette is at the core of Estuary Flow ( https://estuary.dev ), a real-time data platform. Unlike Kafka, Gazette’s architecture is simpler to reason about and operate. It plays well with k8s and is backed by S3 (or any object stor
17.
Gazette: Cloud-native millisecond-latency streaming(github.com)
139 points by danthelion 2y ago|52 comments
18.
by danthelion 2y ago
With the 2-pass strategy, we can write arbitrary row group sizes while using a fixed amount of memory, with probably 100-200 MiB of overhead for the parquet file processing, depending on how large the metadata is for the scratch file. witho
19.
Memory Efficient Data Streaming to Parquet Files(estuary.dev)
28 points by danthelion 2y ago|2 comments
20.
Change data capture done correctly(estuary.dev)
2 points by danthelion 2y ago|0 comments
21.
A Mental Framework for Side Projects(arecadata.com)
1 points by danthelion 2y ago|0 comments
22.
Running Doom on Snowflake(arecadata.com)
2 points by danthelion 3y ago|0 comments
23.
SQL for Google Sheets with DuckDB(arecadata.com)
3 points by danthelion 3y ago|1 comments
24.
What I would do differently getting into Data Engineering [in 2024](arecadata.com)
1 points by danthelion 3y ago|0 comments
25.
by danthelion 3y ago
I wrote git-genie to automate commit message writing with GPT & pre-commit hooks, works surprisingly well (most of the time) - https://github.com/danthelion/git-genie
26.
Show HN: git-genie: pre-commit hook that writes commit messages for you(github.com)
4 points by danthelion 3y ago|1 comments
27.
by danthelion 3y ago
Git-genie can now be installed as a pre-commit hook which enables commit message generation automatically based on the staged changes using GPT.
28.
Data Engineering Sucks(arecadata.com)
1 points by danthelion 3y ago|0 comments
29.
Data Debt Is Not Evil: A Pragmatic Perspective(arecadata.com)
1 points by danthelion 3y ago|0 comments
30.
by danthelion 3y ago
Yes, that was precisely the motivation! (+ some interest in learning how to publish a Chrome extension).

More ›