dancrystalbeach

5 ms·

1.
by dancrystalbeach 23d ago
What is even DuckDB Labs???
2.
by dancrystalbeach 1mo ago
I’ve been watching this story play on repeat for some time now; the death of tools like Spark and Kafka has been heralded from the treetops for a decade now, yet here we are … living in a Spark and Kafka world. Someone once told someone els
3.
by dancrystalbeach 1mo ago
A few things have not changed in the age of AI, where the LLM has replaced 3/4 of the brain power of a whole generation of engineers who once found their meaning in the code they wrote. I mean, I hate to say it’s true, but a lot of fol
4.
by dancrystalbeach 2mo ago
I've seen this topic arise a few times, sometimes shouted from a Substack, other times running as a strong undercurrent among some unhappy technologist. Either way, it boils down to the same problem that an entire generation of softwar
5.
by dancrystalbeach 2mo ago
I mean, if the classic “relational model” for data was being written about in 1970, then we are indeed handling a topic that has withstood the sands and flying bytes of time and space. It makes one wonder if in another 50 years, on Mars, so
6.
by dancrystalbeach 2mo ago
Amazing, we can do something fun and novel with Rust ... without the use of any LLMs.
7.
by dancrystalbeach 2mo ago
I know calling Cloudflare the new kid on the block is a little tongue-in-cheek, but it seems fair when applied to Data Platform tooling. My guess is that when someone says "Cloudflare," the first thing that comes to mind isn'
8.
by dancrystalbeach 2mo ago
This is a strange feeling for me. Something new and exciting from AWS besides another mass layoff? What’s wrong with those people? It’s almost like they care or something. Definitely sus. If you’re like me, you’ve spent decades under the bi
9.
by dancrystalbeach 3mo ago
It's a race to the bottom, as per usual, inside the inner tech sanctum where legions of software grifters burn tokens in dark terminals, sending up a sweet-smelling aroma to the Sardaukar licking up the costly sacrifice of the faithful
10.
by dancrystalbeach 3mo ago
I'm going to tell your mom. I’m going to do it. You can’t stop me. And you, Grandma, for that matter. I’m calling them right now, like now. You’re an addict; you have to stop. Someone has to stop you. Might as well be me. It started ou
11.
by dancrystalbeach 4mo ago
I don’t know what it is about Apache Arrow, that GOAT of data engineering, that snuck in like a weasel through the backdoor, we all woke up one day and found out Arrow is the Atalas of the data world, holding up the systems we depend on and
12.
by dancrystalbeach 4mo ago
I hadn’t thought about it much lately, but depending on your point of view, Kafka is either at the height of its rise or on a slow downward spiral. Maybe both? We do live in the age of abstraction; businesses at large seem to be less willin
13.
by dancrystalbeach 4mo ago
If we had a dollar for every time someone came along to become the Apache Parquet killer, we would all be living on the side of a mountain tending to our alpacas. A boy can dream, can’t he?
14.
by dancrystalbeach 4mo ago
Yeah, so … I’ve heard rumbling and mumblings about, here and there. But I had yet to try it out for myself. I trust nothing I can’t put my hands on. Something about being raised in the cornfields of the Midwest, always be skeptical of anyth
15.
by dancrystalbeach 4mo ago
It’s hard to find the bright, shining stars amid the doom and gloom the tech world seems to be floundering in. When the going gets tough, I like to remind myself that there are lots of new and exciting tools released in the last few years,
16.
by dancrystalbeach 4mo ago
Apache Arrow entered the data scene quietly; for years, it languished in obscurity, unheard of and uncared for by the data community. Back in the olden days of 2022, which feels like another world, I was happily using and writing about Arro
17.
by dancrystalbeach 4mo ago
Ok, Spark isn’t dead. Before you leave, I’m sorry for lying to you. Sorta. Kinda. Not really. Undoubtedly Apache Spark has reached its zenith, shot like a rocket out of the Databricks barrel into the sky. The world is shifting though, even
18.
by dancrystalbeach 4mo ago
ypically, when the Harvard Business Review publishes something, especially techy, people tend to pay attention. Well, that is, unless it goes against the ultra-psyops-capitalism that drives most of the known world, in the form of extracting
19.
by dancrystalbeach 5mo ago
For years, I've wondered at the intricate designs of MAKE files, venvs, poetry, binaries, uvs, yml, and many other such spells that engineers through the ages have deemed necessary to provide a consistent runtime environment. Project t
20.
by dancrystalbeach 5mo ago
I’ve been wanting to hit on this little topic for a few months, but spring has sprung, and the woods and river are irresistible in their calling to me. Also, if I’m going to be poor and homeless, living in a van down by the river because AI
21.
by dancrystalbeach 5mo ago
Let’s keep moving and talk about the guiding principles of architectural design for the data platform. These principles have technical implications, but many are logical and theoretical and should inform our decisions when choosing tooling,
22.
by dancrystalbeach 5mo ago
Building data platforms that actually work and provide value has to be grounded in the day-to-day realities of the messy data systems you will encounter in the real world. Neither life nor the systems we will encounter exists in a vacuum. E
23.
by dancrystalbeach 5mo ago
Well, I had my moment, the tipping or boiling point, I guess you would call it. I’ve had enough, and, in a moment of fury, I ripped Polars from its Lambda throne and supplanted it with DuckDB. Been a long time coming.
24.
by dancrystalbeach 6mo ago
When you first consider a data platform as a whole, the architecture and infrastructure serve as the foundation upon which everything else is built. For our purposes, we can use the two words, architecture and infrastructure, interchangeabl
25.
by dancrystalbeach 6mo ago
Look, you don’t have to, but if you want to, here we go. Also, you should probably go find one of those elusive AI Engineers that are now showing up on LinkedIn full of wisdom and foresight. Maybe they know what’s going on. Either way, I’ve
26.
by dancrystalbeach 6mo ago
In this episode of Data Engineering Central, I sit down with Kevin, who spent 13 years working at Apple before walking away at the end of 2025.
27.
by dancrystalbeach 6mo ago
Sometimes I lean back in my chair, before work starts, and dream of the time long ago, before AI ruined everything. Back in the olden days when simplicity was > than complexity. It seems those days are long gone. People all of a sudden w
28.
by dancrystalbeach 7mo ago
Since time immemorial, data folks of all shapes and sizes have fought their way through a maze of GitHub repos, Confluence pages, and random scripts … in search of all the places that calculate x, y, or z.
29.
by dancrystalbeach 7mo ago
Well, you know I simply can’t resist throwing my hat in the ring. If everyone else is going to stand on a soapbox and decry the end of all data pipelines as we know it, you can be sure I’m going to climb to the top of the old oak tree and s
30.
by dancrystalbeach 7mo ago
I can see all you old-school SQL Server buggers just squirming in your seats, hot under the collar. How dare they speak of such sacred things, holy, without reference and a nod to the past? Methinks someone, somewhere, once said that there

More ›