rogermavis

6 ms·

1. 1.▲
   by rogermavis 1y ago

   To ensure I'm not speaking incorrectly (as I was going from memory), I grep'ed my several years' of databricks notes. Oh boy.. the memories came flooding back! We had 8 data engineers onboarding the org to databricks, it was
2. 2.▲
   by rogermavis 1y ago

   > you are comparing it here to something that's always on That's the point. Our org was told databricks would solve problems we just didn't have. Serverful has some wonderful advantages: simplicity, (ironically) cheaper (t
3. 3.▲
   by rogermavis 1y ago

   My stack atm is neovim, python/R, an EC2 and postgres (sometimes Sql Server). Some use of arrow and duckdb. For queries on less than few hundred GB this stack does great. Fast, familiar, the ec2 is running 24/7 so it's there
4. 4.▲
   by rogermavis 1y ago

   It provides central place to store and query data. A big org might have a few hundred databases for various purposes - databricks lets data engineers set up pipelines to ETL that data into databricks and when the data is there it can be que