abrazensunset

4 ms·

1.
by abrazensunset 3y ago
This has loose overlap with: - Materialize - Flink SQL - Arroyo - Readyset - RisingWave - Timeplus - Pathway - Dozer - ReadySet - Snowflake dynamic tables - Native materialized views in OLTP databases - Just having a stack of views in your
2.
by abrazensunset 3y ago
Agree with this. Snowflake has best-in-class dev experience and performance for Spark-like workloads (so ETL or unconstrained analytics queries). It has close to worst-in-class performance as a serving layer. If you're creating an envi
3.
by abrazensunset 3y ago
Please don't do this. If you want an Airflow-ish approach without punishing your future self, pick Prefect. Otherwise go with Temporal. Above all do not adopt Airflow for the use cases you describe in 2023
4.
by abrazensunset 3y ago
If you use Julia, Makie crushes this use case and comes with great Python interop. https://github.com/holoviz/datashader is a good one in the Python ecosystem.
5.
by abrazensunset 4y ago
What distinguishes this from the more well-known OpenMetadata project?
6.
by abrazensunset 4y ago
"Lakehouse" usually means a data lake (bunch of files in object storage with some arbitrary structure) that has an open source "table format" making it act like a database. E.g. using Iceberg or Delta Lake to handle dele
7.
by abrazensunset 4y ago
There is a huge round of "data observability" startups that address exactly this. As a category it was overfunded prior to the VC squeeze. Some of them are actually good. They all have various strengths and weaknesses with respect
8.
by abrazensunset 4y ago
I think it's more a matter of comparing minivans (cloud "DWH" engines) to sports cars (Clickhouse et al) here. Snowflake's performance characteristics & ops paradigm have always been more consistent with managed Spar
9.
by abrazensunset 4y ago
"I want to write my orchestration in Python and I'm comfortable hosting my own compute" -> Prefect (lightweight) or Dagster (heavier but featureful) "My team already knows Airflow and/or I want to pay Astronomer
10.
by abrazensunset 4y ago
I'm a heavy Prefect user and was also very confused about the initial rewrite, even after reading several summaries. My best advice is to just try using 2.0 (Orion). Here's how I'd summarize the difference: Prefect 1.0 feels
11.
Ask HN: What does your team use to sync and manage secrets?
4 points by abrazensunset 4y ago|2 comments
12.
by abrazensunset 4y ago
+1, the Pandas API is somewhere between mediocre and bad, and results in garbage code unless you use it in a carefully constrained way (which is admittedly true of many complete languages, much less libraries that organically evolved severa
13.
by abrazensunset 5y ago
It's worth noting that hydraulic fracturing itself is rarely the problem. Issues come from moving fluid volumes somewhere else during production: subsidence due to extraction from the place where the hydrocarbons are, or (most often) f
14.
by abrazensunset 5y ago
+1, Migadu is simple, reliable, easy, and just works. Cost is low enough it might as well be free. I've only ever had one issue (from incorrectly interpreting documentation) and had an actual human respond and walk me through it. The o
15.
by abrazensunset 6y ago
In that situation (dual usage modes) I think I'd rather have the primary data store be Materialize, and just snapshot Materialize views back to your warehouse (or even just to an object store). Then you could use that static store for
16.
by abrazensunset 6y ago
In my experience (dependency-heavy data engineering & ML), Poetry is unbearably slow[^1]. Great interface/workflow, though. [1] https://github.com/python-poetry/poetry/issues/2094
17.
by abrazensunset 6y ago
Anyone here with practical experience using [Flying Squid]( https://github.com/HazyResearch/flyingsquid ) over the open-source Snorkel library? I'm curious if this platform re-uses some of that line of research or i