hichkaker

6 ms·

1.
Datafold (YC S20) is hiring Engineer to build AI-powered data migration tools(ycombinator.com)
1 points by hichkaker 2y ago
2.
Modern Data Stack – open-source Edition(datafold.com)
3 points by hichkaker 2y ago|0 comments
3.
by hichkaker 3y ago
Thank you for mentioning Data Diff! Founder of Datafold here. We built Data Diff to solve a variety of problems that we encountered as data engineers: (A) Testing SQL code changes by diffing the output of production/dev versions of SQL
4.
by hichkaker 3y ago
As a data engineer, validating your SQL code as you develop pipelines is hard. It requires running the code to materialize a "development" dataset, then auditing the resulting dataset to ensure your code produces the correct outpu
5.
Show HN: Audit dbt models as you code(datafold.com)
3 points by hichkaker 3y ago|1 comments
6.
by hichkaker 4y ago
I did data engineering for 6 years and am building a company to automate SQL validation for dbt users. First, by “testing SQL pipelines”, I assume you mean testing changes to SQL code as part of the development workflow? (vs. monitoring pip
7.
Data-diff v0.3: DuckDB, efficient in-database diffing and more
26 points by hichkaker 4y ago|1 comments
8.
The Day You Stopped Breaking Your Data(datafold.com)
2 points by hichkaker 4y ago|0 comments
9.
by hichkaker 4y ago
As a data engineer myself, totally agree about the abuse of the word “data”. What we strived for when naming was to make it self-describing as much as possible. Since the tool does one thing - diff datasets - we could name it “dataset diff”
10.
Show HN: Data Diff – compare tables of any size across databases
127 points by hichkaker 4y ago|21 comments
11.
by hichkaker 4y ago
Datafold automates data pipeline testing for data engineers. With Datafold, data engineers can deal with data quality issues in the pull request by seeing how a change to source code impacts data produced throughout the entire data pipeline
12.
by hichkaker 4y ago
We actually built a tool exactly for that: https://www.datafold.com/data-diff The core use case is helping data engineers building analytical pipelines test the changes to their code prior to deploying, but can also work fo
13.
by hichkaker 4y ago
Datafold automates data pipeline testing for data engineers. With Datafold, data engineers can deal with data quality issues in the pull request by seeing how a change to source code impacts data produced throughout the entire data pipeline
14.
by hichkaker 4y ago
Datafold (YC S20) | https://www.datafold.com Datafold automates data pipeline testing for data engineers. With Datafold, data engineers can deal with data quality issues in the pull request by seeing how a change to source code
15.
by hichkaker 6y ago
We've been using Tella for product demos. A huge upgrade from hacking together QuickTime recordings!
16.
by hichkaker 6y ago
Thank you for sharing! I assume we are talking about analytical, not transactional data: > Diff'ing data is one of the weakest and most cumbersome ways to verify correctness. It depends on the use case: if the goal is to assess the
17.
by hichkaker 6y ago
Thank you for pointing it out, I've never noticed but now that you brought it up will do a few more iterations on our design to avoid the collision.
18.
by hichkaker 6y ago
Support for MS SQL is in progress. We can deploy in your DC using with Docker or K8S.
19.
by hichkaker 6y ago
If diffing datasets within the same physical database, generate SQL, execute in the database, analyze and render results. If diffing datasets across physically different databases, e.g. PostgreSQL <> Snowflake or 2 distinct MySQL serv
20.
by hichkaker 6y ago
Totally fair! We offer free trials for on-prem. Please ping us at hn@datafold.com to set it up.
21.
by hichkaker 6y ago
You raised a great point. Data testing methods can perhaps be broken down to two main categories: 1. "Unit testing" – validating assumptions about the data that you define explicitly and upfront (e.g. "x <= value < Y&qu
22.
by hichkaker 6y ago
Thank you for the feedback! Agree with you about the niche. Diff is our first tool that helps test changes in the ETL code , and the impact is correlated with the size and complexity of the codebase. Diff also provides us a wedge into the
23.
by hichkaker 6y ago
Thank you for the suggestion! We're leaning towards Option 1: free diffing for datasets < 1M rows. Option 2 seems a bit tricker since we are in a way creating a new tool category and it can be harder to convince someone to pay befor
24.
by hichkaker 6y ago
If you can materialize those .bcp files as tables, then yes, absolutely.
25.
by hichkaker 6y ago
Thank you for asking deep questions and providing specific examples! The degree of scale and complexity varies significantly between companies, so we definitely can't claim that every ETL developer should use such a tool. Philosophical
26.
by hichkaker 6y ago
> But will this problem be big enough for VC investment is the question? That's a great question. Thinking about where problems arise in data pipelines, there are fundamentally two moving pieces: 1) Your data – you're continuou
27.
by hichkaker 6y ago
So there are two main workflows: 1) Run diff as part of CI (on code change), that can be done via Github workflow (or other CI). The Github workflow takes in a YAML config file, and also is customizable so you can fit it according your need
28.
by hichkaker 6y ago
Yes, you can diff to physically distinct databases. MySQL support is in progress. There is a REST API that accepts the same parameters as in the UI and returns the diff results in JSON + optionally, materializes the tables in your DB with r
29.
by hichkaker 6y ago
Those tools are definitely vastly powerful. Have you used either of them? TBH, I haven't, but judging from our current post-Informatica users and by reading questions on Informatica/Talend official user forums, I concluded that th
30.
by hichkaker 6y ago
Thanks! We are going to add an example to our website and perhaps publish it as a Github App. The flow is: For every whitelisted (say, SQL) file changed by a pull request, our bot compares the production data (produced by the master branch)

More ›