tang8330

6 ms·

1. 1.▲
   Artie (YC S23) Is Hiring Software Engineers(jobs.ashbyhq.com)

   1 points by tang8330 2mo ago
2. 2.▲
   by tang8330 3mo ago

   Great question - there's no Debezium under the hood. Artie has its own Reader and Transfer layers, built from scratch. TOAST columns: Artie has automatic detection built in. If a TOAST column hasn't changed, its value won't a
3. 3.▲
   Show HN: Artie – Real-time data replication to your warehouse, now self-serve(artie.com)

   26 points by tang8330 3mo ago|6 comments
4. 4.▲
   Show HN: Artie – Real-time data replication to your data warehouse, self-serve(artie.com)

   3 points by tang8330 3mo ago|0 comments
5. 5.▲
   Senior BizOps at Artie (San Francisco)(ycombinator.com)

   1 points by tang8330 10mo ago
6. 6.▲
   Artie (YC S23) Is Hiring a Senior Product Marketing Manager (SF)(ycombinator.com)

   1 points by tang8330 1y ago
7. 7.▲
   Artie (YC S23) Is Hiring Engineer #3(ycombinator.com)

   1 points by tang8330 1y ago
8. 8.▲
   Artie (YC S23) Is Hiring Engineer #3(ycombinator.com)

   1 points by tang8330 2y ago
9. 9.▲
   Artie (YC S23) Is Hiring a Founding Engineer (Distributed Systems)(ycombinator.com)

   1 points by tang8330 2y ago
10. 10.▲
    by tang8330 2y ago

    Thanks for the flag. We'll get that fixed!
11. 11.▲
    Artie (YC S23) Is Hiring a Founding Designer

    1 points by tang8330 2y ago
12. 12.▲
    Artie (YC S23) Is Hiring(ycombinator.com)

    1 points by tang8330 2y ago
13. 13.▲
    Running Redshift at Scale(blog.artie.so)

    32 points by tang8330 3y ago|7 comments
14. 14.▲
    Preventing replication slot overflow on Postgres DB (AWS RDS)(blog.artie.so)

    4 points by tang8330 3y ago|1 comments
15. 15.▲
    by tang8330 3y ago

    Artie OSS: https://github.com/artie-labs/transfer Artie Cloud: https://www.artie.so/
16. 16.▲
    Artie (YC S23) is hiring a founding engineer(ycombinator.com)

    1 points by tang8330 3y ago
17. 17.▲
    by tang8330 3y ago

    On a single unbounded (CPU + mem) Debezium running on a VM extracting Postgres, I was able to clock in about 7-10m/hr. You could increase the # of tasks, but then it'll hinder your DB perf. Also, this is on your primary DB. We fou
18. 18.▲
    by tang8330 3y ago

    Hi! Yes, our next 2 connectors are going to be S3 and DynamoDB. GitHub Issue: https://github.com/artie-labs/transfer/issues/157 If you'd like to be another design partner for us on this, do reach out. I&
19. 19.▲
    by tang8330 3y ago

    Definitely. For staging temporary tables to merge for our Redshfit destination, we're uploading it to S3. We will be creating a S3 destination with TSV, Avro, Parquet format support verily soon.
20. 20.▲
    by tang8330 3y ago

    For Postgres, we have our own custom snapshotter that is capable of doing parallel snapshots against your read replica and not incur WAL growth. More details here: https://news.ycombinator.com/item?id=36855338 For MySQL and
21. 21.▲
    by tang8330 3y ago

    Thank you so much for the support!
22. 22.▲
    by tang8330 3y ago

    Thanks for the comment! Your comment regarding DDL is interesting. Today, this is what happens: 1/ Column doesn't exist in the destination, we'll create it based on our typing inference from the data type (important: not the
23. 23.▲
    by tang8330 3y ago

    Hm, perhaps I wasn't being clear, apologies for that. What I am proposing above is ways to provide a view to teams that do not want real-time data while keeping your underlying dataset in real-time.
24. 24.▲
    by tang8330 3y ago

    Thanks for your feedback! > Something McKinley doesn't address is that it's quite advantageous if the values in your data warehouse don't change intra-day because this lets business users reach consensus. Whereas if Bob ru
25. 25.▲
    by tang8330 3y ago

    For now, we're super focused on databases as sources. We really want to do this well before we move on to other data sources such as APIs.
26. 26.▲
    by tang8330 3y ago

    Thank you!!
27. 27.▲
    by tang8330 3y ago

    Definitely. Do you expect the resulting data to also be in binary / bytes format in your DWH? I ask because there's a workaround by setting `binary.handling.mode` to a STRING type [1]. Transfer will then automatically pick this up
28. 28.▲
    by tang8330 3y ago

    That's fascinating! Thanks for providing more color, support for geometric shapes coming! https://github.com/artie-labs/transfer/issues/155
29. 29.▲
    by tang8330 3y ago

    Eventual consistency FTW!
30. 30.▲
    by tang8330 3y ago

    Main value props are low latency OLTP data in your DWH and lower DWH ingestion costs.