4 ms·

Seems like a really useful tool. Would your system support Postgres Aurora on AWS as a source database? Or does it require some lower-level access to Postgres s

Seems like a really useful tool. Would your system support Postgres Aurora on AWS as a source database? Or does it require some lower-level access to Postgres server? We are currently using DMS to send data to S3 and from there to Snowflake.

saisrirampur 3y ago
PeerDB should work or Aurora PostgreSQL. It should work for both log based (CDC) and query based replication. Log based because Aurora supports pgoutput plugin. Curious, are you leveraging CDC to move data to S3? or more query (batch) based?
- oskarpearson 3y ago
  We use DMS in continuous replication mode, which appears to use CDC under the hood according to https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.C... In our setup DMS pushes Parquet files on s3. Snowflake then loads data from there. We’ve occasionally had to do a full table sync from scratch, which is painfully slow. We are going to have to do that in the very near future - when we are upgrading from Postgres 11 to Postgres 15. The S3 step also seems unnecessarily complicated, since we have to expire data from the bucket. How does PeerDB handle things like schema changes? Would the change replicate to Snowflake? (I’m sure this is in the docs, but I’m supposed to be on holiday this week ) Thanks for the quick reply.
  - saisrirampur 3y ago
    Gotcha, that really helps. Schema changes feature is coming soon! We are actively working on it. This thread captures our thinking around it - https://news.ycombinator.com/item?id=36895220 https://news.ycombinator.com/item?id=36895220 Also have a good holiday! :)
    - netvarun 3y ago
      I think you were referring to this thread: https://news.ycombinator.com/item?id=36897010 https://news.ycombinator.com/item?id=36897010
      - saisrirampur 3y ago
        Thank you for pointing to the right thread! :)

saisrirampur 3y ago

PeerDB should work or Aurora PostgreSQL. It should work for both log based (CDC) and query based replication. Log based because Aurora supports pgoutput plugin. Curious, are you leveraging CDC to move data to S3? or more query (batch) based?

oskarpearson 3y ago
We use DMS in continuous replication mode, which appears to use CDC under the hood according to https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.C... In our setup DMS pushes Parquet files on s3. Snowflake then loads data from there. We’ve occasionally had to do a full table sync from scratch, which is painfully slow. We are going to have to do that in the very near future - when we are upgrading from Postgres 11 to Postgres 15. The S3 step also seems unnecessarily complicated, since we have to expire data from the bucket. How does PeerDB handle things like schema changes? Would the change replicate to Snowflake? (I’m sure this is in the docs, but I’m supposed to be on holiday this week ) Thanks for the quick reply.
- saisrirampur 3y ago
  Gotcha, that really helps. Schema changes feature is coming soon! We are actively working on it. This thread captures our thinking around it - https://news.ycombinator.com/item?id=36895220 https://news.ycombinator.com/item?id=36895220 Also have a good holiday! :)
  - netvarun 3y ago
    I think you were referring to this thread: https://news.ycombinator.com/item?id=36897010 https://news.ycombinator.com/item?id=36897010
    - saisrirampur 3y ago
      Thank you for pointing to the right thread! :)