6 ms·

Is there an easy way to have ClickHouse Cloud ingest data from MySQL hosted in Amazon RDS?

by base 4y ago

thomoco 4y ago
There are a few options for migrating or synchronizing data from MySQL - I'd recommend starting with this page in the ClickHouse Docs - there is a nice video there that explains some of those options: https://clickhouse.com/docs/en/integrations/migration/ https://clickhouse.com/docs/en/integrations/migration/ Depending on what you are trying to achieve, you could use clickhouse-local with the MySQL engine to move data, or could use an ETL/ETL tool to migrate/sync
- hodgesrm 4y ago
  Good point. If you just want to copy data far and away the easiest way to transfer data is using MySQLDatabaseEngine. You can even copy table definitions. Watch for issues with Decimal datatypes if you do this.
  - morelisp 4y ago
    Be careful with this engine, it's easy to accidentally expose the password as you only need table read permissions if it wasn't set up using an external credential file. https://github.com/ClickHouse/ClickHouse/issues/3311 https://github.com/ClickHouse/ClickHouse/issues/3311 I also had some pretty bad join performance (CH table joined to MySQL table), the quick solution to both of these is that we instead use the table function (https://clickhouse.com/docs/en/sql-reference/table-functions/mysql/ https://clickhouse.com/docs/en/sql-reference/table-functions...) to copy the data periodically.
    - hodgesrm 4y ago
      Use Named Collections to protect credentials. They are very handy. Here's an article that discusses use in the JDBC Bridge. https://altinity.com/blog/connecting-clickhouse-to-external-data-sources-using-the-jdbc-bridge https://altinity.com/blog/connecting-clickhouse-to-external-... Disclaime: I work for Altinity.
hodgesrm 4y ago
Check out the Altinity Sink Connector for ClickHouse [0]. This is advancing quite quickly and already has prod deployments. Please feel free to try it out. [0] https://github.com/Altinity/clickhouse-sink-connector https://github.com/Altinity/clickhouse-sink-connector

Is there an easy way to have ClickHouse Cloud ingest data from MySQL hosted in Amazon RDS?

by base 4y ago

Is there an easy way to have ClickHouse Cloud ingest data from MySQL hosted in Amazon RDS?

There are a few options for migrating or synchronizing data from MySQL - I'd recommend starting with this page in the ClickHouse Docs - there is a nice video there that explains some of those options: https://clickhouse.com/docs/en/integrations/migration/ https://clickhouse.com/docs/en/integrations/migration/ Depending on what you are trying to achieve, you could use clickhouse-local with the MySQL engine to move data, or could use an ETL/ETL tool to migrate/sync

hodgesrm 4y ago
Good point. If you just want to copy data far and away the easiest way to transfer data is using MySQLDatabaseEngine. You can even copy table definitions. Watch for issues with Decimal datatypes if you do this.
- morelisp 4y ago
  Be careful with this engine, it's easy to accidentally expose the password as you only need table read permissions if it wasn't set up using an external credential file. https://github.com/ClickHouse/ClickHouse/issues/3311 https://github.com/ClickHouse/ClickHouse/issues/3311 I also had some pretty bad join performance (CH table joined to MySQL table), the quick solution to both of these is that we instead use the table function (https://clickhouse.com/docs/en/sql-reference/table-functions/mysql/ https://clickhouse.com/docs/en/sql-reference/table-functions...) to copy the data periodically.
  - hodgesrm 4y ago
    Use Named Collections to protect credentials. They are very handy. Here's an article that discusses use in the JDBC Bridge. https://altinity.com/blog/connecting-clickhouse-to-external-data-sources-using-the-jdbc-bridge https://altinity.com/blog/connecting-clickhouse-to-external-... Disclaime: I work for Altinity.

hodgesrm 4y ago

Check out the Altinity Sink Connector for ClickHouse [0]. This is advancing quite quickly and already has prod deployments. Please feel free to try it out. [0] https://github.com/Altinity/clickhouse-sink-connector https://github.com/Altinity/clickhouse-sink-connector