6 ms·

Might be worth looking into Airbyte!

• thejosh 5y ago
  Airbyte looks great, and the UI is fantastic. It uses Java for its connectors, and looks great but has issues importing a massive dataset into S3 as there is a chunk limit of 10k, and each chunk size is 5mb :).
  • jgraettinger1 5y ago
    We've used https://github.com/estuary/connectors/pkgs/container/source-s3 https://github.com/estuary/connectors/pkgs/container/source-... to load data sets in the many terabytes. Caveat that, while it's implemented to Airbyte's spec, we've only used it with Flow.
  • jeanlaf 5y ago
    That’s the current focus of the team. Consolidating those connectors :)