9 ms·

Building a High Performance Data Integration Framework in Go

123 points by sghosh2 4y ago

yevpats 4y ago
Awesome to see this on HN! Founder/Author here. I'll be happy to answer questions.
- sghosh2 4y ago
  Great post! I was especially impressed to hear about the 80% of plug-in code being auto-generated through the SDK.
- StefanWestfal 4y ago
  Thank you for putting your time into the project! Besides this project I like sqlc a lot but in both scenarios I was wondering how one gets started in code generation tooling? So that I could contribute, extend or even create my own (I.e. I would like to extend sqlc for better integration with FastAPI).
  - yevpats 4y ago
    Thanks! I think the best place for code generation in Go would be: 1) Go templating: https://pkg.go.dev/text/template https://pkg.go.dev/text/template 2) Go reflection: https://go.dev/blog/laws-of-reflection https://go.dev/blog/laws-of-reflection (there are many articles and documentation on this). Then I would try to do something small to experience both libraries hands-on as otherwise it's hard to get the hang of it. Once you did a small project I would go and try to contribute a small fix/feat to sqlc to understand sqlc code structure and from there potentially a bigger feature/extension for sqlc.
Xeoncross 4y ago
Code generation is becoming really important in Go. Why even use an ORM when https://sqlc.dev/ https://sqlc.dev/ will generate everything from vanilla SQL? Why make the frontend team write a Typescript client when https://goa.design https://goa.design on the backend will produce an OpenAPI schema they can just point a https://openapi-generator.tech https://openapi-generator.tech at? Why write out GraphQL boilerplate when https://github.com/99designs/gqlgen https://github.com/99designs/gqlgen will take your GQL typedef and generate it all for you based on how you want it to look. Why write validation rules when you can just define your input struct and let https://github.com/mustafaakin/gongular https://github.com/mustafaakin/gongular generate the rest for you? Honestly, I'm loving this. I want to focus on the entities and business logic - not writing yet another handler/resolver for basic auth + CRUD work.
- yevpats 4y ago
  100% ORMs becoming less and less relevant especially in Go with code generation and a bit of copilot help :). Also, Protobuf is another (not new) but good example of client and server side code generation which passed the time test.
- zinclozenge 4y ago
  Gongular looks pretty cool, but it seems like development stopped. Is there any other library that does similar things? I love learning about new frameworks/libraries in golang for exactly that reason of reducing boilerplate and spending more time on logic.
  - stuzenz 4y ago
    Interesting response here in the issues. The project is still active. It seems that feature requests and issues are not being asked of the project. https://github.com/mustafaakin/gongular/issues/21 https://github.com/mustafaakin/gongular/issues/21
- jerf 4y ago
  It would be interesting for the annual Go survey to include a question about how much code generation is being used in the wild. My personal impression is that it is not widespread at all, but I phrase it that way on purpose.
  - halpmeh 4y ago
    We've been using it for mocks and protocol buffers for many years. I'm sure we're not alone.
- lairv 4y ago
  sqlc is great but what I dislike about it is that it forces you into using only the supported underlying drivers, for example with postgresql the pgx v5 driver has been out for some times now but sqlc only support v4 (this might have changed recently I haven't checked). Overall there are many reasons why I would like to generate code slightly differently from what is done in the lib, I wish there was a way to have a highly customizable code generation experience
elchief 4y ago
Seems like strange choices for "CloudQuery vs Others". Why not compare against FiveTran, Airbyte, Meltano or other EL tools? Also, It'd be nice to know what the transfer protocol is like. What format is used to transfer between a Source and Destination?
- yevpats 4y ago
  Great question! We specifically started with connectors/plugins to cloud infrastructure providers and other infrastructure vendors - This is why most of our users migrated and/or compared us to the tools specified here (https://www.cloudquery.io/docs/cq-vs-others/overview https://www.cloudquery.io/docs/cq-vs-others/overview). There are no connectors for AWS, GCP, Azure in FiveTran, Airbyte, Meltano but as we extend the number of our source plugins I believe this question will come up more and we will add those products as well to our comparison list. The protocol is just standard/plain gRPC (https://www.cloudquery.io/docs/developers/architecture https://www.cloudquery.io/docs/developers/architecture). Protobuf is defined here - https://github.com/cloudquery/plugin-sdk/tree/main/internal/pb https://github.com/cloudquery/plugin-sdk/tree/main/internal/... Basically just the normalized, verified data + metadata, types
  - elchief 4y ago
    understood :) thanks for the reply
  - tomrod 4y ago
    Will you adopt the singer interfaces? That's what Meltano does, and it's chefs kiss.
    - yevpats 4y ago
      Interesting. We will take a look at this even though we usually build our own connectors due the huge concurrency/speed advantages when using CloudQuery SDK.
- sghosh2 4y ago
  Much more of a focus on security and infrastructure than other EL tools. Like these security policies for example - https://www.cloudquery.io/docs/core-concepts/policies https://www.cloudquery.io/docs/core-concepts/policies
laerus 4y ago
So golang macros when? Honestly the language is the new JS and just another one to remind us how miserable software can be with half baked enterprizy solutions to just get the job done faster.
- philosopher1234 4y ago
  This is a lot of confidence for an opinion that doesn't seem very thought through. There are a lot of downsides to macros, and a lot of (apparently under-appreciated) upsides to code generation.

123 points by sghosh2 4y ago

yevpats 4y ago

Awesome to see this on HN! Founder/Author here. I'll be happy to answer questions.

sghosh2 4y ago
Great post! I was especially impressed to hear about the 80% of plug-in code being auto-generated through the SDK.
StefanWestfal 4y ago
Thank you for putting your time into the project! Besides this project I like sqlc a lot but in both scenarios I was wondering how one gets started in code generation tooling? So that I could contribute, extend or even create my own (I.e. I would like to extend sqlc for better integration with FastAPI).
- yevpats 4y ago
  Thanks! I think the best place for code generation in Go would be: 1) Go templating: https://pkg.go.dev/text/template https://pkg.go.dev/text/template 2) Go reflection: https://go.dev/blog/laws-of-reflection https://go.dev/blog/laws-of-reflection (there are many articles and documentation on this). Then I would try to do something small to experience both libraries hands-on as otherwise it's hard to get the hang of it. Once you did a small project I would go and try to contribute a small fix/feat to sqlc to understand sqlc code structure and from there potentially a bigger feature/extension for sqlc.

Xeoncross 4y ago

Code generation is becoming really important in Go. Why even use an ORM when https://sqlc.dev/ https://sqlc.dev/ will generate everything from vanilla SQL? Why make the frontend team write a Typescript client when https://goa.design https://goa.design on the backend will produce an OpenAPI schema they can just point a https://openapi-generator.tech https://openapi-generator.tech at? Why write out GraphQL boilerplate when https://github.com/99designs/gqlgen https://github.com/99designs/gqlgen will take your GQL typedef and generate it all for you based on how you want it to look. Why write validation rules when you can just define your input struct and let https://github.com/mustafaakin/gongular https://github.com/mustafaakin/gongular generate the rest for you? Honestly, I'm loving this. I want to focus on the entities and business logic - not writing yet another handler/resolver for basic auth + CRUD work.

yevpats 4y ago
100% ORMs becoming less and less relevant especially in Go with code generation and a bit of copilot help :). Also, Protobuf is another (not new) but good example of client and server side code generation which passed the time test.
zinclozenge 4y ago
Gongular looks pretty cool, but it seems like development stopped. Is there any other library that does similar things? I love learning about new frameworks/libraries in golang for exactly that reason of reducing boilerplate and spending more time on logic.
- stuzenz 4y ago
  Interesting response here in the issues. The project is still active. It seems that feature requests and issues are not being asked of the project. https://github.com/mustafaakin/gongular/issues/21 https://github.com/mustafaakin/gongular/issues/21
jerf 4y ago
It would be interesting for the annual Go survey to include a question about how much code generation is being used in the wild. My personal impression is that it is not widespread at all, but I phrase it that way on purpose.
- halpmeh 4y ago
  We've been using it for mocks and protocol buffers for many years. I'm sure we're not alone.
lairv 4y ago
sqlc is great but what I dislike about it is that it forces you into using only the supported underlying drivers, for example with postgresql the pgx v5 driver has been out for some times now but sqlc only support v4 (this might have changed recently I haven't checked). Overall there are many reasons why I would like to generate code slightly differently from what is done in the lib, I wish there was a way to have a highly customizable code generation experience

elchief 4y ago

Seems like strange choices for "CloudQuery vs Others". Why not compare against FiveTran, Airbyte, Meltano or other EL tools? Also, It'd be nice to know what the transfer protocol is like. What format is used to transfer between a Source and Destination?

yevpats 4y ago
Great question! We specifically started with connectors/plugins to cloud infrastructure providers and other infrastructure vendors - This is why most of our users migrated and/or compared us to the tools specified here (https://www.cloudquery.io/docs/cq-vs-others/overview https://www.cloudquery.io/docs/cq-vs-others/overview). There are no connectors for AWS, GCP, Azure in FiveTran, Airbyte, Meltano but as we extend the number of our source plugins I believe this question will come up more and we will add those products as well to our comparison list. The protocol is just standard/plain gRPC (https://www.cloudquery.io/docs/developers/architecture https://www.cloudquery.io/docs/developers/architecture). Protobuf is defined here - https://github.com/cloudquery/plugin-sdk/tree/main/internal/pb https://github.com/cloudquery/plugin-sdk/tree/main/internal/... Basically just the normalized, verified data + metadata, types
- elchief 4y ago
  understood :) thanks for the reply
- tomrod 4y ago
  Will you adopt the singer interfaces? That's what Meltano does, and it's chefs kiss.
  - yevpats 4y ago
    Interesting. We will take a look at this even though we usually build our own connectors due the huge concurrency/speed advantages when using CloudQuery SDK.
sghosh2 4y ago
Much more of a focus on security and infrastructure than other EL tools. Like these security policies for example - https://www.cloudquery.io/docs/core-concepts/policies https://www.cloudquery.io/docs/core-concepts/policies

laerus 4y ago

So golang macros when? Honestly the language is the new JS and just another one to remind us how miserable software can be with half baked enterprizy solutions to just get the job done faster.

philosopher1234 4y ago
This is a lot of confidence for an opinion that doesn't seem very thought through. There are a lot of downsides to macros, and a lot of (apparently under-appreciated) upsides to code generation.