7 ms·

Become a better data engineer on a shoestring (free resources)

161 points by aratob 5y ago

blowski 5y ago
Can anyone vouch for the quality on this site? I've been looking for data engineering resources for my team, but don't want to share anything shoddy.
- gigatexal 5y ago
  My title says Senior Data Engineer* and from what I can tell it looks like a collection of interesting things but nothing so revolutionary. You're better off identifying what aspects your team is weak in and seeking out experts or training in that specifically. * titles are kinda shit analogs for skill
- mywittyname 5y ago
  Looking over the resources, it doesn't seem worthwhile. The difficult part is not learning how to code, or work with SQL. The hard part is learning the platform and tooling you need to operate at scale. The ecosystem is full of tools that are great for certain workloads, but terrible for others. Your best bet is to start by getting an overview of the tools available for your team. If you're using AWS, GCP, or Azure, they each have data engineer-oriented certifications. So take a look at what tools those certification courses cover and start there. If you are not in a cloud environment, take a look at Apache Airflow, Beam, Storm, or Hadoop. Most of the tooling provided by the big cloud providers is either a rip off of one these products, or is merely a hosted version (i.e, GCP Cloud Composer is managed Airflow).
  - olsgaarddk 5y ago
    I really got a lot of use out of taking the GCP data engineer course on coursera (the one by google aimed at the cloud cert) and then later taking the actual certification. With that being said it was very focused on BigQuery, and my impression is that it is that all their certs are now basically different variations of a kubernetes certification.
    - mywittyname 5y ago
      I used the Coursera course as well and yea, it's pretty good, so long as you do the labs. To be fair though, BQ is the swiss army knife of GCP data engineering. I was a GCP DE consultant for many years and so much of the pipelines I put together amounted to, shove data into BQ as early as possible, then leverage SQL for transformations. Plus, most Google products have native BQ support (ads, GA, Youtube, etc), which makes it a must-have tool for a lot of companies.
  - monkeybutton 5y ago
    This is a great reply! The greatest difficulty I face is convincing my data engineers to stop reinventing the wheel and leverage existing (and appropriate!) tools. Writing code to directly manage events and batch jobs should be a thing of the past by now. Pick a tool. Configure your jobs, retry policy, etc. and be done with it.
    - soobrosa 5y ago
      If they need convincing they are not engineers and maybe you want to let them go and destroy one of your competitors?
  - stadium 5y ago
    > The hard part is learning the platform and tooling you need to operate at scale Harder still is defining the right problem to solve between new projects and maintaining existing infra. The tech problems is easy. Solving the people and organizational problems with the right tech pays the big bucks in data engineering.
  - soobrosa 5y ago
    We teach the difficult part at the Academy in a cloud agnostic fashion so you don't necessary have to suck up their marketing driven certis. We don't see any new projects starting out with Airflow, Beam, Storm or Hadoop, so that's not a good choice for anyone, only if your plan is to keep horrible legacy stacks alive as a freelancer.
lixtra 5y ago
I‘m not aware of any decent data engineer for whom money is the bottleneck. It’s always time and mental capacity.
- preetamjinka 5y ago
  Wouldn't the target audience for these resources be those who are not already decent data engineers?
- primax 5y ago
  Have you considered that largest part of the target audience for becoming a better data engineer isn't people who are already great at data engineering..?
frankbreetz 5y ago
Does anyone have a list of high quality data engineering, that aren't free? I don't mind paying, but there is so much out there and most of it is not great.
- soobrosa 5y ago
  The original post has some paid resources https://www.dataengineering.academy/pipeline-data-engineering-academy-blog/become-a-data-engineer-on-a-shoestring https://www.dataengineering.academy/pipeline-data-engineerin... or you can also join our full-time/part-time course https://www.dataengineering.academy/curriculum https://www.dataengineering.academy/curriculum
exdsq 5y ago
Any tips for getting started as a data engineer? I actually like cleaning stuff so maybe it's the career for me
- wswope 5y ago
  Are you asking in terms of career or in terms of building up skill?
caffeine 5y ago
Let’s say you are the Lead Data Engineer of a small data-driven company. You need to define a strategy, pick a stack of tools, decide how data is going to be stored and normalised, what the workflow will be from ad-hoc, exploratory studies to productionized inference. Are there any good resources out there that are useful to this person? I am this person right now and I need to find some good guidance.
- cgio 5y ago
  I am that person too, only for a big (non-data) money driven company :-) So many resources that you can easily get lost. Martin Kleppmann's Designing Data Intensive applications can be your starting point. It helps you establish a basic to advanced understanding on quite a few of the concepts that will be coming up and some key principles to drive your strategy from a technical perspective (you'll need a few facets of your strategy for different audiences). Then move to a more corporate focused presentation with Piethein Strengholt's Data Management at Scale (business facing aspects of your strategy, incl. governance forums etc. unless if you are short-term lucky enough to not have them due to size - long term unlucky as you'll have to establish them or drive others to do so). At this point, after a few discussions you should be getting a feeling of what the direction will be in terms of where your data will be stored, how you do data quality, how you process, how you expose, infrastructure etc. Dozens of books on the individual elements of your stack. Try to link them back to Kleppmann or other more specialized but still conceptual books (e.g. if you do streaming you could look into Flow Architectures by Urquhart, Streaming Systems By Akidau et.al. etc.) Then you can move to inference, etc. I am not at that stage yet, so no specific advice. In my case, I see inference etc. as more of something I can address after data are on the platform, but not sure what the state you are facing is. I guess you can start looking into trendy stuff like MLOps etc. Good luck! It's really exciting working on this domain!
  - soobrosa 5y ago
    Kleppmann will not help you in real life situations, it's a textbook that you might read to understand what are the things you don't have to/need to worry about.
- 5y ago
soobrosa 5y ago
Hey, I'm the actual author of the blog post, big thanks to aratob for sharing it and to all for the conversation. A small addition: one of our advisors, Dr. Martin Loetzsch just shared for free the material he's teaching also at Pipeline Data Engineering Academy: https://www.youtube.com/watch?v=8HlNG8bdlM0 https://www.youtube.com/watch?v=8HlNG8bdlM0 https://www.youtube.com/watch?v=24Uvo5vZJWA https://www.youtube.com/watch?v=24Uvo5vZJWA

161 points by aratob 5y ago

Can anyone vouch for the quality on this site? I've been looking for data engineering resources for my team, but don't want to share anything shoddy.

gigatexal 5y ago
My title says Senior Data Engineer* and from what I can tell it looks like a collection of interesting things but nothing so revolutionary. You're better off identifying what aspects your team is weak in and seeking out experts or training in that specifically. * titles are kinda shit analogs for skill
mywittyname 5y ago
Looking over the resources, it doesn't seem worthwhile. The difficult part is not learning how to code, or work with SQL. The hard part is learning the platform and tooling you need to operate at scale. The ecosystem is full of tools that are great for certain workloads, but terrible for others. Your best bet is to start by getting an overview of the tools available for your team. If you're using AWS, GCP, or Azure, they each have data engineer-oriented certifications. So take a look at what tools those certification courses cover and start there. If you are not in a cloud environment, take a look at Apache Airflow, Beam, Storm, or Hadoop. Most of the tooling provided by the big cloud providers is either a rip off of one these products, or is merely a hosted version (i.e, GCP Cloud Composer is managed Airflow).
- olsgaarddk 5y ago
  I really got a lot of use out of taking the GCP data engineer course on coursera (the one by google aimed at the cloud cert) and then later taking the actual certification. With that being said it was very focused on BigQuery, and my impression is that it is that all their certs are now basically different variations of a kubernetes certification.
  - mywittyname 5y ago
    I used the Coursera course as well and yea, it's pretty good, so long as you do the labs. To be fair though, BQ is the swiss army knife of GCP data engineering. I was a GCP DE consultant for many years and so much of the pipelines I put together amounted to, shove data into BQ as early as possible, then leverage SQL for transformations. Plus, most Google products have native BQ support (ads, GA, Youtube, etc), which makes it a must-have tool for a lot of companies.
- monkeybutton 5y ago
  This is a great reply! The greatest difficulty I face is convincing my data engineers to stop reinventing the wheel and leverage existing (and appropriate!) tools. Writing code to directly manage events and batch jobs should be a thing of the past by now. Pick a tool. Configure your jobs, retry policy, etc. and be done with it.
  - soobrosa 5y ago
    If they need convincing they are not engineers and maybe you want to let them go and destroy one of your competitors?
- stadium 5y ago
  > The hard part is learning the platform and tooling you need to operate at scale Harder still is defining the right problem to solve between new projects and maintaining existing infra. The tech problems is easy. Solving the people and organizational problems with the right tech pays the big bucks in data engineering.
- soobrosa 5y ago
  We teach the difficult part at the Academy in a cloud agnostic fashion so you don't necessary have to suck up their marketing driven certis. We don't see any new projects starting out with Airflow, Beam, Storm or Hadoop, so that's not a good choice for anyone, only if your plan is to keep horrible legacy stacks alive as a freelancer.

lixtra 5y ago

I‘m not aware of any decent data engineer for whom money is the bottleneck. It’s always time and mental capacity.

preetamjinka 5y ago
Wouldn't the target audience for these resources be those who are not already decent data engineers?
primax 5y ago
Have you considered that largest part of the target audience for becoming a better data engineer isn't people who are already great at data engineering..?

frankbreetz 5y ago

Does anyone have a list of high quality data engineering, that aren't free? I don't mind paying, but there is so much out there and most of it is not great.

soobrosa 5y ago
The original post has some paid resources https://www.dataengineering.academy/pipeline-data-engineering-academy-blog/become-a-data-engineer-on-a-shoestring https://www.dataengineering.academy/pipeline-data-engineerin... or you can also join our full-time/part-time course https://www.dataengineering.academy/curriculum https://www.dataengineering.academy/curriculum

exdsq 5y ago

Any tips for getting started as a data engineer? I actually like cleaning stuff so maybe it's the career for me

wswope 5y ago
Are you asking in terms of career or in terms of building up skill?

caffeine 5y ago

Let’s say you are the Lead Data Engineer of a small data-driven company. You need to define a strategy, pick a stack of tools, decide how data is going to be stored and normalised, what the workflow will be from ad-hoc, exploratory studies to productionized inference. Are there any good resources out there that are useful to this person? I am this person right now and I need to find some good guidance.

cgio 5y ago
I am that person too, only for a big (non-data) money driven company :-) So many resources that you can easily get lost. Martin Kleppmann's Designing Data Intensive applications can be your starting point. It helps you establish a basic to advanced understanding on quite a few of the concepts that will be coming up and some key principles to drive your strategy from a technical perspective (you'll need a few facets of your strategy for different audiences). Then move to a more corporate focused presentation with Piethein Strengholt's Data Management at Scale (business facing aspects of your strategy, incl. governance forums etc. unless if you are short-term lucky enough to not have them due to size - long term unlucky as you'll have to establish them or drive others to do so). At this point, after a few discussions you should be getting a feeling of what the direction will be in terms of where your data will be stored, how you do data quality, how you process, how you expose, infrastructure etc. Dozens of books on the individual elements of your stack. Try to link them back to Kleppmann or other more specialized but still conceptual books (e.g. if you do streaming you could look into Flow Architectures by Urquhart, Streaming Systems By Akidau et.al. etc.) Then you can move to inference, etc. I am not at that stage yet, so no specific advice. In my case, I see inference etc. as more of something I can address after data are on the platform, but not sure what the state you are facing is. I guess you can start looking into trendy stuff like MLOps etc. Good luck! It's really exciting working on this domain!
- soobrosa 5y ago
  Kleppmann will not help you in real life situations, it's a textbook that you might read to understand what are the things you don't have to/need to worry about.
5y ago

soobrosa 5y ago

Hey, I'm the actual author of the blog post, big thanks to aratob for sharing it and to all for the conversation. A small addition: one of our advisors, Dr. Martin Loetzsch just shared for free the material he's teaching also at Pipeline Data Engineering Academy: https://www.youtube.com/watch?v=8HlNG8bdlM0 https://www.youtube.com/watch?v=8HlNG8bdlM0 https://www.youtube.com/watch?v=24Uvo5vZJWA https://www.youtube.com/watch?v=24Uvo5vZJWA