7 ms·
The Data Engineering track seems to attract data analysts and non-engineers. I’m one of them, have worked with many others in my same bucket. What makes Data En
by iblaine 3y ago
The Data Engineering track seems to attract data analysts and non-engineers. I’m one of them, have worked with many others in my same bucket. What makes Data Engineers excel, I think, is to have all the skills of a software engineer + the ability to think like a data analyst.
SQL is definitely part of that journey, and more skills should be mastered if you’re going to do things like use Kafka, airflow or spark.
- woleium 3y agofor a moment there I thought you said "what makes data engineers is excel"
- jeremyjh 3y agodo not speaks its name
- sanderjd 3y agoThe problem is that if you have all the skills of a software engineer, lots of people would pay you more to do different things than they will to do "data engineering", which is often a lot about cobbling different tools together and doing a bunch of custom transformations and validations that are more tedious than creative.
- iblaine 3y agoAgreed. The job title of Data Engineer gets abused. Some DE's are Analysts, others are Backend SWEs. Their goals are aligned, to create value out of data, and their methods may differ greatly.
- _bk90 3y agoA data engineer is as much a software engineer as a back-end engineer or front-end engineer and paid equivalently.
- sanderjd 3y agoI would certainly prefer it if that were the case, but it largely is not, in my experience.
- _bk90 3y agoIt seems like your experience doesn't overlap with data engineering, in that case.
- sanderjd 3y agoOk. I think this is just going to end up being "no true scotsman". I'll say "in many companies, data engineers are more doing BI data preparation and presentation than building distributed processing infrastructure or whatever and they are compensated more like analysts than general software engineers" and you'll say "well that's not data engineering". My argument is not an "ought" argument, it's an "is". I agree with you that data engineering "ought" to be, if anything, a more difficult specialized practice of software engineering, because you need all the software engineering skills and then also specialized data skills. But I'm saying that what it often "is", is more like a specialization of BI and analyst roles. I think the OP is a pretty good demonstration of this. If "all you need is SQL", does that sound more like a specialization of a software engineer skillset or an analyst skillset? I think the latter... And sure you can say, "well the article is wrong, if all you need is SQL, that's not data engineering", but we're just back to "no true scotsman"; I believe it is common to see the role this way.
- tm-guimaraes 3y agoSame as “game developer” But that’s not how markets work. Also a job title doesn’t really say much. On all of these 4 roles there’s people doing trivial work, as there’s people doing very deep technical work. Same with pushing the envelope on tech.