15 ms·

I feel the term "Data Engineer" gets used for a lot of catch-all "we have problems that need an owner" situations. There's not much consistency across job post

by FZ1 7y ago

I feel the term "Data Engineer" gets used for a lot of catch-all "we have problems that need an owner" situations. There's not much consistency across job postings and interviews for this kind of thing. I just interviewed for one "Data Engineer" position which consisted of nearly 100% stored procedures. No one knew what else to call it, and they didn't want to advertise for a DBA, because there were no real DBA responsibilities. So "Data Engineer" was chosen. Another "Data Engineer" position was almost entirely Spark. There was no SQL involved - they expected all applicants to be Spark experts, with a deep knowledge of Scala. It's hard to know what to expect out of "Data Engineer" positions until you walk into a place and start asking questions in the interview.

luckydata 7y ago
I feel the name "data engineer" is perfectly fine for both positions. It's the job of the job post to clarify the requirements. I don't see an issue with looking for a "data engineer with deep experience building and maintaining data pipelines based on spark". What would you call it? "Spark operator"?
- FZ1 7y ago
  > It's the job of the job post to clarify the requirements. I couldn't agree more. That's the exact thing I'm saying is a problem here. When stuff is wrong, and we don't have enough people, and don't have a well-defined job, but need someone to just handle it .. "Data Engineer". You get there, and everybody has a different idea as to what you should be doing, and what skills you should have.
  - wlesieutre 7y ago
    Is this any different from “software engineer”?
    - FZ1 7y ago
      SE usually at least knows what language they're expecting you to mainly work in. Not always the case for DE.
      - wlesieutre 7y ago
        Ah gotcha, the problem isn't that it's a vague job title, the problem is that the job description for a particular position may not set expectations clearly enough
        FZ1 7y ago
        No - I'm trying to say that people often don't even know what the DE really needs to do. They know they have back-end problems, and need someone to clean them up. But often the hiring manager believes it's probably mostly SQL or mostly python, and advertises that. They really don't know - because someone left, and they are trying to replace them. So you get there, and talk to the hiring manager, and they're like "are you good in SQL?". Then you talk to someone closer to the problem, and they're like "most of it's in Java - how's your Java?". Then someone even closer to the position says, "well, most of the work we need done is really in Scala". It's your job to just take it over, and make it happen. "Fix the data" - using our existing patchwork of tools ... This isn't always the case - but given the nature of the problem, this happens a hell of a lot more often than with Soft Eng's.
    - bfung 7y ago
      Nope. Just someone that is willing to use tools in addition to build tools to deal with data. Most who consider themselves software engineers only want to code and build new, not use existing. Mostly a mindset thing.
opportune 7y ago
I had to stop calling one of my previous roles “data engineering” because people thought I was just doing SQL script ETL... which really was maybe 30% of the job, but apparently that’s too menial to give you programmer cred.
- FZ1 7y ago
  hmm ... I didn't think about that. I wonder if the term is building a negative connotation?
closeparen 7y ago
For us it's simply the runtime you target. A frontend engineer writes code for the browser. A mobile engineer writes code for iOS or Android. A backend engineer writes code for servers. A data engineer writes code for the analytical/warehouse environment. It was typically backend engineers doing the data pipelines to expose records from their own services, until the company got bigger and the analysts got more demanding, and that became a dedicated role.
- moandcompany 7y ago
  Engineers are professionals in the business of solving problems :)
  - walshemj 7y ago
    Scientists with thumbs was one definition I heard
darkerside 7y ago
Doesn't the term software engineer side from the same ambiguity?
dntbnmpls 7y ago
> I just interviewed for one "Data Engineer" position which consisted of nearly 100% stored procedures. How about data or database programmer? I've also heard "Data Analyst", "SQL Developer", etc. But I agree that "data engineer", "software engineer", "network engineer", etc are too general to really be meaningful.
moandcompany 7y ago
You're seeing the same thing happen with the "Data Engineer" title as what has happened with "Data Scientist." In the hiring market, many positions that were previously titled "Data Analyst"/etc were renamed "Data Scientist." Similarly "Database Engineer"/"Data Warehouse Engineer"/"DBA"/etc were renamed "Data Engineer." Why? Various reasons: One might be the need to give people a sexier title because they don't want to be "just a data analyst." Another case might be, hiring managers and recruiting teams changing their title to capture more views for people searching using these terms. And why are people searching for jobs using these terms? Articles like the ones in the original post, and "Data Scientist: The sexiest job of the 21st century" ;)
amelius 7y ago
Most programming is just plumbing with data, so I'd say the term data engineer would fit most programmers, most of the time.

by FZ1 7y ago

luckydata 7y ago

I feel the name "data engineer" is perfectly fine for both positions. It's the job of the job post to clarify the requirements. I don't see an issue with looking for a "data engineer with deep experience building and maintaining data pipelines based on spark". What would you call it? "Spark operator"?

FZ1 7y ago
> It's the job of the job post to clarify the requirements. I couldn't agree more. That's the exact thing I'm saying is a problem here. When stuff is wrong, and we don't have enough people, and don't have a well-defined job, but need someone to just handle it .. "Data Engineer". You get there, and everybody has a different idea as to what you should be doing, and what skills you should have.
- wlesieutre 7y ago
  Is this any different from “software engineer”?
  - FZ1 7y ago
    SE usually at least knows what language they're expecting you to mainly work in. Not always the case for DE.
    - wlesieutre 7y ago
      Ah gotcha, the problem isn't that it's a vague job title, the problem is that the job description for a particular position may not set expectations clearly enough
      - FZ1 7y ago
        No - I'm trying to say that people often don't even know what the DE really needs to do. They know they have back-end problems, and need someone to clean them up. But often the hiring manager believes it's probably mostly SQL or mostly python, and advertises that. They really don't know - because someone left, and they are trying to replace them. So you get there, and talk to the hiring manager, and they're like "are you good in SQL?". Then you talk to someone closer to the problem, and they're like "most of it's in Java - how's your Java?". Then someone even closer to the position says, "well, most of the work we need done is really in Scala". It's your job to just take it over, and make it happen. "Fix the data" - using our existing patchwork of tools ... This isn't always the case - but given the nature of the problem, this happens a hell of a lot more often than with Soft Eng's.
  - bfung 7y ago
    Nope. Just someone that is willing to use tools in addition to build tools to deal with data. Most who consider themselves software engineers only want to code and build new, not use existing. Mostly a mindset thing.

opportune 7y ago

I had to stop calling one of my previous roles “data engineering” because people thought I was just doing SQL script ETL... which really was maybe 30% of the job, but apparently that’s too menial to give you programmer cred.

FZ1 7y ago
hmm ... I didn't think about that. I wonder if the term is building a negative connotation?

closeparen 7y ago

For us it's simply the runtime you target. A frontend engineer writes code for the browser. A mobile engineer writes code for iOS or Android. A backend engineer writes code for servers. A data engineer writes code for the analytical/warehouse environment. It was typically backend engineers doing the data pipelines to expose records from their own services, until the company got bigger and the analysts got more demanding, and that became a dedicated role.

moandcompany 7y ago
Engineers are professionals in the business of solving problems :)
- walshemj 7y ago
  Scientists with thumbs was one definition I heard

darkerside 7y ago

Doesn't the term software engineer side from the same ambiguity?

dntbnmpls 7y ago

> I just interviewed for one "Data Engineer" position which consisted of nearly 100% stored procedures. How about data or database programmer? I've also heard "Data Analyst", "SQL Developer", etc. But I agree that "data engineer", "software engineer", "network engineer", etc are too general to really be meaningful.

moandcompany 7y ago

You're seeing the same thing happen with the "Data Engineer" title as what has happened with "Data Scientist." In the hiring market, many positions that were previously titled "Data Analyst"/etc were renamed "Data Scientist." Similarly "Database Engineer"/"Data Warehouse Engineer"/"DBA"/etc were renamed "Data Engineer." Why? Various reasons: One might be the need to give people a sexier title because they don't want to be "just a data analyst." Another case might be, hiring managers and recruiting teams changing their title to capture more views for people searching using these terms. And why are people searching for jobs using these terms? Articles like the ones in the original post, and "Data Scientist: The sexiest job of the 21st century" ;)

amelius 7y ago

Most programming is just plumbing with data, so I'd say the term data engineer would fit most programmers, most of the time.