6 ms·

The Software Scientist

89 points by carljoseph 12y ago

alexandros 12y ago
A fantastic article, and I hope we see more relevant research gathered under this banner. Not having a name for something can be a big problem in even knowing what's out there. As a side-point, I am often shocked that we don't see more research into how to make software teams more productive coming out of places like Google/Facebook/Microsoft etc, given that these companies (claim to) live and die by the effectiveness of their engineering. If a large amount of their expenditure is in software, to the point they'll buy entire companies just for the engineers, why aren't they financing the large-scale experiments that will settle certain questions for good?
- collyw 12y ago
  Is there any evidence that Facebook/Google/Microsoft are productive? They have a lot of money and a lot of engineers to throw at a problem. When I worked at an investment bank, the code quality was crap, and the staff turnover was high. Didn't matter, because they literally had enough money to throw at the problem and keep hiring more people.
gphil 12y ago
I don't understand how this role doesn't fall under the scenario of: "The engineer becomes a manager" In order to do a superlative job as a manager of a software engineering team, one has to master these same skills. Perhaps the term "Software Scientist" will help distinguish good software engineering management practices from bad ones, but I fail to see how "The Software Scientist" is not a management role.
- cafebeen 12y ago
  Also, most academic and research scientists manage a team of students and post-docs who do the actual lab work/analysis/writing. The article's description of a "software scientist" might be more accurately named "software research assistant".
- jbarmash 12y ago
  Agreed. The only way that the post makes sense to me is that if the organization is large enough, have a role that supports CTO / VPE's efforts and tried to do it in a more data-driven way. So this way CTO / VPE still have those responsibilities, but have a person who is fully focused on measuring the impact of changes and getting data. I think these would be very special roles and very few organizations, so not enough to really talk about it as an alternative to the two traditional tracks. That said, there is certainly there is value in this role.
gtirloni 12y ago
The first paragraph seems to imply that, to understand recursive algorithms, you need to be young. And rightly so, the software industry is not for old people. Did I get the argument right?
- crimsonalucard 12y ago
  I didn't get that, what does a software engineer with a basic grasp of recursion have to do with "software science"?
- blackbagboys 12y ago
  No, he is saying that if you understand recursive algorithms, you should be able to apply that reasoning to reason yourself out of a career on software, since it is a given that most developers will be forced out of engineering by a certain age
cafebeen 12y ago
To summarize, the article suggests making the "software science" job title to: * "reduce career path anxiety" * "enhance engineering organizations' self-understanding" * "advance the general understanding of the right way to develop software" * "raise awareness about the sponsoring engineering organization" * "bring out some of those 'big picture' thinkers from the woodwork" The third point is the only one close to the traditional role of a scientist, though "general understanding" takes more than data and statistics from a single organization. Otherwise, this seems to be a pretty blatant misappropriation of the name "scientist"!
jeffdavis 12y ago
Would such a person practice real science, or just apply the first test that cones to mind, and ignore all of the confounding factors?
- hnnewguy 12y ago
  I agree. This sounds ominous: >Statistical education. This is currently optional The data world is going to be an interesting place in the near future, filled with people running statistical tests they don't truly understand, and drawing spurious conclusions. But, I guess it already works that way in some fields.
greenyoda 12y ago
"By and large, programming is a young man’s game" As a middle-aged programmer who is a much better programmer now than I was when I was young, I'd argue with the article's initial premise.
- AndrewDucker 12y ago
  Agreed. It assumes that years of experience mean you shouldn't be programming. Which seems...odd to me.
  - jvehent 12y ago
    Maybe the author understands programming as "writing lines of code", and not as "architecting software". Architecture requires experience. Writing the fastest quick sort in a given language is an exercise for fresh young brains.
    - AndrewDucker 12y ago
      "Writing the fastest quick sort in a given language is an exercise for fresh young brains." I can't see the reasoning here. Surely it's a job for someone who has written previous sorting algorithms, understands the trade-offs between speed and memory, and understand the pitfalls that can be fallen into when sorting different kinds of data?
      - collyw 12y ago
        An experienced brain would look up the pre-written libraries, and use the best of those rather than writing his own buggy implementation.
- deleted 12y ago
  [deleted]
- frozenport 12y ago
  >>As a middle-aged programmer What do your younger colleagues think?
gaius 12y ago
Actual LOL. So we have programmers, developers, software engineers, coders and now... Software scientists?? Oh well it will be a good filter on CVs.
- Chronic30 12y ago
  I'm a software physicist myself.
  - ScottBurson 12y ago
    I think I'll go with "Software Alchemist" :-)
    - gaius 12y ago
      I forgot software architect too
  - waterlesscloud 12y ago
    I keep reading this as software psychic. Which I guess is pretty much what most debugging turns out to be.
- eng_monkey 12y ago
  Do not forget about any one able to write R scripts is a Data Scientist now!
Mz 12y ago
The military is a "young man's game." Most new recruits are 18. But the people in charge, who do the strategizing and so on, are not. It sounds to me like the article is suggesting another way for experienced programmers to be held in high esteem, not suggesting that old programmers are no good. So I honestly do not understand most of the criticisms of this article being posted here on HN. Maybe there is no real difference between a "software scientist" and a manager. Maybe there is. But it doesn't sound to me like this piece in any way insults older programmers.
- seanmcdirmid 12y ago
  You have older NCOs in the military as well as senior officers, but for the most part rank and seniority is a pyramid in the millitary; just like most hyper competitive dev shops...there are older people at the top, just much fewer of them than younger people...and where did the other people go who fell off the pyramid? As a PhD holding computer scientist and practicing researcher, I cringe at the title "software scientist"....
  - Animats 12y ago
    You have older NCOs in the military as well as senior officers, but for the most part rank and seniority is a pyramid in the military. More so than in civilian life. The US military has a formal "up or out" policy and a schedule: https://en.wikipedia.org/wiki/High_Year_Tenure https://en.wikipedia.org/wiki/High_Year_Tenure . That's for enlisted men. For officers, it's tougher, although more complicated; if you're passed over for promotion twice, you're usually out.
  - Mz 12y ago
    It's fine if you cringe. :-) I am just saying that I simply do not see the slam on older programmers that others seem to be taking away from this article. It seems to me there are plenty of older programmers on HN and some of most esteemed folks here are not 22-25 years old and are, instead, a good bit older (in some cases, old enough to have kids that age, in theory if not in reality). (In fact, I suspect none of them are that young -- but I don't know all the ages of, say, everyone on the leaderboard, so I can't assert that as fact. Edit: Also, not everyone that age here can rightly be called a programmer.)
perlgeek 12y ago
If I ever reach the level of an IBM Fellow or similar as a software engineer, I won't care about whether you'll call me a "software scientist" or "man with long beard" or "that guy". I promise. > The engineer becomes a kind of nomadic hermit (“Principal Engineer”) "hermit" implies some level of isolation. Is that really the case of distinguished/principal/fellow engineers? Is there any evidence or data available?
- michaelochurch 12y ago
  Principal and Distinguished are, usually, titles added to convince HR to pay market salaries. Mgr: "We'll need $140k to get him." HR: "The band for Senior Engineer is $100k to 125k." Mgr: "So you can't do $140?" HR: "Nope." Mgr: "What if I hire him for a *Principal* Engineer position?" HR: "I need to talk to the CFO about whether we need Principal Engineers."
michaelochurch 12y ago
This is just another title upgrade (first "software engineer" in lieu of lowly "programmer", then "software scientist") that represents a half-baked understanding of the problem, which is that typical business-driven engineering doesn't work well, isn't very challenging, and is enough of an intellectual wasteland that, after 7-10 years, you'll have learned everything worth learning unless you become an elite engineer (e.g. Principal "hermit", thus named because top-1% engineers don't have much company except at a few dozen companies worldwide) or a manager. That all is true. The ghetto of the average line-of-business software engineer is not a place where one can be over 35 and unembarrassed about remaining there. This said, I think that the "data science" title is silly. It used to mean "what we call machine learning to make it sound more practical and less AI-ish", then it meant "watered down job that occasionally involves running an off-the-shelf regression or clustering algorithm", and then it meant "any job that involves data". I don't hold out higher hopes for "software scientist". We seem, in this industry, to have given up on restoring integrity to software engineering itself, so that everyone over 30 is strategizing to become some kind of XWP ("X Who Programs") rather than a JAP "Just A Programmer"), as I've discussed at length here: http://michaelochurch.wordpress.com/2012/08/26/xwp-vs-jap/ http://michaelochurch.wordpress.com/2012/08/26/xwp-vs-jap/ . We are so desperate to establish ourselves as better than "those programmers" who crank out Java classes and can't function without an IDE... and I understand this, because I don't want to be lumped in with them on compensation or autonomy. That said, I find it sad that we haven't managed to sell what we do (solve problems using software) as something with intrinsic integrity. Because we've been raping the word "scientist" (e.g. "data scientist" for someone who worked with Hadoop once) for half a decade or more (and should really give it back to the people who have the right to use it) I would be more inclined to call myself a software strategist. Career-wise, I'm far past the level of the average engineer, and refuse to do (unless it's my own company, in which case I'll do the most undignified work just because it needs to get done) the sort of jobs that regular "software engineers" do in most companies (e.g. line-of-business Java-mines work) and I don't think of code as where I add value, but rather my knowledge about problem solving in a more general sense. I've probably evolved out of an engineering role (as defined by the business) and into that of a strategist who does some engineering and coding (especially because it can be a lot of fun). Of course, I'm hesitant to write a "software strategist" essay, because I don't want to see that title get mangled either.
eng_monkey 12y ago
Another article that assumes programmers are engineers. Since I like gardening, I suppose I should be called an environmental scientist.
- jacquesm 12y ago
  Horticultural engineer is the term I think you're looking for.
mparr4 12y ago
>You’d think that any programmer with a basic grasp of recursive algorithms would be hesitant to take a job as a software engineer. What does this mean? Is software engineering "below" programming? Do software engineers write less code? I've only been in the industry for a few years, mostly working on my own. I've assumed these titles were arbitrary. Are they not?
jacquesm 12y ago
Young people are very popular because they don't know how to negotiate their worth and hence are cheap. Older people may be more experienced but compared to hiring two or even three (more?) younger ones they will produce less code that will (probably) be less buggy and a bit better laid out but you won't be able to get them to sleep under their desks or work unpaid overtime. And those other advantages won't show right away. So in the end the younger guys get more done for the money and they are probably a lot more pliable.
- swatow 12y ago
  If you're so good at negotiating that you end up without a job because no one will pay what you demand, then you are actually bad at negotiating.
  - jacquesm 12y ago
    No, they get the job 6 months later, to clean up the mess at double the original pay...
    - swatow 12y ago
      That doesn't seem consistent with your first post. Overall, would you argue that older engineers do or don't suffer worse outcomes because they require higher pay? Is it the case that employers hire younger engineers instead, or is it the case that in the long run, older engineers get jobs at the higher pay they demand?
      - jacquesm 12y ago
        Younger people tend to be the ones hired early on in the life cycle of a product or company because they're cheap, easy to push into a mold (or so management likes to think) and can do just as good a job (or maybe even quicker) than older people in making something that visually looks like it might work. Then, as the project matures you'll find that that gained speed comes at a price, a price that will sooner or later offset the higher wages demanded by the ones that are further along in their life (dependents, more aware of their value). If there are young really good programmers (it does happen, I've met a couple) then they tend to be exploited even worse, they will end up creating a large amount of value for peanuts. But that's rare enough that it doesn't change the situation too much. So I don't see any inconsistency, it's just time shifted.
programminggeek 12y ago
I think of this role as "The Fixer". On most teams I've worked on there is someone who ends up doing the hard parts, knows the obscure parts of the system, and tends to end up optimizing queries, cleaning up algorithms, and generally guides the long term technical direction, whether they have a title to go with it or not. In my experience, while a software scientist or fixer is highly valued, they aren't necessarily highly paid moreso than anyone else on a team. Also, a technical lead is not always going to do this job or do it well if given it. The largest part of development doesn't rely on someone with this specialization to write the initial code for any feature, yet without this skillset a team will ship some terrible code without realizing it. I don't really see any group hiring for this kind of role.
cocoalabs 12y ago
✊🍆

89 points by carljoseph 12y ago

A fantastic article, and I hope we see more relevant research gathered under this banner. Not having a name for something can be a big problem in even knowing what's out there. As a side-point, I am often shocked that we don't see more research into how to make software teams more productive coming out of places like Google/Facebook/Microsoft etc, given that these companies (claim to) live and die by the effectiveness of their engineering. If a large amount of their expenditure is in software, to the point they'll buy entire companies just for the engineers, why aren't they financing the large-scale experiments that will settle certain questions for good?

collyw 12y ago
Is there any evidence that Facebook/Google/Microsoft are productive? They have a lot of money and a lot of engineers to throw at a problem. When I worked at an investment bank, the code quality was crap, and the staff turnover was high. Didn't matter, because they literally had enough money to throw at the problem and keep hiring more people.

gphil 12y ago

I don't understand how this role doesn't fall under the scenario of: "The engineer becomes a manager" In order to do a superlative job as a manager of a software engineering team, one has to master these same skills. Perhaps the term "Software Scientist" will help distinguish good software engineering management practices from bad ones, but I fail to see how "The Software Scientist" is not a management role.

cafebeen 12y ago
Also, most academic and research scientists manage a team of students and post-docs who do the actual lab work/analysis/writing. The article's description of a "software scientist" might be more accurately named "software research assistant".
jbarmash 12y ago
Agreed. The only way that the post makes sense to me is that if the organization is large enough, have a role that supports CTO / VPE's efforts and tried to do it in a more data-driven way. So this way CTO / VPE still have those responsibilities, but have a person who is fully focused on measuring the impact of changes and getting data. I think these would be very special roles and very few organizations, so not enough to really talk about it as an alternative to the two traditional tracks. That said, there is certainly there is value in this role.

gtirloni 12y ago

The first paragraph seems to imply that, to understand recursive algorithms, you need to be young. And rightly so, the software industry is not for old people. Did I get the argument right?

crimsonalucard 12y ago
I didn't get that, what does a software engineer with a basic grasp of recursion have to do with "software science"?
blackbagboys 12y ago
No, he is saying that if you understand recursive algorithms, you should be able to apply that reasoning to reason yourself out of a career on software, since it is a given that most developers will be forced out of engineering by a certain age

cafebeen 12y ago

To summarize, the article suggests making the "software science" job title to: * "reduce career path anxiety" * "enhance engineering organizations' self-understanding" * "advance the general understanding of the right way to develop software" * "raise awareness about the sponsoring engineering organization" * "bring out some of those 'big picture' thinkers from the woodwork" The third point is the only one close to the traditional role of a scientist, though "general understanding" takes more than data and statistics from a single organization. Otherwise, this seems to be a pretty blatant misappropriation of the name "scientist"!

jeffdavis 12y ago

Would such a person practice real science, or just apply the first test that cones to mind, and ignore all of the confounding factors?

hnnewguy 12y ago
I agree. This sounds ominous: >Statistical education. This is currently optional The data world is going to be an interesting place in the near future, filled with people running statistical tests they don't truly understand, and drawing spurious conclusions. But, I guess it already works that way in some fields.

greenyoda 12y ago

"By and large, programming is a young man’s game" As a middle-aged programmer who is a much better programmer now than I was when I was young, I'd argue with the article's initial premise.

AndrewDucker 12y ago
Agreed. It assumes that years of experience mean you shouldn't be programming. Which seems...odd to me.
- jvehent 12y ago
  Maybe the author understands programming as "writing lines of code", and not as "architecting software". Architecture requires experience. Writing the fastest quick sort in a given language is an exercise for fresh young brains.
  - AndrewDucker 12y ago
    "Writing the fastest quick sort in a given language is an exercise for fresh young brains." I can't see the reasoning here. Surely it's a job for someone who has written previous sorting algorithms, understands the trade-offs between speed and memory, and understand the pitfalls that can be fallen into when sorting different kinds of data?
    - collyw 12y ago
      An experienced brain would look up the pre-written libraries, and use the best of those rather than writing his own buggy implementation.
deleted 12y ago
[deleted]
frozenport 12y ago
>>As a middle-aged programmer What do your younger colleagues think?

gaius 12y ago

Actual LOL. So we have programmers, developers, software engineers, coders and now... Software scientists?? Oh well it will be a good filter on CVs.

Chronic30 12y ago
I'm a software physicist myself.
- ScottBurson 12y ago
  I think I'll go with "Software Alchemist" :-)
  - gaius 12y ago
    I forgot software architect too
- waterlesscloud 12y ago
  I keep reading this as software psychic. Which I guess is pretty much what most debugging turns out to be.
eng_monkey 12y ago
Do not forget about any one able to write R scripts is a Data Scientist now!

Mz 12y ago

The military is a "young man's game." Most new recruits are 18. But the people in charge, who do the strategizing and so on, are not. It sounds to me like the article is suggesting another way for experienced programmers to be held in high esteem, not suggesting that old programmers are no good. So I honestly do not understand most of the criticisms of this article being posted here on HN. Maybe there is no real difference between a "software scientist" and a manager. Maybe there is. But it doesn't sound to me like this piece in any way insults older programmers.

seanmcdirmid 12y ago
You have older NCOs in the military as well as senior officers, but for the most part rank and seniority is a pyramid in the millitary; just like most hyper competitive dev shops...there are older people at the top, just much fewer of them than younger people...and where did the other people go who fell off the pyramid? As a PhD holding computer scientist and practicing researcher, I cringe at the title "software scientist"....
- Animats 12y ago
  You have older NCOs in the military as well as senior officers, but for the most part rank and seniority is a pyramid in the military. More so than in civilian life. The US military has a formal "up or out" policy and a schedule: https://en.wikipedia.org/wiki/High_Year_Tenure https://en.wikipedia.org/wiki/High_Year_Tenure . That's for enlisted men. For officers, it's tougher, although more complicated; if you're passed over for promotion twice, you're usually out.
- Mz 12y ago
  It's fine if you cringe. :-) I am just saying that I simply do not see the slam on older programmers that others seem to be taking away from this article. It seems to me there are plenty of older programmers on HN and some of most esteemed folks here are not 22-25 years old and are, instead, a good bit older (in some cases, old enough to have kids that age, in theory if not in reality). (In fact, I suspect none of them are that young -- but I don't know all the ages of, say, everyone on the leaderboard, so I can't assert that as fact. Edit: Also, not everyone that age here can rightly be called a programmer.)

perlgeek 12y ago

If I ever reach the level of an IBM Fellow or similar as a software engineer, I won't care about whether you'll call me a "software scientist" or "man with long beard" or "that guy". I promise. > The engineer becomes a kind of nomadic hermit (“Principal Engineer”) "hermit" implies some level of isolation. Is that really the case of distinguished/principal/fellow engineers? Is there any evidence or data available?

michaelochurch 12y ago
Principal and Distinguished are, usually, titles added to convince HR to pay market salaries. Mgr: "We'll need $140k to get him." HR: "The band for Senior Engineer is $100k to 125k." Mgr: "So you can't do $140?" HR: "Nope." Mgr: "What if I hire him for a *Principal* Engineer position?" HR: "I need to talk to the CFO about whether we need Principal Engineers."

michaelochurch 12y ago

This is just another title upgrade (first "software engineer" in lieu of lowly "programmer", then "software scientist") that represents a half-baked understanding of the problem, which is that typical business-driven engineering doesn't work well, isn't very challenging, and is enough of an intellectual wasteland that, after 7-10 years, you'll have learned everything worth learning unless you become an elite engineer (e.g. Principal "hermit", thus named because top-1% engineers don't have much company except at a few dozen companies worldwide) or a manager. That all is true. The ghetto of the average line-of-business software engineer is not a place where one can be over 35 and unembarrassed about remaining there. This said, I think that the "data science" title is silly. It used to mean "what we call machine learning to make it sound more practical and less AI-ish", then it meant "watered down job that occasionally involves running an off-the-shelf regression or clustering algorithm", and then it meant "any job that involves data". I don't hold out higher hopes for "software scientist". We seem, in this industry, to have given up on restoring integrity to software engineering itself, so that everyone over 30 is strategizing to become some kind of XWP ("X Who Programs") rather than a JAP "Just A Programmer"), as I've discussed at length here: http://michaelochurch.wordpress.com/2012/08/26/xwp-vs-jap/ http://michaelochurch.wordpress.com/2012/08/26/xwp-vs-jap/ . We are so desperate to establish ourselves as better than "those programmers" who crank out Java classes and can't function without an IDE... and I understand this, because I don't want to be lumped in with them on compensation or autonomy. That said, I find it sad that we haven't managed to sell what we do (solve problems using software) as something with intrinsic integrity. Because we've been raping the word "scientist" (e.g. "data scientist" for someone who worked with Hadoop once) for half a decade or more (and should really give it back to the people who have the right to use it) I would be more inclined to call myself a software strategist. Career-wise, I'm far past the level of the average engineer, and refuse to do (unless it's my own company, in which case I'll do the most undignified work just because it needs to get done) the sort of jobs that regular "software engineers" do in most companies (e.g. line-of-business Java-mines work) and I don't think of code as where I add value, but rather my knowledge about problem solving in a more general sense. I've probably evolved out of an engineering role (as defined by the business) and into that of a strategist who does some engineering and coding (especially because it can be a lot of fun). Of course, I'm hesitant to write a "software strategist" essay, because I don't want to see that title get mangled either.

eng_monkey 12y ago

Another article that assumes programmers are engineers. Since I like gardening, I suppose I should be called an environmental scientist.

jacquesm 12y ago
Horticultural engineer is the term I think you're looking for.

mparr4 12y ago

>You’d think that any programmer with a basic grasp of recursive algorithms would be hesitant to take a job as a software engineer. What does this mean? Is software engineering "below" programming? Do software engineers write less code? I've only been in the industry for a few years, mostly working on my own. I've assumed these titles were arbitrary. Are they not?

jacquesm 12y ago

Young people are very popular because they don't know how to negotiate their worth and hence are cheap. Older people may be more experienced but compared to hiring two or even three (more?) younger ones they will produce less code that will (probably) be less buggy and a bit better laid out but you won't be able to get them to sleep under their desks or work unpaid overtime. And those other advantages won't show right away. So in the end the younger guys get more done for the money and they are probably a lot more pliable.

swatow 12y ago
If you're so good at negotiating that you end up without a job because no one will pay what you demand, then you are actually bad at negotiating.
- jacquesm 12y ago
  No, they get the job 6 months later, to clean up the mess at double the original pay...
  - swatow 12y ago
    That doesn't seem consistent with your first post. Overall, would you argue that older engineers do or don't suffer worse outcomes because they require higher pay? Is it the case that employers hire younger engineers instead, or is it the case that in the long run, older engineers get jobs at the higher pay they demand?
    - jacquesm 12y ago
      Younger people tend to be the ones hired early on in the life cycle of a product or company because they're cheap, easy to push into a mold (or so management likes to think) and can do just as good a job (or maybe even quicker) than older people in making something that visually looks like it might work. Then, as the project matures you'll find that that gained speed comes at a price, a price that will sooner or later offset the higher wages demanded by the ones that are further along in their life (dependents, more aware of their value). If there are young really good programmers (it does happen, I've met a couple) then they tend to be exploited even worse, they will end up creating a large amount of value for peanuts. But that's rare enough that it doesn't change the situation too much. So I don't see any inconsistency, it's just time shifted.

programminggeek 12y ago

I think of this role as "The Fixer". On most teams I've worked on there is someone who ends up doing the hard parts, knows the obscure parts of the system, and tends to end up optimizing queries, cleaning up algorithms, and generally guides the long term technical direction, whether they have a title to go with it or not. In my experience, while a software scientist or fixer is highly valued, they aren't necessarily highly paid moreso than anyone else on a team. Also, a technical lead is not always going to do this job or do it well if given it. The largest part of development doesn't rely on someone with this specialization to write the initial code for any feature, yet without this skillset a team will ship some terrible code without realizing it. I don't really see any group hiring for this kind of role.

cocoalabs 12y ago

✊🍆