58 ms·

Hey, wait – is employee performance Gaussian distributed?

328 points by timdellinger 2y ago

xphilter 2y ago
Yeah good luck. I don’t think any hr decisions have ever been about data; it’s about following norms. If you can get the rand corp or heritage foundation to adopt this policy then maybe corporations would look into it.
- timdellinger 2y ago
  Interestingly enough, I remember in my younger days being inspired by Rand Corp's 1950's era game theory work on e.g. mutually assured destruction. It later occurred to me that I don't need to be employed by a think tank to write think pieces! That being said, I like to think that startups growing into large corporations have an opportunity to be better when it comes to things like performance management.
  - hobs 2y ago
    As soon as the market actually incentivizes it, which it almost never does, it will get better. Most of the big companies just throw endless interviews, high pressure firings, and a lot of money at the problem and make the people below them solve the rest of the problems. They see how much they are paying for the mess, but any medium term effort is torpedoed because of all the other things the business focuses on (lack of resources for the process and training), and other powerful individuals who want to put their own brand on hiring and firing who have significantly more ego than sense.
- thrance 2y ago
  The Heritage Foundation would probably fire every competent employees and replace them with partisan sycophants, like they plan to do with America in Project 2025.
datadrivenangel 2y ago
If you assume that people are promoted to their level of incompetence -- terminal responsibility level, then you would expect that level adjusted performance should approach a Gaussian?
- riehwvfbk 2y ago
  No, because there simply aren't enough high-level employees at the top in any given company for a meaningful sample. You'd have to compare across companies; I guess the stock market does that indirectly.
riazrizvi 2y ago
This is a well constructed empty argument because it glosses over the central concern, ‘employee performance’. Without defining that we have no idea what the graph represents.
- michaelmior 2y ago
  I'm not sure this is the argument the author is making, but you could claim that the rest of the argument is true for any (or most) reasonable measure of employee performance that a company actually cares about.
  - nradov 2y ago
    You could claim anything, but is there hard quantitative data to support such a claim? Or are we just guessing?
    - michaelmior 2y ago
      The author presents some data in the article. Also, the absence of hard quantitative data doesn't necessarily make it a complete guess. (At least not any more than starting with the assumption of a Gaussian distribution.)
- bhickey 2y ago
  For analyses like this it just doesn't matter. Pick a metric and measure it over your workforce. Across the universe of salient metrics of interest you won't see a gaussian across your workforce. In a previous job I modelled this and concluded that due to measurement error and year-over-yead enrichment, Welchian rank-and-yank results in firing people at random.
  - pembrook 2y ago
    All of Jack Welch’s management tactics should be considered suspect now. His performance at GE was 100% fueled by financial leveraging that blew up in 2009, basically killing the company. Nobody should be taking management lessons from this guy.
nonameiguess 2y ago
It's worth hammering on this point as much as possible hoping a few people listen, but there is at least one other important point about employee performance. If you're allocating bonuses, a single year's performance is probably a good way to do that, assuming you can accurately measure it. When you're talking retention and promotion, though, you're making a prediction of future performance, possibly at a variety of different jobs. That is even harder to do and more poorly reflected in the last year's results. You have some analogies to sports performance in this article, and you see this kind of thing all the time there. Guy does great in a single year, gets a huge, possibly long-term contract, then tanks. On the other hand, one of the better dynasties of the past decade was accomplished by the Golden State Warriors in the US NBA thanks to underpaying one of the all-time great players in NBA history because he suffered a series of ankle injuries early in his career and scared off other suitors. Single-year performance isn't necessarily reflective of a person's true mean abilities, and their place in the Pareto distribution won't be the same at all levels of advancement and responsiblity, either. The problem, from a company's perspective, is you probably need to retain everyone at least five years, and actually give them a wide variety of assignments in that time, to really get any usable data about their long-term prospects.
- stego-tech 2y ago
  Literally this. I’ve been banging on about this my entire career, not that corporate leaders tend to listen to the riff-raff. Especially in tech companies, they tend to only evaluate promotions and raises based on the past half-year of work, rather than a repeated pattern of successes across a diverse array of tasks and backgrounds over a significant period of time (years); even then, you only get the promotion if you’re on the right team, doing the right work, at the right time, and for the right leader. This leads to otherwise stellar performers going elsewhere, because the janitors, maintainers, and firefighters in an organization never get properly rewarded, respected, or recognized by leaders. Said leaders pass this off as “bad performers”, failing to realize the importance of superb talent working on less-than-stellar projects that keep the company running efficiently. The only people who benefit from performance reviews are shareholders whose price pops when layoffs happen, and those who game the system for their own political ends. Top talent never really thrives in these, because they’re too busy doing actually meaningful and important work.
throwaway48476 2y ago
Setting aside the issue of defining a function for 'employee performance', this glosses over the invisible interactions. An employee in a dysfunctional organization will perform worse than if they were in a well functioning one because they don't have to waste time dealing with people and processes that are a hindrance.
dogleash 2y ago
To me the biggest insight here is that no matter what data science you're trying to do on a group of employees, the people you already have decided should be fired or promoted from that group are outliers and should be removed from the sample. There are certainly times that you would want them included, but those can be classified under "budgeting," not gaining insight on a workforce.
- ses1984 2y ago
  Doesn’t the inclusion or exclusion of these people heavily depend on what type of insight you’re trying to get out of the data?
bparsons 2y ago
Unless you are measuring the output of people on simple assembly lines, it is very difficult to define "performance". In a properly functioning team, people perform different, discrete roles which are probably not entirely understood by other team members or management.
iambateman 2y ago
As employees, our expectations for performance management come from the system of giving grades in school. What's interesting is that school grades often doesn't follow a normal distribution, especially for easier classes. I suspect that getting an "A" was possible for 95%+ of students in my gym class and only 5-10% of the students in my organic chemistry class. In the same way, some jobs are much easier to do well than others. So we should expect that virtually all administrative positions will have "exceptional" performance, which is to say that they were successful at doing all of the tasks they were asked to do. But for people who's responsibility-set is more consequential, even slightly-above average performance could be 10x more meaningful to the company.
- nightski 2y ago
  Having a shifted mean doesn't mean they aren't a normal distribution. Not saying they are necessarily, but the anecdote you are providing isn't convincing.
  - dowager_dan99 2y ago
    if your scale doesn't have the atomic values at the top end to differentiate the data it's not a normal, it's Pareto or Zipf or some other power law.
  - kurthr 2y ago
    Perhaps, but due to the sampling of the distribution you would likely never know. If 95% of your samples fit in the top 3 bins, you can’t say much at all with certainty. Poisson, Gaussian, binomial, Boltzmann, gamma…
    - deleted 2y ago
      [deleted]
  - marian_ivanco 2y ago
    That is not IMHO what he is trying to say, you don't shift the distribution, you measure if somebody passed a test. I the test is "passable" then one side of "distribution" is at least cut off. E.g. it's normal (and sometimes expected) that the whole class will pass without issues.
hemloc_io 2y ago
Cool data/idea, and anecdotally lines up with my experiance at BigCos from a coworker perspective. But in my experiance employee perf evals are more political than data based. At the end of the day a lot of mgmt at BigCo, esp these days, wants that 10% quota for firing as a weapon/soft layoff and the "data" is a fig leaf to make that happen. More generously it's considered a forcing function for managers to actually find underperformers in their orgs, even if they don't exist. Either way it's not really based on anything other than their own confirmation bias. IME the scrutiny of perf evaluation is basically tied to the trajectory of the company and labor market conditions. Even companies with harder perf expectations during the good times of ~2021 relaxed their requirements.
jampa 2y ago
Going through some performance reviews as a manager, I always try to push back a bit against the bell curve. It kinda reminds me of the "stack ranking.". There are also some factors to be considered: If you are in a hiring freeze or not promoting, most of the curve should shift right, assuming you are hiring great people. They will probably perform better quarter after quarter. Some might counter-argue that if everyone performs better, this should be the "new expectation," but I disagree: the market sets expectations. If you have someone at a senior level with expectations of staff, for example, they won't be in the company for long. I hired many great engineers who later said they only looked for a new job because they were never promoted despite being overperformers.
bhouston 2y ago
I would feel better if this was derived from empirical data rather than just rhetoric. This seems super testable, no? There is probably a ton of data already in different industries with regards to productivity. Even if human talent have a Pareto distribution (which is not clear), the people employed by a company are a selected sub-set of that population, which would likely have a different distribution depending on how they are selected and the task at hand. I think that any of these simplified distributions are likely not generalizable across companies and industries (e.g. productivity of AWS or Google employees are likely not distributed like employees of MacDonalds or Wal*Mart because of the difference in hiring procedures and the nature of the tasks.) Get hard data within the companies and industry you are in and then you can make some arguments. Otherwise, I feel it is too easy to just be talking up a sand castle that has no solid footing.
- drcwpl 2y ago
  Agree with you - although, rhetorically speaking, I have come across many instances which the author refers to "of low performers are 3x as common as high performers." This is unfortunate as I always think do your best, and as Tyler Cowen states - Average is Over. So agree it would have been way better to use empirical data to back up this claim especially.
- KK7NIL 2y ago
  The problem is that intellectual productivity is generally not possible to measure directly, so you instead end up with indirect measurements that assume a Gaussian distribution. IQ is famously Gaussian distributed... mainly because it's defined that way, not because human "intelligence" (good luck defining that) is Gaussian. If you look at board game Elo ratings (poor test for intelligence but we'll ignore that), they do not follow a Gaussian distribution, even though Elo assumes a Gaussian distribution for game outcomes (but not the population). So that's good evidence that aptitude/skill in intellectual subjects isn't Gaussian (but it's also not Pareto iirc).
  - bhouston 2y ago
    > so you instead end up with indirect measurements that assume a Gaussian distribution. 100%. I was going to write something similar. > If you look at board game Elo ratings (poor test for intelligence but we'll ignore that), they do not follow a Gaussian distribution, even though Elo assumes a Gaussian distribution for game outcomes (but not the population). So that's good evidence that aptitude/skill in intellectual subjects isn't Gaussian (but it's also not Pareto iirc). Interesting, yeah, Elo is quite interesting. And one can view hiring in a company as something like selecting people for Elo above a certain score, but with some type of error distribution on top of that, probably Gaussian error. So what does a one sided Elo distribution look like with gaussian error in picking people above that Elo limit?
doctorpangloss 2y ago
This article: "Wouldn't it be cool if when you measure employee performance, it turned out to fit a Pareto distribution better than a Gaussian?" Would that be cool? We could posit the implications of all sorts of improbabilities. But I feel more strongly about how cool it would be that P = NP. All this aside, being laid off sucks - being pushed out, even when you're a high performer, sucks even more. The truth is that "data science" does not help you process grief the way reading Dostoevsky does, so maybe getting an A in your liberal arts education is valuable even when you are working as a software developer.
morkalork 2y ago
If you ever look at tranditional human-driven sales data, you'll often see a small percentage of top performers absolutely dominating the total sales volume. So yes, employee performance is not Gaussian at all.
Joel_Mckay 2y ago
"Hey wait - is [arbitrary metrics] Gaussian distributed?" =3
- riazrizvi 2y ago
  This whole thread smh. It feels like a military power convinced it can win a war by flying around in airplanes at 30,000’ and they are here vigorously discussing their insane tactics. It’s time for me to leave Silicon Valley.
wing-_-nuts 2y ago
One reason I'd never work for a company with a 'bottom 10% gets PIP'd' mentality is that it directly conflicts with my goal of self development. Of course I want to be on a great team where everyone performs better than I do. That's how I hone my craft! It just seems really wasteful to have to cull the bottom 10% of every team, even if that team is performing well. I wish there was a list of companies that subscribed to that mentality, so I could avoid them.
dmurray 2y ago
> For what it’s worth, human height is also Gaussian, and that’s correlated with workplace success. Height is generally not considered to be Gaussian and this is exactly the kind of statistics mistake the author seems to be accusing employers of. Adult height is somewhere between Gaussian and bimodal.
- timdellinger 2y ago
  Fair enough. Perhaps better stated as "adult human height is approximately Gaussian for a given biological sex", with an asterisk that environmental factors stretch the distribution. I love the anecdote that people born in the American colonies came back to England to visit family, and were remarkably taller compared to their cousins due to environmental factors.
warrentr 2y ago
In the work rules book about google, Bock claims (apparently using a lot of real data from google) that employee performance follows a power law distribution.
deleted 2y ago
[deleted]
seiferteric 2y ago
A lot of focus on employee performance, but relatively little on management performance. I always wonder how a once great company can slowly decline into irrelevance. Take yahoo for example, it could only be due to management failure over several decades right? How can companies optimize for management performance?
- bornfreddy 2y ago
  Firing 10% each year would be a great start in many companies. ;)
  - fargle 2y ago
    firing the right 10% each year would be great, but that's the problem. can you expect the management correctly identify the right people to fire, especially when they are themselves heavily over-represented in that class?
    - seiferteric 2y ago
      Maybe employees get a vote on who to fire.
      - fargle 2y ago
        well, that's simply not how any business works, even in the most extreme naively theoretical socialistic case. why would a company do that? - employees don't (generally) own the company. even for some reason they were forced to, in practice, a cadre of employees (most likely the ones you'd want to fire in the first place), would instantly form some new ad-hoc political layer to exercise political power and control the organization, voting, etc. so you just end up with a layer of useless management and a independent layer of useless politicians, neither of which are willing to fire themselves.
AtlasBarfed 2y ago
1) performance reviews are never aligned with employee value, because companies are strongly invested to take excess production from employees and transfer it to management, secondarily shareholders 2) the are also not aligned with the replacement cost of employees because the religion of management is that labor is effortlessly replaceable and low value 3) employee retention is not aligned with corporate performance in Machiavellian middle management, it is aligned with manager promotion for things like loyalty and maintaining fiefdom power, budgetary size, headcount, etc 4) there are no absolute or ever directly derived metrics in software development that have ever worked, to say nothing of other positions Those are off the top of my head.
xmly 2y ago
Well, managers are trying to make it Gaussian, but underlying is actually power law.
crazygringo 2y ago
This is very unconvincing. The author already admits one reason why: > But there are low-performing employees at large corporations; we’ve all seen them. My perspective is that they’re hiring errors. Yes, hiring errors should be addressed, but it’s not clear that there’s an obvious specific percentage of the workforce that is the result of hiring errors. I think it is clear that we expect a certain percentage of hiring "errors". And that they are not binary but rather a continuum. And that there are lots of other factors like employees who were great when they were hired but stopped caring and are "coasting" or just burnt out, who got promoted or transferred when they shouldn't have been and are bad at their new level/role, and so forth. The Pareto distribution isn't particularly relevant here, because a hiring process isn't trying to get a whole slice of the overall labor market with clear cutoffs. For any position, it's trying to maximize the performance it can get at a given salary, and we have no reason to expect the errors it makes in under- and over-estimating performance to be anything but relatively symmetric. So a Gaussian distribution is a far more reasonable assumption than a slice of the Pareto distribution, when you look at the multiplicity of factors involved.
- wavemode 2y ago
  > So a Gaussian distribution is a far more reasonable assumption than a slice of the Pareto distribution It's not an assumption. See the evidence referenced in the footnotes.
  - crazygringo 2y ago
    Incorrect. It is absolutely an assumption. The "evidence" in the footnotes is about national salary data. Not the distribution for any individual position at a company. And it is entirely possible (and probable) that performance at each position is distributed as a Gaussian, and all those Gaussians add up to a Pareto at a population level. But you simply cannot take national-level data and assume it applies at the micro level. That's not how statistics works.
    - wavemode 2y ago
      > And it is entirely possible (and probable) that performance at each position is distributed as a Gaussian Find me any research agreeing with this statement (spoiler - you won't, because researchers have broadly reached consensus to the opposite)
jedberg 2y ago
One of the things I loved about working at Netflix was that the base assumption was that everyone was a top performer. If you weren't a top performer, you were given a severance check. The analogy we used was a sports team. Pro sports teams have really good players and great players. Some people are superstars, but unless you're at least really really good you're not on the team. Performance and compensation were completely separate, which was also nice. Performance evals were 360 peer reviews, and compensation was determined mostly by HR based on what it was costing to bring in new hires, and then bumping everyone up to that level. So at least at Netflix 10 years ago, performance wasn't really distributed at all. Everyone was top 10% industrywide.
- brabel 2y ago
  It's really difficult for me to believe that they really got 10% top performers. For one, knowing the cut-throat nature of employment there, I would expect only a minority of developers would be willing to try working there, despite the awesome rewards. Another reason I really don't trust that to be true is that I've never seen a good way to measure who is a top performer and who is not. I don't think there's one, people are good in different things, even within the same job... for one assignment, Joe may be the best, but for another, Mary is the winner (but again, to measure this reliably and objectively is nearly impossible IMHO for anything related to knowledge work - and I've read lots of research in this area!). Finally, just as a cheap shot at Netflix, sorry I can't resist as a customer: they absolutely suck at the most basic stuff in their business, which is to produce good content in the first place, and very importantly, NOT FREAKING CANCEL the best content! I won't even mention how horrible their latest big live stream was... oh well, I just did :D.
  - kube-system 2y ago
    > the most basic stuff in their business, which is to produce good content in the first place, and very importantly, NOT FREAKING CANCEL the best content! It isn't that simple. Making money from content is not 1-to-1 related with the quality of the content. There are many examples of great content that doesn't make money, and many examples of content that makes a lot of money that isn't great. Also there are many differing opinions on what 'great content' even is.
ianbicking 2y ago
"IQ is Gaussian" – it was pointed out somewhere, and only then became obvious to me, that IQ is not Gaussian. The distribution is manufactured. If you have 1000 possible IQ questions, you can ask a bunch of people those questions, and then pick out 100 questions that form a Gaussian distribution. This is how IQ tests are created. This is not unreasonable... if you picked out 100 super easy questions you wouldn't get much information, everyone would be in the "knows quite a lot" category. But you could try to create a uniform distribution, for instance, and still have a test that is usefully sensitive. But if you worry about the accuracy of the test then a Gaussian distribution is kind of convenient... there's this expectation that 50th percentile is not that different than 55th percentile, and people mostly care about that 5% difference only with 90th vs 95th. (But I don't think people care much about the difference between 10th percentile and 5th... which might imply an actual Pareto distribution, though I think it probably reflects more on societal attention) Anyway, kind of an aside, but also similar to what the article itself is talking about
- FredPret 2y ago
  This is a subtle aspect of intelligence measurement that not many people think about. To go from an IQ of 100 to 130 might require an increase in brainpower of x, and from 130 to 170 might require 3x for example, and from 170-171 might be 9x compared to 100. We have to have a relative scale and contrive a Gaussian from the scores because we don’t have an absolute measure of intelligence. It would be a monumental achievement if computer science ever advances to the point where we have a mathematical way of determining the minimum absolute intelligence required to solve a given problem.
  - logicchains 2y ago
    >It would be a monumental achievement if computer science ever advances to the point where we have a mathematical way of determining the minimum absolute intelligence required to solve a given problem For a huge number of problems (including many on IQ tests) computer science does in fact have a mathematical way of determining the minimum absolute amount of compute necessary to solve the problem. That's what complexity theory is. Then it's just a matter of estimating someone's "compute" from how fast they solve a given class of problems relative to some reference computer.
sangnoir 2y ago
> Performance management, as practiced in many large corporations in 2024, is an outdated technology that is in need of an update Author made a couple of fundamental mistakes: the first is they assume employees are (or should be) paid according to how much they "individually" earned the company. Employers strive to pay employees the minimum they can bear, on employer's terms. Those terms are information asymmetry and a Gaussian distribution. Fairness is the last thing one should expect from employers, but being honest about this is not good for morale, so instead, they rely on keeping employees uninformed, while the employers collude to gather everyone's remuneration history via the Work Number. The second mistake they made is assume that companies would prioritize being lean and trimming the mediocre & bottom 5%. There are other considerations, combined productivity is more important than having individual superstars working on the shiniest features. How much revenue do you think a janitor or café staffer generates? Close to zero. The same goes for engineering. Someone has to do the unglamorous staff, or you end up with a dysfunctional company, with amazing talent (on paper). Edit: there's an infamous graph that shows when aggregate worker productivity and average income. The two tracked closely, rising in tandem until the 1970s, where they got decoupled. With income becoming much flatter, and productivity continuing to rise. That's how the world has been for the past 50 years on the macro and the micro
- diggan 2y ago
  > the first is they assume employees are (or should be) paid according to how much they earned the company From the perspective of a employee and/or human, that does seem like the most fair way of distributing what the company earns, sans the money that gets reinvested straight back into the business itself. But I'd guess that'd be more of a co-operative, and less like the typical for-profit company most companies are today.
  - dingnuts 2y ago
    Did you even finish reading the comment you're replying to? It explicitly explains why employees who do not generate revenue are still valuable. What you're describing, that money would go to whoever brings in revenue directly, is the myopic viewpoint of Sales with an emphasis on closing deals with nothing else. If it wasn't for the rest of the work, there'd be nothing to sell!
TrainedMonkey 2y ago
Employee performance MEASUREMENT appears to be Gaussian distributed. To my first simple, and let's be real probably somewhat wrong, approximation there are roughly 3 things that go into it. 1. There is a certain skill in communicating all the important things you've done, we shall lump likability + politicking into this one for convenience. 2. There is a premium that is placed on shiny new features and saving the day heroics. A lot less priority is placed on refactoring and solving the problems before they require heroics. 3. Finally there are individual's technical and self-management skills. I.E. it's important to work on important things and be good at it.
philipov 2y ago
> How much revenue do you think a janitor or café staffer generates? Close to zero. The same goes for engineering. Someone has to do the unglamorous staff, or you end up with a dysfunctional company, with amazing talent (on paper). If the company would be dysfunctional without that janitor or software engineer, and not bring in as much revenue as a result, it sounds like the model that attributes close to zero revenue to them is already dysfunctional. If the company can't function without the janitor, then a significant portion of the revenue of the company should be attributed to them.
- sangnoir 2y ago
  Sound like you're expecting employers to strive for fairness. Instead, they are striving for profits for the capital class. The labor class gets the minimum possible amount to reach the shareholders primary goal.
  - philipov 2y ago
    It sounds like you're confusing what they do currently and what the system should be set up to encourage instead. That things are broken right now is not a valid argument in favor of the status quo. The point you make only proves why it is so important that unions should have as much economic power as corporations do, so that the buy and sell sides of the labor market have negotiation parity.
    - sangnoir 2y ago
      I'm being descriptive, bot prescriptive: I'm stating what the priorities are under a capitalist system without the rose-colored glasses offered by the Just-world fallacy.
      - philipov 2y ago
        In a well-functioning capitalistic system, the sell side of the labor market has equal power with the buy side. When the buy and sell sides of a market have a huge power imbalance, this leads to market failure, which is contrary to the goals of a capitalist system, as it results in inefficient allocation of capital.
irrational 2y ago
Is it Q4 at a lot of companies? How many companies align their fiscal calendar with the yearly calendar? Our Q4 is March-May.
spyckie2 2y ago
So… 1) treat poor performers as bad hires and ignore them in your dataset 2) treat 10x performers as needing to be promoted and also ignore them in your data 3) treat everyone else as relatively equal …and use “Pareto distribution” and “no one has mentioned this before” to write a blog post? Is the point of the article to get people who disagree with 10% corporate culling a pseudo intellectual economic buzzword argument to stroke their hatred of an inefficient hr practice? If so: 1) 10% culling in performance review is a mechanism to cull “bad hires”. I find it difficult to understand how the author can argue it’s a bad practice and then state that you cull bad hires from your dataset without thinking that they are the same thing or at least largely overlapping. 2) If the author is proposing to separate performance review, culling bad hires, and promotions, into 3 separate systems and assume no overlap, he should think through the structural issues more. While it’s possible to design a management structure where the organization is at a constant state of no bad hires, all 10xers promoted, that is putting a lot of responsibility on individual managers to run review, culling and promotion by themselves at a very high level. It’s brittle - a few bad managers not running the system can easily leave your organization bloated with bad hires and no fallback (fallback = performance review process). 3) The system of performance review is equally about risk management to the business as it is about rewarding your employees. IMO, the author’s framing simplifies the problem too much and pushes the complexity out for other people to deal with. It’s the kind of thinking that is damaging to organizations… I wonder if there is a process to cull this kind of thinking from your org… wait what time of year is it??
29athrowaway 2y ago
I guess developers should have a pay structure similar to sales when you make part of your money from bonuses tied to results. But those results are hard to evaluate because shipping something fast can have bugs found after the reward date.
wavemode 2y ago
This concept is not new - see [0]. There's ample research that Welchian stack ranking, and assuming a Gaussian distribution of employee performance, is not well-founded. Even its original pioneers (General Electric) have abandoned the practice (see [1]). Not sure why there are so many commenters here defending the Gaussian model. Most researchers at this point agree that a pareto distribution is more realistic. [0]: https://hbr.org/2022/01/we-need-to-let-go-of-the-bell-curve https://hbr.org/2022/01/we-need-to-let-go-of-the-bell-curve [1]: https://qz.com/428813/ge-performance-review-strategy-shift https://qz.com/428813/ge-performance-review-strategy-shift
uoaei 2y ago
I suspect you can dig into any metric here and find that they are explicitly determined in terms of an assumption of underlying normality.
_vaporwave_ 2y ago
> a helpful order of magnitude estimate is that the hiring process all told costs the company approximately a year’s salary It feels weird to gloss over this since transaction costs this high have a huge impact on how the system should be designed.
igorkraw 2y ago
The author looks at "observables" of performance without considering whether there might be confounders such as those discussed in great nuance here https://onlinelibrary.wiley.com/doi/full/10.1111/joes.12328 https://onlinelibrary.wiley.com/doi/full/10.1111/joes.12328 . He cites similar work by William Shockley who taught both electrical engineering and scientific racism at Stanford https://en.wikipedia.org/wiki/William_Shockley https://en.wikipedia.org/wiki/William_Shockley (no swipe at the author, just pointing at the biased motiviations of some of the researchers foundational to the idea of "high performers"). In general, when you see pareto structures or power laws, you should think of compound or cascade effects, which in human structures generally means some form of social mediation. Affinity for a desireable skill might be gaussian, but the selection process means that the people who _get_ to do that skill might become pareto shaped because if you aren't much better than the next guy, you wouldn't stably stay at the top. Similar logic can hold for other expressions. In general, I wish more people would read https://blackwells.co.uk/bookshop/product/Causality-by-Judea-Pearl/9780521895606 https://blackwells.co.uk/bookshop/product/Causality-by-Judea... or at least the more accessible https://mixtape.scunning.com/ https://mixtape.scunning.com/ before starting to conjecture from data about social systems - the math will tell you what you can and cannot speculate on. (fun exercise: draw the causal models of IQ in https://dagitty.net/ https://dagitty.net/ and ponder the results)
drc500free 2y ago
I've recently been working with a lot of service center productivity data. Staff productivity (customers/hour) is pretty close to a gaussian, with some skew towards many slight underperformers and few overperformers. However, any single customer interaction is exponential or weibull distributed.
PaulHoule 2y ago
It depends on the job. If you are interested in the American caste system you should read this classic https://www.amazon.com/Remember-me-God-Myron-Kaufmann/dp/B00005XSLQ https://www.amazon.com/Remember-me-God-Myron-Kaufmann/dp/B00... Which tells the story of a Jewish person who fails to persevere against prejudice in a multifaceted and sensitive way. In one scene he gets a job as a bank teller and then realizes in some jobs you’ve got the potential to screw up but no potential to distinguish yourself. The world needs people to milk cows every morning, a job you can screw up but not do it 10x better than competent, there is no Pareto or other “exceptional events” distributions for many essential jobs. ER doctors, taxicab drivers, astronauts, etc. (Productivity is a product of the system + the people) I worked on one system that had a 40 minute build if you wanted it to be reliable, the people I picked it up from could not build it reliably which is why the project has been going in circles for 1.5 years before I showed up. With no assistance (and orders that I was not supposed to spend time speeding up my build because it didn’t directly help the customer) I got it to a 20 minute build. Other folks on the team thought I was a real dope because my build took too long and I was always complaining but they couldn’t build it reliably at all.. I mas two major releases of a product with revolutionary performance in one year at which point I felt that I’d done the honorable thing and that I’d feel less backlash anywhere else whether or not I was creating more value —- so I moved on, and was told by recruiters that they hadn’t found a replacement for me in six months. Had the place I was working at had a 2 minute build they might never had hired me because they would have had the product ready long before.
hammock 2y ago
Why would performance be pareto distributed? Not saying it isn't, just wish we would unpack that idea a bit more. IQ and other personality traits are gaussian, with which I would expect performance to be correlated But, the mythical "10X employee" would seem to imply pareto, along with 80/20 notions of both personnel and an individual employee's day-to-day workload How do we resolve this dichotomy?
estebarb 2y ago
Some years ago I started doing graphs of code contributions across the year (yeah, wrong thing to measure, I know). A funny thing is that people considered "high performers" could be made the worse performers depending on how you cut the data. Basically, performance had a wave behavior, and nobody was at 100% all the time. That is a good argument for diverse hiring: people will have bad days/seasons, fact of life. If the team is diverse is less probable that those bad days will correlate between different employees.
- mdnahas 2y ago
  I did that once and one contributor had a huge negative value. His team had reimplemented a feature and his major contribution was deleting the old version.
thesz 2y ago
> IQ is Gaussian. The Big Five Personality Trait known as Conscientiousness is likewise Gaussian. For what it’s worth, human height is also Gaussian... Height cannot be negative, thus, it is not Gaussian. IQ cannot be negative too. Great many things that most people think are Gaussians, are not. One of such distributions that describe one-sided values, log-normal distribution (logarithms of values are distributed normally) has interesting property that for some d values x=mean+d are more probable than values x=mean-d (heavy tail). Also, sum of log-normal-distributed values does not converge to Gaussian distribution.
- hinkley 2y ago
  > IQ cannot be negative Have you been keeping up with current events?
hinkley 2y ago
Doesn’t quite work with Heart Shaped Box, but ok.
soniman 2y ago
It's the marginal dollar that contributes profit so the marginal employees are actually the most profitable.
directevolve 2y ago
I had assumed stack ranking was specifically designed to force managers to fire low performers, without relying on their individual judgment. Since nobody likes to fire, this overcomes the inertia, and since relying on personal judgment exposed you to legal risk and principle agent problems, a simple rule was substituted. The author’s proposal to go back to managerial discretion would of course be incompatible with that intention. I do wonder whether those implementing stack ranking are really that committed to a particular statistical model of employee productivity, or if they’re trying to solve a human and legal problem with an algorithm.
cynicalpeace 2y ago
If you take this as true- what does it imply?
losthalo 2y ago
X + Y + [XY] = 8 X = the individual's contribution Y = the contribution of the system they work within [XY] = the interaction of the individual with the system 8 represents some measure of productivity, e.g., rate of errors, millions of dollars in profit, whatever you're measuring The person who can solve for X is competent to rate people on their performance. What to do instead of (destructively) rating people? Build better systems for doing the work, make their work easier, give them psychological safety and job security so they can relax and enjoy their work and share better methods with each other. (All paraphrased from W. Edwards Deming.) Competition within organizations is for amateurs.
- Ginger-Pickles 2y ago
  CTRL+F “Deming” - Thank you. Organizations with vitality reason about the operation of the whole system, not simply the performance of actors in isolation.
graycat 2y ago
> Hey, wait – is employee performance Gaussian distributed? Well, the Gaussian distribution gives positive probability to any interval of the real line, including the whole real line (probability 1), so, strictly speaking, no. But maybe the issue is a distribution with a bell curve or even with just a unique maximum and falling off monotonically from that maximum. Well, then, in my college teaching, still no: Instead, commonly, roughly, there were three kinds of students: (1) understood the material at least reasonably well, (2) understood some of the material a little, and (3) should have just dropped the course but from me got by with a gentleman C. So, the distribution had a peak for each of (1) -- (3), three peaks, no Gaussian! Approximate Gaussian is guaranteed, under meager assumptions, from the central limit theorem (CLT) of averaging random variables, the easiest case, independent, identically distributed (IID), and, more depending on how advanced the CLT proof is. A proof due to Lindeberg-Feller long was, maybe still is, regarded as the most powerful CLT. Apparently ~100 years ago, especially in education, the CLT was commonly regarded as standard, true, without question, maybe some law of nature. Maybe some of the people measuring IQ, SAT scores, etc. also thought this about the Gaussian. For me, I, in mathematical and applied probability, care first about finite expectation, conditional independence, independence, several convergence results (e.g., the martingale convergence theorem), then IID, and hardly at all, Gaussian.
psychoslave 2y ago
The main take away, to my mind is "are we measuring the right things?" Like, is the system helping to maximize happiness distribution within humanity while maintaining biodiversity in its highest concomitant expectable dynamics?
pajko 2y ago
The problem is that the performance cannot be measured really, do there's nothing to talk about. KPIs like LOC/month or resolved tickets/month does not tell anything. Nor does completed projects/year, nor does the project's size. Name a single thing which does not depend on external stuff. Have fun with an idiotic client, or an idiotic management. If you want to remove the people who do not work, then do that. If you do not know who they are, fire your managers. Or better, fire yourself too, as it seems like you dont' know what you are doing. If you want to lower costs, fire managers and sales people first.
mdnahas 2y ago
I’m an economist who has looked into something like this. This paper is about intra-company performance. I’ve look at incomes of the whole population. There, you find two distribution. Wages seem to follow a log-normal distribution, while income from investments follows a heavy-tailed distribution, like a Pareto. Wages tend to be smaller than asset income. Top sports players and musicians work for wages and become billionaires. Startup founders, who own assets, become trillionaires. Obviously, there are differences. Wages are not productivity. (But the article didn’t say how productivity was measured.). Also, a company can choose who joins and leaves it. So one company’s wage distribution doesn’t have to follow the distribution of the wider economy.

328 points by timdellinger 2y ago

Yeah good luck. I don’t think any hr decisions have ever been about data; it’s about following norms. If you can get the rand corp or heritage foundation to adopt this policy then maybe corporations would look into it.

timdellinger 2y ago
Interestingly enough, I remember in my younger days being inspired by Rand Corp's 1950's era game theory work on e.g. mutually assured destruction. It later occurred to me that I don't need to be employed by a think tank to write think pieces! That being said, I like to think that startups growing into large corporations have an opportunity to be better when it comes to things like performance management.
- hobs 2y ago
  As soon as the market actually incentivizes it, which it almost never does, it will get better. Most of the big companies just throw endless interviews, high pressure firings, and a lot of money at the problem and make the people below them solve the rest of the problems. They see how much they are paying for the mess, but any medium term effort is torpedoed because of all the other things the business focuses on (lack of resources for the process and training), and other powerful individuals who want to put their own brand on hiring and firing who have significantly more ego than sense.
thrance 2y ago
The Heritage Foundation would probably fire every competent employees and replace them with partisan sycophants, like they plan to do with America in Project 2025.

datadrivenangel 2y ago

If you assume that people are promoted to their level of incompetence -- terminal responsibility level, then you would expect that level adjusted performance should approach a Gaussian?

riehwvfbk 2y ago
No, because there simply aren't enough high-level employees at the top in any given company for a meaningful sample. You'd have to compare across companies; I guess the stock market does that indirectly.

riazrizvi 2y ago

This is a well constructed empty argument because it glosses over the central concern, ‘employee performance’. Without defining that we have no idea what the graph represents.

michaelmior 2y ago
I'm not sure this is the argument the author is making, but you could claim that the rest of the argument is true for any (or most) reasonable measure of employee performance that a company actually cares about.
- nradov 2y ago
  You could claim anything, but is there hard quantitative data to support such a claim? Or are we just guessing?
  - michaelmior 2y ago
    The author presents some data in the article. Also, the absence of hard quantitative data doesn't necessarily make it a complete guess. (At least not any more than starting with the assumption of a Gaussian distribution.)
bhickey 2y ago
For analyses like this it just doesn't matter. Pick a metric and measure it over your workforce. Across the universe of salient metrics of interest you won't see a gaussian across your workforce. In a previous job I modelled this and concluded that due to measurement error and year-over-yead enrichment, Welchian rank-and-yank results in firing people at random.
- pembrook 2y ago
  All of Jack Welch’s management tactics should be considered suspect now. His performance at GE was 100% fueled by financial leveraging that blew up in 2009, basically killing the company. Nobody should be taking management lessons from this guy.

nonameiguess 2y ago

It's worth hammering on this point as much as possible hoping a few people listen, but there is at least one other important point about employee performance. If you're allocating bonuses, a single year's performance is probably a good way to do that, assuming you can accurately measure it. When you're talking retention and promotion, though, you're making a prediction of future performance, possibly at a variety of different jobs. That is even harder to do and more poorly reflected in the last year's results. You have some analogies to sports performance in this article, and you see this kind of thing all the time there. Guy does great in a single year, gets a huge, possibly long-term contract, then tanks. On the other hand, one of the better dynasties of the past decade was accomplished by the Golden State Warriors in the US NBA thanks to underpaying one of the all-time great players in NBA history because he suffered a series of ankle injuries early in his career and scared off other suitors. Single-year performance isn't necessarily reflective of a person's true mean abilities, and their place in the Pareto distribution won't be the same at all levels of advancement and responsiblity, either. The problem, from a company's perspective, is you probably need to retain everyone at least five years, and actually give them a wide variety of assignments in that time, to really get any usable data about their long-term prospects.

stego-tech 2y ago
Literally this. I’ve been banging on about this my entire career, not that corporate leaders tend to listen to the riff-raff. Especially in tech companies, they tend to only evaluate promotions and raises based on the past half-year of work, rather than a repeated pattern of successes across a diverse array of tasks and backgrounds over a significant period of time (years); even then, you only get the promotion if you’re on the right team, doing the right work, at the right time, and for the right leader. This leads to otherwise stellar performers going elsewhere, because the janitors, maintainers, and firefighters in an organization never get properly rewarded, respected, or recognized by leaders. Said leaders pass this off as “bad performers”, failing to realize the importance of superb talent working on less-than-stellar projects that keep the company running efficiently. The only people who benefit from performance reviews are shareholders whose price pops when layoffs happen, and those who game the system for their own political ends. Top talent never really thrives in these, because they’re too busy doing actually meaningful and important work.

throwaway48476 2y ago

Setting aside the issue of defining a function for 'employee performance', this glosses over the invisible interactions. An employee in a dysfunctional organization will perform worse than if they were in a well functioning one because they don't have to waste time dealing with people and processes that are a hindrance.

dogleash 2y ago

To me the biggest insight here is that no matter what data science you're trying to do on a group of employees, the people you already have decided should be fired or promoted from that group are outliers and should be removed from the sample. There are certainly times that you would want them included, but those can be classified under "budgeting," not gaining insight on a workforce.

ses1984 2y ago
Doesn’t the inclusion or exclusion of these people heavily depend on what type of insight you’re trying to get out of the data?

bparsons 2y ago

Unless you are measuring the output of people on simple assembly lines, it is very difficult to define "performance". In a properly functioning team, people perform different, discrete roles which are probably not entirely understood by other team members or management.

iambateman 2y ago

As employees, our expectations for performance management come from the system of giving grades in school. What's interesting is that school grades often doesn't follow a normal distribution, especially for easier classes. I suspect that getting an "A" was possible for 95%+ of students in my gym class and only 5-10% of the students in my organic chemistry class. In the same way, some jobs are much easier to do well than others. So we should expect that virtually all administrative positions will have "exceptional" performance, which is to say that they were successful at doing all of the tasks they were asked to do. But for people who's responsibility-set is more consequential, even slightly-above average performance could be 10x more meaningful to the company.

nightski 2y ago
Having a shifted mean doesn't mean they aren't a normal distribution. Not saying they are necessarily, but the anecdote you are providing isn't convincing.
- dowager_dan99 2y ago
  if your scale doesn't have the atomic values at the top end to differentiate the data it's not a normal, it's Pareto or Zipf or some other power law.
- kurthr 2y ago
  Perhaps, but due to the sampling of the distribution you would likely never know. If 95% of your samples fit in the top 3 bins, you can’t say much at all with certainty. Poisson, Gaussian, binomial, Boltzmann, gamma…
  - deleted 2y ago
    [deleted]
- marian_ivanco 2y ago
  That is not IMHO what he is trying to say, you don't shift the distribution, you measure if somebody passed a test. I the test is "passable" then one side of "distribution" is at least cut off. E.g. it's normal (and sometimes expected) that the whole class will pass without issues.

hemloc_io 2y ago

Cool data/idea, and anecdotally lines up with my experiance at BigCos from a coworker perspective. But in my experiance employee perf evals are more political than data based. At the end of the day a lot of mgmt at BigCo, esp these days, wants that 10% quota for firing as a weapon/soft layoff and the "data" is a fig leaf to make that happen. More generously it's considered a forcing function for managers to actually find underperformers in their orgs, even if they don't exist. Either way it's not really based on anything other than their own confirmation bias. IME the scrutiny of perf evaluation is basically tied to the trajectory of the company and labor market conditions. Even companies with harder perf expectations during the good times of ~2021 relaxed their requirements.

jampa 2y ago

Going through some performance reviews as a manager, I always try to push back a bit against the bell curve. It kinda reminds me of the "stack ranking.". There are also some factors to be considered: If you are in a hiring freeze or not promoting, most of the curve should shift right, assuming you are hiring great people. They will probably perform better quarter after quarter. Some might counter-argue that if everyone performs better, this should be the "new expectation," but I disagree: the market sets expectations. If you have someone at a senior level with expectations of staff, for example, they won't be in the company for long. I hired many great engineers who later said they only looked for a new job because they were never promoted despite being overperformers.

bhouston 2y ago

I would feel better if this was derived from empirical data rather than just rhetoric. This seems super testable, no? There is probably a ton of data already in different industries with regards to productivity. Even if human talent have a Pareto distribution (which is not clear), the people employed by a company are a selected sub-set of that population, which would likely have a different distribution depending on how they are selected and the task at hand. I think that any of these simplified distributions are likely not generalizable across companies and industries (e.g. productivity of AWS or Google employees are likely not distributed like employees of MacDonalds or Wal*Mart because of the difference in hiring procedures and the nature of the tasks.) Get hard data within the companies and industry you are in and then you can make some arguments. Otherwise, I feel it is too easy to just be talking up a sand castle that has no solid footing.

drcwpl 2y ago
Agree with you - although, rhetorically speaking, I have come across many instances which the author refers to "of low performers are 3x as common as high performers." This is unfortunate as I always think do your best, and as Tyler Cowen states - Average is Over. So agree it would have been way better to use empirical data to back up this claim especially.
KK7NIL 2y ago
The problem is that intellectual productivity is generally not possible to measure directly, so you instead end up with indirect measurements that assume a Gaussian distribution. IQ is famously Gaussian distributed... mainly because it's defined that way, not because human "intelligence" (good luck defining that) is Gaussian. If you look at board game Elo ratings (poor test for intelligence but we'll ignore that), they do not follow a Gaussian distribution, even though Elo assumes a Gaussian distribution for game outcomes (but not the population). So that's good evidence that aptitude/skill in intellectual subjects isn't Gaussian (but it's also not Pareto iirc).
- bhouston 2y ago
  > so you instead end up with indirect measurements that assume a Gaussian distribution. 100%. I was going to write something similar. > If you look at board game Elo ratings (poor test for intelligence but we'll ignore that), they do not follow a Gaussian distribution, even though Elo assumes a Gaussian distribution for game outcomes (but not the population). So that's good evidence that aptitude/skill in intellectual subjects isn't Gaussian (but it's also not Pareto iirc). Interesting, yeah, Elo is quite interesting. And one can view hiring in a company as something like selecting people for Elo above a certain score, but with some type of error distribution on top of that, probably Gaussian error. So what does a one sided Elo distribution look like with gaussian error in picking people above that Elo limit?

doctorpangloss 2y ago

This article: "Wouldn't it be cool if when you measure employee performance, it turned out to fit a Pareto distribution better than a Gaussian?" Would that be cool? We could posit the implications of all sorts of improbabilities. But I feel more strongly about how cool it would be that P = NP. All this aside, being laid off sucks - being pushed out, even when you're a high performer, sucks even more. The truth is that "data science" does not help you process grief the way reading Dostoevsky does, so maybe getting an A in your liberal arts education is valuable even when you are working as a software developer.

morkalork 2y ago

If you ever look at tranditional human-driven sales data, you'll often see a small percentage of top performers absolutely dominating the total sales volume. So yes, employee performance is not Gaussian at all.

Joel_Mckay 2y ago

"Hey wait - is [arbitrary metrics] Gaussian distributed?" =3

riazrizvi 2y ago
This whole thread smh. It feels like a military power convinced it can win a war by flying around in airplanes at 30,000’ and they are here vigorously discussing their insane tactics. It’s time for me to leave Silicon Valley.

wing-_-nuts 2y ago

One reason I'd never work for a company with a 'bottom 10% gets PIP'd' mentality is that it directly conflicts with my goal of self development. Of course I want to be on a great team where everyone performs better than I do. That's how I hone my craft! It just seems really wasteful to have to cull the bottom 10% of every team, even if that team is performing well. I wish there was a list of companies that subscribed to that mentality, so I could avoid them.

dmurray 2y ago

> For what it’s worth, human height is also Gaussian, and that’s correlated with workplace success. Height is generally not considered to be Gaussian and this is exactly the kind of statistics mistake the author seems to be accusing employers of. Adult height is somewhere between Gaussian and bimodal.

timdellinger 2y ago
Fair enough. Perhaps better stated as "adult human height is approximately Gaussian for a given biological sex", with an asterisk that environmental factors stretch the distribution. I love the anecdote that people born in the American colonies came back to England to visit family, and were remarkably taller compared to their cousins due to environmental factors.

warrentr 2y ago

In the work rules book about google, Bock claims (apparently using a lot of real data from google) that employee performance follows a power law distribution.

deleted 2y ago

[deleted]

seiferteric 2y ago

A lot of focus on employee performance, but relatively little on management performance. I always wonder how a once great company can slowly decline into irrelevance. Take yahoo for example, it could only be due to management failure over several decades right? How can companies optimize for management performance?

bornfreddy 2y ago
Firing 10% each year would be a great start in many companies. ;)
- fargle 2y ago
  firing the right 10% each year would be great, but that's the problem. can you expect the management correctly identify the right people to fire, especially when they are themselves heavily over-represented in that class?
  - seiferteric 2y ago
    Maybe employees get a vote on who to fire.
    - fargle 2y ago
      well, that's simply not how any business works, even in the most extreme naively theoretical socialistic case. why would a company do that? - employees don't (generally) own the company. even for some reason they were forced to, in practice, a cadre of employees (most likely the ones you'd want to fire in the first place), would instantly form some new ad-hoc political layer to exercise political power and control the organization, voting, etc. so you just end up with a layer of useless management and a independent layer of useless politicians, neither of which are willing to fire themselves.

AtlasBarfed 2y ago

1) performance reviews are never aligned with employee value, because companies are strongly invested to take excess production from employees and transfer it to management, secondarily shareholders 2) the are also not aligned with the replacement cost of employees because the religion of management is that labor is effortlessly replaceable and low value 3) employee retention is not aligned with corporate performance in Machiavellian middle management, it is aligned with manager promotion for things like loyalty and maintaining fiefdom power, budgetary size, headcount, etc 4) there are no absolute or ever directly derived metrics in software development that have ever worked, to say nothing of other positions Those are off the top of my head.

xmly 2y ago

Well, managers are trying to make it Gaussian, but underlying is actually power law.

crazygringo 2y ago

This is very unconvincing. The author already admits one reason why: > But there are low-performing employees at large corporations; we’ve all seen them. My perspective is that they’re hiring errors. Yes, hiring errors should be addressed, but it’s not clear that there’s an obvious specific percentage of the workforce that is the result of hiring errors. I think it is clear that we expect a certain percentage of hiring "errors". And that they are not binary but rather a continuum. And that there are lots of other factors like employees who were great when they were hired but stopped caring and are "coasting" or just burnt out, who got promoted or transferred when they shouldn't have been and are bad at their new level/role, and so forth. The Pareto distribution isn't particularly relevant here, because a hiring process isn't trying to get a whole slice of the overall labor market with clear cutoffs. For any position, it's trying to maximize the performance it can get at a given salary, and we have no reason to expect the errors it makes in under- and over-estimating performance to be anything but relatively symmetric. So a Gaussian distribution is a far more reasonable assumption than a slice of the Pareto distribution, when you look at the multiplicity of factors involved.

wavemode 2y ago
> So a Gaussian distribution is a far more reasonable assumption than a slice of the Pareto distribution It's not an assumption. See the evidence referenced in the footnotes.
- crazygringo 2y ago
  Incorrect. It is absolutely an assumption. The "evidence" in the footnotes is about national salary data. Not the distribution for any individual position at a company. And it is entirely possible (and probable) that performance at each position is distributed as a Gaussian, and all those Gaussians add up to a Pareto at a population level. But you simply cannot take national-level data and assume it applies at the micro level. That's not how statistics works.
  - wavemode 2y ago
    > And it is entirely possible (and probable) that performance at each position is distributed as a Gaussian Find me any research agreeing with this statement (spoiler - you won't, because researchers have broadly reached consensus to the opposite)

jedberg 2y ago

One of the things I loved about working at Netflix was that the base assumption was that everyone was a top performer. If you weren't a top performer, you were given a severance check. The analogy we used was a sports team. Pro sports teams have really good players and great players. Some people are superstars, but unless you're at least really really good you're not on the team. Performance and compensation were completely separate, which was also nice. Performance evals were 360 peer reviews, and compensation was determined mostly by HR based on what it was costing to bring in new hires, and then bumping everyone up to that level. So at least at Netflix 10 years ago, performance wasn't really distributed at all. Everyone was top 10% industrywide.

brabel 2y ago
It's really difficult for me to believe that they really got 10% top performers. For one, knowing the cut-throat nature of employment there, I would expect only a minority of developers would be willing to try working there, despite the awesome rewards. Another reason I really don't trust that to be true is that I've never seen a good way to measure who is a top performer and who is not. I don't think there's one, people are good in different things, even within the same job... for one assignment, Joe may be the best, but for another, Mary is the winner (but again, to measure this reliably and objectively is nearly impossible IMHO for anything related to knowledge work - and I've read lots of research in this area!). Finally, just as a cheap shot at Netflix, sorry I can't resist as a customer: they absolutely suck at the most basic stuff in their business, which is to produce good content in the first place, and very importantly, NOT FREAKING CANCEL the best content! I won't even mention how horrible their latest big live stream was... oh well, I just did :D.
- kube-system 2y ago
  > the most basic stuff in their business, which is to produce good content in the first place, and very importantly, NOT FREAKING CANCEL the best content! It isn't that simple. Making money from content is not 1-to-1 related with the quality of the content. There are many examples of great content that doesn't make money, and many examples of content that makes a lot of money that isn't great. Also there are many differing opinions on what 'great content' even is.

ianbicking 2y ago

"IQ is Gaussian" – it was pointed out somewhere, and only then became obvious to me, that IQ is not Gaussian. The distribution is manufactured. If you have 1000 possible IQ questions, you can ask a bunch of people those questions, and then pick out 100 questions that form a Gaussian distribution. This is how IQ tests are created. This is not unreasonable... if you picked out 100 super easy questions you wouldn't get much information, everyone would be in the "knows quite a lot" category. But you could try to create a uniform distribution, for instance, and still have a test that is usefully sensitive. But if you worry about the accuracy of the test then a Gaussian distribution is kind of convenient... there's this expectation that 50th percentile is not that different than 55th percentile, and people mostly care about that 5% difference only with 90th vs 95th. (But I don't think people care much about the difference between 10th percentile and 5th... which might imply an actual Pareto distribution, though I think it probably reflects more on societal attention) Anyway, kind of an aside, but also similar to what the article itself is talking about

FredPret 2y ago
This is a subtle aspect of intelligence measurement that not many people think about. To go from an IQ of 100 to 130 might require an increase in brainpower of x, and from 130 to 170 might require 3x for example, and from 170-171 might be 9x compared to 100. We have to have a relative scale and contrive a Gaussian from the scores because we don’t have an absolute measure of intelligence. It would be a monumental achievement if computer science ever advances to the point where we have a mathematical way of determining the minimum absolute intelligence required to solve a given problem.
- logicchains 2y ago
  >It would be a monumental achievement if computer science ever advances to the point where we have a mathematical way of determining the minimum absolute intelligence required to solve a given problem For a huge number of problems (including many on IQ tests) computer science does in fact have a mathematical way of determining the minimum absolute amount of compute necessary to solve the problem. That's what complexity theory is. Then it's just a matter of estimating someone's "compute" from how fast they solve a given class of problems relative to some reference computer.

sangnoir 2y ago

> Performance management, as practiced in many large corporations in 2024, is an outdated technology that is in need of an update Author made a couple of fundamental mistakes: the first is they assume employees are (or should be) paid according to how much they "individually" earned the company. Employers strive to pay employees the minimum they can bear, on employer's terms. Those terms are information asymmetry and a Gaussian distribution. Fairness is the last thing one should expect from employers, but being honest about this is not good for morale, so instead, they rely on keeping employees uninformed, while the employers collude to gather everyone's remuneration history via the Work Number. The second mistake they made is assume that companies would prioritize being lean and trimming the mediocre & bottom 5%. There are other considerations, combined productivity is more important than having individual superstars working on the shiniest features. How much revenue do you think a janitor or café staffer generates? Close to zero. The same goes for engineering. Someone has to do the unglamorous staff, or you end up with a dysfunctional company, with amazing talent (on paper). Edit: there's an infamous graph that shows when aggregate worker productivity and average income. The two tracked closely, rising in tandem until the 1970s, where they got decoupled. With income becoming much flatter, and productivity continuing to rise. That's how the world has been for the past 50 years on the macro and the micro

diggan 2y ago
> the first is they assume employees are (or should be) paid according to how much they earned the company From the perspective of a employee and/or human, that does seem like the most fair way of distributing what the company earns, sans the money that gets reinvested straight back into the business itself. But I'd guess that'd be more of a co-operative, and less like the typical for-profit company most companies are today.
- dingnuts 2y ago
  Did you even finish reading the comment you're replying to? It explicitly explains why employees who do not generate revenue are still valuable. What you're describing, that money would go to whoever brings in revenue directly, is the myopic viewpoint of Sales with an emphasis on closing deals with nothing else. If it wasn't for the rest of the work, there'd be nothing to sell!

TrainedMonkey 2y ago

Employee performance MEASUREMENT appears to be Gaussian distributed. To my first simple, and let's be real probably somewhat wrong, approximation there are roughly 3 things that go into it. 1. There is a certain skill in communicating all the important things you've done, we shall lump likability + politicking into this one for convenience. 2. There is a premium that is placed on shiny new features and saving the day heroics. A lot less priority is placed on refactoring and solving the problems before they require heroics. 3. Finally there are individual's technical and self-management skills. I.E. it's important to work on important things and be good at it.

philipov 2y ago

> How much revenue do you think a janitor or café staffer generates? Close to zero. The same goes for engineering. Someone has to do the unglamorous staff, or you end up with a dysfunctional company, with amazing talent (on paper). If the company would be dysfunctional without that janitor or software engineer, and not bring in as much revenue as a result, it sounds like the model that attributes close to zero revenue to them is already dysfunctional. If the company can't function without the janitor, then a significant portion of the revenue of the company should be attributed to them.

sangnoir 2y ago
Sound like you're expecting employers to strive for fairness. Instead, they are striving for profits for the capital class. The labor class gets the minimum possible amount to reach the shareholders primary goal.
- philipov 2y ago
  It sounds like you're confusing what they do currently and what the system should be set up to encourage instead. That things are broken right now is not a valid argument in favor of the status quo. The point you make only proves why it is so important that unions should have as much economic power as corporations do, so that the buy and sell sides of the labor market have negotiation parity.
  - sangnoir 2y ago
    I'm being descriptive, bot prescriptive: I'm stating what the priorities are under a capitalist system without the rose-colored glasses offered by the Just-world fallacy.
    - philipov 2y ago
      In a well-functioning capitalistic system, the sell side of the labor market has equal power with the buy side. When the buy and sell sides of a market have a huge power imbalance, this leads to market failure, which is contrary to the goals of a capitalist system, as it results in inefficient allocation of capital.

irrational 2y ago

Is it Q4 at a lot of companies? How many companies align their fiscal calendar with the yearly calendar? Our Q4 is March-May.

spyckie2 2y ago

So… 1) treat poor performers as bad hires and ignore them in your dataset 2) treat 10x performers as needing to be promoted and also ignore them in your data 3) treat everyone else as relatively equal …and use “Pareto distribution” and “no one has mentioned this before” to write a blog post? Is the point of the article to get people who disagree with 10% corporate culling a pseudo intellectual economic buzzword argument to stroke their hatred of an inefficient hr practice? If so: 1) 10% culling in performance review is a mechanism to cull “bad hires”. I find it difficult to understand how the author can argue it’s a bad practice and then state that you cull bad hires from your dataset without thinking that they are the same thing or at least largely overlapping. 2) If the author is proposing to separate performance review, culling bad hires, and promotions, into 3 separate systems and assume no overlap, he should think through the structural issues more. While it’s possible to design a management structure where the organization is at a constant state of no bad hires, all 10xers promoted, that is putting a lot of responsibility on individual managers to run review, culling and promotion by themselves at a very high level. It’s brittle - a few bad managers not running the system can easily leave your organization bloated with bad hires and no fallback (fallback = performance review process). 3) The system of performance review is equally about risk management to the business as it is about rewarding your employees. IMO, the author’s framing simplifies the problem too much and pushes the complexity out for other people to deal with. It’s the kind of thinking that is damaging to organizations… I wonder if there is a process to cull this kind of thinking from your org… wait what time of year is it??

29athrowaway 2y ago

I guess developers should have a pay structure similar to sales when you make part of your money from bonuses tied to results. But those results are hard to evaluate because shipping something fast can have bugs found after the reward date.

wavemode 2y ago

This concept is not new - see [0]. There's ample research that Welchian stack ranking, and assuming a Gaussian distribution of employee performance, is not well-founded. Even its original pioneers (General Electric) have abandoned the practice (see [1]). Not sure why there are so many commenters here defending the Gaussian model. Most researchers at this point agree that a pareto distribution is more realistic. [0]: https://hbr.org/2022/01/we-need-to-let-go-of-the-bell-curve https://hbr.org/2022/01/we-need-to-let-go-of-the-bell-curve [1]: https://qz.com/428813/ge-performance-review-strategy-shift https://qz.com/428813/ge-performance-review-strategy-shift

uoaei 2y ago

I suspect you can dig into any metric here and find that they are explicitly determined in terms of an assumption of underlying normality.

_vaporwave_ 2y ago

> a helpful order of magnitude estimate is that the hiring process all told costs the company approximately a year’s salary It feels weird to gloss over this since transaction costs this high have a huge impact on how the system should be designed.

igorkraw 2y ago

The author looks at "observables" of performance without considering whether there might be confounders such as those discussed in great nuance here https://onlinelibrary.wiley.com/doi/full/10.1111/joes.12328 https://onlinelibrary.wiley.com/doi/full/10.1111/joes.12328 . He cites similar work by William Shockley who taught both electrical engineering and scientific racism at Stanford https://en.wikipedia.org/wiki/William_Shockley https://en.wikipedia.org/wiki/William_Shockley (no swipe at the author, just pointing at the biased motiviations of some of the researchers foundational to the idea of "high performers"). In general, when you see pareto structures or power laws, you should think of compound or cascade effects, which in human structures generally means some form of social mediation. Affinity for a desireable skill might be gaussian, but the selection process means that the people who _get_ to do that skill might become pareto shaped because if you aren't much better than the next guy, you wouldn't stably stay at the top. Similar logic can hold for other expressions. In general, I wish more people would read https://blackwells.co.uk/bookshop/product/Causality-by-Judea-Pearl/9780521895606 https://blackwells.co.uk/bookshop/product/Causality-by-Judea... or at least the more accessible https://mixtape.scunning.com/ https://mixtape.scunning.com/ before starting to conjecture from data about social systems - the math will tell you what you can and cannot speculate on. (fun exercise: draw the causal models of IQ in https://dagitty.net/ https://dagitty.net/ and ponder the results)

drc500free 2y ago

I've recently been working with a lot of service center productivity data. Staff productivity (customers/hour) is pretty close to a gaussian, with some skew towards many slight underperformers and few overperformers. However, any single customer interaction is exponential or weibull distributed.

PaulHoule 2y ago

It depends on the job. If you are interested in the American caste system you should read this classic https://www.amazon.com/Remember-me-God-Myron-Kaufmann/dp/B00005XSLQ https://www.amazon.com/Remember-me-God-Myron-Kaufmann/dp/B00... Which tells the story of a Jewish person who fails to persevere against prejudice in a multifaceted and sensitive way. In one scene he gets a job as a bank teller and then realizes in some jobs you’ve got the potential to screw up but no potential to distinguish yourself. The world needs people to milk cows every morning, a job you can screw up but not do it 10x better than competent, there is no Pareto or other “exceptional events” distributions for many essential jobs. ER doctors, taxicab drivers, astronauts, etc. (Productivity is a product of the system + the people) I worked on one system that had a 40 minute build if you wanted it to be reliable, the people I picked it up from could not build it reliably which is why the project has been going in circles for 1.5 years before I showed up. With no assistance (and orders that I was not supposed to spend time speeding up my build because it didn’t directly help the customer) I got it to a 20 minute build. Other folks on the team thought I was a real dope because my build took too long and I was always complaining but they couldn’t build it reliably at all.. I mas two major releases of a product with revolutionary performance in one year at which point I felt that I’d done the honorable thing and that I’d feel less backlash anywhere else whether or not I was creating more value —- so I moved on, and was told by recruiters that they hadn’t found a replacement for me in six months. Had the place I was working at had a 2 minute build they might never had hired me because they would have had the product ready long before.

hammock 2y ago

Why would performance be pareto distributed? Not saying it isn't, just wish we would unpack that idea a bit more. IQ and other personality traits are gaussian, with which I would expect performance to be correlated But, the mythical "10X employee" would seem to imply pareto, along with 80/20 notions of both personnel and an individual employee's day-to-day workload How do we resolve this dichotomy?

estebarb 2y ago

Some years ago I started doing graphs of code contributions across the year (yeah, wrong thing to measure, I know). A funny thing is that people considered "high performers" could be made the worse performers depending on how you cut the data. Basically, performance had a wave behavior, and nobody was at 100% all the time. That is a good argument for diverse hiring: people will have bad days/seasons, fact of life. If the team is diverse is less probable that those bad days will correlate between different employees.

mdnahas 2y ago
I did that once and one contributor had a huge negative value. His team had reimplemented a feature and his major contribution was deleting the old version.

thesz 2y ago

> IQ is Gaussian. The Big Five Personality Trait known as Conscientiousness is likewise Gaussian. For what it’s worth, human height is also Gaussian... Height cannot be negative, thus, it is not Gaussian. IQ cannot be negative too. Great many things that most people think are Gaussians, are not. One of such distributions that describe one-sided values, log-normal distribution (logarithms of values are distributed normally) has interesting property that for some d values x=mean+d are more probable than values x=mean-d (heavy tail). Also, sum of log-normal-distributed values does not converge to Gaussian distribution.

hinkley 2y ago
> IQ cannot be negative Have you been keeping up with current events?

hinkley 2y ago

Doesn’t quite work with Heart Shaped Box, but ok.

soniman 2y ago

It's the marginal dollar that contributes profit so the marginal employees are actually the most profitable.

directevolve 2y ago

I had assumed stack ranking was specifically designed to force managers to fire low performers, without relying on their individual judgment. Since nobody likes to fire, this overcomes the inertia, and since relying on personal judgment exposed you to legal risk and principle agent problems, a simple rule was substituted. The author’s proposal to go back to managerial discretion would of course be incompatible with that intention. I do wonder whether those implementing stack ranking are really that committed to a particular statistical model of employee productivity, or if they’re trying to solve a human and legal problem with an algorithm.

cynicalpeace 2y ago

If you take this as true- what does it imply?

losthalo 2y ago

X + Y + [XY] = 8 X = the individual's contribution Y = the contribution of the system they work within [XY] = the interaction of the individual with the system 8 represents some measure of productivity, e.g., rate of errors, millions of dollars in profit, whatever you're measuring The person who can solve for X is competent to rate people on their performance. What to do instead of (destructively) rating people? Build better systems for doing the work, make their work easier, give them psychological safety and job security so they can relax and enjoy their work and share better methods with each other. (All paraphrased from W. Edwards Deming.) Competition within organizations is for amateurs.

Ginger-Pickles 2y ago
CTRL+F “Deming” - Thank you. Organizations with vitality reason about the operation of the whole system, not simply the performance of actors in isolation.

graycat 2y ago

> Hey, wait – is employee performance Gaussian distributed? Well, the Gaussian distribution gives positive probability to any interval of the real line, including the whole real line (probability 1), so, strictly speaking, no. But maybe the issue is a distribution with a bell curve or even with just a unique maximum and falling off monotonically from that maximum. Well, then, in my college teaching, still no: Instead, commonly, roughly, there were three kinds of students: (1) understood the material at least reasonably well, (2) understood some of the material a little, and (3) should have just dropped the course but from me got by with a gentleman C. So, the distribution had a peak for each of (1) -- (3), three peaks, no Gaussian! Approximate Gaussian is guaranteed, under meager assumptions, from the central limit theorem (CLT) of averaging random variables, the easiest case, independent, identically distributed (IID), and, more depending on how advanced the CLT proof is. A proof due to Lindeberg-Feller long was, maybe still is, regarded as the most powerful CLT. Apparently ~100 years ago, especially in education, the CLT was commonly regarded as standard, true, without question, maybe some law of nature. Maybe some of the people measuring IQ, SAT scores, etc. also thought this about the Gaussian. For me, I, in mathematical and applied probability, care first about finite expectation, conditional independence, independence, several convergence results (e.g., the martingale convergence theorem), then IID, and hardly at all, Gaussian.

psychoslave 2y ago

The main take away, to my mind is "are we measuring the right things?" Like, is the system helping to maximize happiness distribution within humanity while maintaining biodiversity in its highest concomitant expectable dynamics?

pajko 2y ago

The problem is that the performance cannot be measured really, do there's nothing to talk about. KPIs like LOC/month or resolved tickets/month does not tell anything. Nor does completed projects/year, nor does the project's size. Name a single thing which does not depend on external stuff. Have fun with an idiotic client, or an idiotic management. If you want to remove the people who do not work, then do that. If you do not know who they are, fire your managers. Or better, fire yourself too, as it seems like you dont' know what you are doing. If you want to lower costs, fire managers and sales people first.

mdnahas 2y ago

I’m an economist who has looked into something like this. This paper is about intra-company performance. I’ve look at incomes of the whole population. There, you find two distribution. Wages seem to follow a log-normal distribution, while income from investments follows a heavy-tailed distribution, like a Pareto. Wages tend to be smaller than asset income. Top sports players and musicians work for wages and become billionaires. Startup founders, who own assets, become trillionaires. Obviously, there are differences. Wages are not productivity. (But the article didn’t say how productivity was measured.). Also, a company can choose who joins and leaves it. So one company’s wage distribution doesn’t have to follow the distribution of the wider economy.