5 ms·

In addition to this, metrics do not need targets. repeating for emphasis: metrics do not need targets. You absolutely need measurements and data, and to highl

In addition to this, metrics do not need targets. repeating for emphasis: metrics do not need targets. You absolutely need measurements and data, and to highlight trends.. but target metrics are almost always bullshit. It reminds me of a psychology study done on kids that I wish I had the source for. IIRC when asked to perform a task for no reward (like, how many simple math problems can you solve in a minute?), many performed it well. then they gave the good performers gold stars. Immediately, the kids not getting stars started performing worse than originally. Then they took away the gold stars. Now the top performers started doing worse as well. People just want to be seen and heard. Measure the things that are important WITHOUT deciding in advance what is considered a "good job". this way you can do your 1:1 on the whole picture. "you're slow, but your quality is exceptional" The only exception to this is if the work is simple enough that you can measure it directly against your $$$, and you have a target that ensures you are green. Like maybe a strictly data entry position for a very consistent source of data. In this case, there arent 1:1's to be expected with your employees.. they either hit target or get fired.

hef19898 4y ago
True, if a metric has a target the metric is the target. If there ever was an incentive to game a KPI, that is it. SPC is tricky case in that regard. Because it, kind of, makes the trend and long term development of metric the target. Edit: You had me thinking. Settng a target, or rather target tolerances, for a metric can be necessary if that metric is directly measuring the output of one process that is used by another process. E.g. forecast accuracy, if downstream ops are expected to cope with, say, a deviation of 20 %, that tolerance kind of becomes the goal for the forecast accuracy. This is not some arbritary goal so. For me, setting limits on certain metrics is something different from a target. It is a thin line between those.
- cupofpython 4y ago
  > if downstream ops are expected to cope with, say, a deviation of 20 %, that tolerance kind of becomes the goal for the forecast accuracy. I think a forecasting target of 20% shouldnt be necessary unless you want to be around 19% deviation. Which might be the case if trying to reach 0% has diminishing returns and isnt actually cost effective. In other words, targets make sense when you are literally targeting a sweet-spot. If 0% deviation is desired, then the directive is still the same - try to be as accurate as possible. If "as accurate as possible" is consistently falling below 20% (which is the importance of measuring and data) then you need to address that. Maybe forecasting is broken or infeasible. On ops side, they might not be expected to handle deviations over 20%, but the directive is probably the same either way "do the best you can to manage the deviation". >For me, setting limits on certain metrics is something different from a target. It is a thin line between those. I agree with this in general. It can be tricky to communicate "here is a boundary that absolutely cannot be crossed, but keep in mind you should be nowhere near that boundary anyway" without it impacting peoples personal optimization strategies. You really have to weigh the consequences of what actually happens when that limit is reached, and if it's worth a possible depression of performance from the team.
  - hef19898 4y ago
    >> I agree with this in general. It can be tricky to communicate "here is a boundary that absolutely cannot be crossed, but keep in mind you should be nowhere near that boundary anyway" without it impacting peoples personal optimization strategies. You really have to weigh the consequences of what actually happens when that limit is reached, and if it's worth a possible depression of performance from the team. It all comes down to leadership, doesn't it?
    - cupofpython 4y ago
      Oh definitely.. but a good leader could still fall into some traps of thinking they can offload some of their good leadership decisions into targets. and then employees feel judged rather than seen
owlbite 4y ago
It's worse than that even. As soon as a metric becomes a goal it distorts the entire system. People optimize for the metric discarding anything else. Metric is closed tickets? OK, open more smaller tickets, close aggressively and tell customer to open a new ticket if problem continues. You just have to look at public education being measured by exam scores to see the effects of this writ large.
jerf 4y ago
I hypothesize without much direct experience that the executives of an organization should come to see excessively optimized metrics as a red flag. If what you are measuring has been optimized to the n'th degree that is sufficient proof that something you are not measuring is suffering. There is always something important you are not measuring. It may be something measurable you aren't measuring today, or it may be an unmeasurable, but there's something. Creating a new metric and seeing it go up for a year or two is good. Creating a new metric and seeing it go up and up and up past that ought to be considered suspicious. Not automatically "wrong", but something to dig into, rather than celebrate on cognitive autopilot.
- compiler-guy 4y ago
  There are also diminishing marginal returns of optimizing for a certain metric. The first big steps are relatively easy, and produce a good return on the investment, but as you get deeper into the optimization, the costs of getting that next little bit go up. And people are tempted to cheat, and take short cuts to meet the goal. For example, getting from one nine of uptime to two nines isn't all that hard or expensive (a month of downtime per year to half a week). Getting from three nines to four nines is a lot harder, and a lot more expensive, and if you tie success at the organization to that improvement, people will fudge to reach it. Nearly every metric is like that. Amount of sales, tail latency, recruiting efficacy.
netjiro 4y ago
> study done on kids that I wish I had the source for. There are a lot of them with similar results, going back to 1960s. Kohn "Punished by Rewards" is a good general start for that stuff. I think there was some generalised study/review published in late 90s. Probably Deci and Ryan (?).
- kqr 4y ago
  Also Pink's Drive has a lot more on the research around motivation. Highly recommended for anyone working with people, or having children.

hef19898 4y ago

True, if a metric has a target the metric is the target. If there ever was an incentive to game a KPI, that is it. SPC is tricky case in that regard. Because it, kind of, makes the trend and long term development of metric the target. Edit: You had me thinking. Settng a target, or rather target tolerances, for a metric can be necessary if that metric is directly measuring the output of one process that is used by another process. E.g. forecast accuracy, if downstream ops are expected to cope with, say, a deviation of 20 %, that tolerance kind of becomes the goal for the forecast accuracy. This is not some arbritary goal so. For me, setting limits on certain metrics is something different from a target. It is a thin line between those.

cupofpython 4y ago
> if downstream ops are expected to cope with, say, a deviation of 20 %, that tolerance kind of becomes the goal for the forecast accuracy. I think a forecasting target of 20% shouldnt be necessary unless you want to be around 19% deviation. Which might be the case if trying to reach 0% has diminishing returns and isnt actually cost effective. In other words, targets make sense when you are literally targeting a sweet-spot. If 0% deviation is desired, then the directive is still the same - try to be as accurate as possible. If "as accurate as possible" is consistently falling below 20% (which is the importance of measuring and data) then you need to address that. Maybe forecasting is broken or infeasible. On ops side, they might not be expected to handle deviations over 20%, but the directive is probably the same either way "do the best you can to manage the deviation". >For me, setting limits on certain metrics is something different from a target. It is a thin line between those. I agree with this in general. It can be tricky to communicate "here is a boundary that absolutely cannot be crossed, but keep in mind you should be nowhere near that boundary anyway" without it impacting peoples personal optimization strategies. You really have to weigh the consequences of what actually happens when that limit is reached, and if it's worth a possible depression of performance from the team.
- hef19898 4y ago
  >> I agree with this in general. It can be tricky to communicate "here is a boundary that absolutely cannot be crossed, but keep in mind you should be nowhere near that boundary anyway" without it impacting peoples personal optimization strategies. You really have to weigh the consequences of what actually happens when that limit is reached, and if it's worth a possible depression of performance from the team. It all comes down to leadership, doesn't it?
  - cupofpython 4y ago
    Oh definitely.. but a good leader could still fall into some traps of thinking they can offload some of their good leadership decisions into targets. and then employees feel judged rather than seen

owlbite 4y ago

It's worse than that even. As soon as a metric becomes a goal it distorts the entire system. People optimize for the metric discarding anything else. Metric is closed tickets? OK, open more smaller tickets, close aggressively and tell customer to open a new ticket if problem continues. You just have to look at public education being measured by exam scores to see the effects of this writ large.

jerf 4y ago

I hypothesize without much direct experience that the executives of an organization should come to see excessively optimized metrics as a red flag. If what you are measuring has been optimized to the n'th degree that is sufficient proof that something you are not measuring is suffering. There is always something important you are not measuring. It may be something measurable you aren't measuring today, or it may be an unmeasurable, but there's something. Creating a new metric and seeing it go up for a year or two is good. Creating a new metric and seeing it go up and up and up past that ought to be considered suspicious. Not automatically "wrong", but something to dig into, rather than celebrate on cognitive autopilot.

compiler-guy 4y ago
There are also diminishing marginal returns of optimizing for a certain metric. The first big steps are relatively easy, and produce a good return on the investment, but as you get deeper into the optimization, the costs of getting that next little bit go up. And people are tempted to cheat, and take short cuts to meet the goal. For example, getting from one nine of uptime to two nines isn't all that hard or expensive (a month of downtime per year to half a week). Getting from three nines to four nines is a lot harder, and a lot more expensive, and if you tie success at the organization to that improvement, people will fudge to reach it. Nearly every metric is like that. Amount of sales, tail latency, recruiting efficacy.

netjiro 4y ago

> study done on kids that I wish I had the source for. There are a lot of them with similar results, going back to 1960s. Kohn "Punished by Rewards" is a good general start for that stuff. I think there was some generalised study/review published in late 90s. Probably Deci and Ryan (?).

kqr 4y ago
Also Pink's Drive has a lot more on the research around motivation. Highly recommended for anyone working with people, or having children.