To clarify, I am trying to say that the only purpose of metrics is for humans in a social context. If an AI optimizing for a metric ends up with unintended cons

To clarify, I am trying to say that the only purpose of metrics is for humans in a social context. If an AI optimizing for a metric ends up with unintended consequences, that's a human problem. A human has to go in and redefine a new metric to fix it. There's no way other way around it. The human can't anthropomorphize and blame the machine. A machine can't redefine itself because it doesn't understand human values. So it's important to understand why humans believe in metrics. Humans by default are multi-objective. We don't care about just one or even a few things, we care about a lot of things, all at once, with nobody agreeing on what they are. Identifying the "true direction" we want to go as a collective is impossible. So we pick a good enough direction so everyone can be somewhat aligned. To make it possible to communicate this vision, we compress the world down to few dimensions even though we know that it is absurd. This works fine for humans because our "distributed system" is composed of beings with enough intelligence to notice/alert/convince the rest of the system whenever we are veering off course. Whenever we program AI with these metrics, we forget that we can't direct them we way we direct our sentient brethrens. Any simple metrics are necessarily doomed to fail because it doesn't have this self aligning property. So ultimately it is the programmer's responsibly to go and realign the system back with human values. If necessary, with some social pressure (hence this article). It doesn't seem like there will be any other way.