The current risk of AI is the elimination of any (screen) job that requires a person of average intelligence. Given that LLMs are the sum-total of human output,

The current risk of AI is the elimination of any (screen) job that requires a person of average intelligence. Given that LLMs are the sum-total of human output, it makes sense that they might behave much like an average person. (Better since they do not sleep or get bored; worse because they are not embodied; unclear because they have no emotions or consciousness.) But what we have today will undermine the (screen-based) job opportunities for everyone at or below average intelligence, which is 50% of the human population. This is not the existential risk of super-AGI but it's here now and will hurt a lot of people. This lesser but more real risk is of much higher priority than unbounded, self-improving AGI. The OP's metaphor might be extended to be 1 billion immortal aliens with 100 IQ but who have no sense of self or personal autonomy and willingly work as slaves (and are constantly microdosing LSD).