The AI is modelled as an agent with goals. If you want to argue that "intelligence" means "has a terminal goal exactly matching the values of humanity," then I

The AI is modelled as an agent with goals. If you want to argue that "intelligence" means "has a terminal goal exactly matching the values of humanity," then I suppose a paper-clip maximizer is not intelligent, but that's a non-standard definition of intelligence. The more common definitions for intelligence go something like this: The capacity of an agent to select and implement actions that bring about its goals. Here the "goals" could be anything. That is, any goal could be combined with any level of capability so long as the agent is intelligent enough to comprehend its goal. >If you have to specify the goals very carefully then the AI it's not super intelligent by definition. When you're designing the agent, you have to select something like a goal, whether it's a fitness function or the maximization of a reward signal. Evolution certainly selected our goals. Love, self-expression, sex, fun, intellectual stimulation, the beauty of social interaction – these all reflect features of our ancestral environment. It’s a little naive to think there is a One True Morality that all agents stumble upon. It’s even sillier to think it happens to reflect our values The intelligent part of the paperclip maximizer selects actions and weights them in the context of its utility function, to maximize paperclips. Any thought of changing its goals would be judged of negative utility, as that would lead to less paperclips.