7 ms·

> We also don't really understand how consciousness works either. Sort of a nitpick, but people concerned about AI risk usually don't postulate that an AI need

> We also don't really understand how consciousness works either. Sort of a nitpick, but people concerned about AI risk usually don't postulate that an AI needs to be """conscious""" to be dangerous; it just needs to be a thing that can apply a lot of optimization power in unexpected ways toward goals that aren't perfectly aligned with what humans generally want.

circlefavshape 9y ago
Oh! Good insight! ... but we already have that, right? Undesirable outcomes from rulesets that govern the behaviour of a large number of (conscious or unconscious) actors are a pretty standard feature of civilisation
- convexfunction 9y ago
  Yes, we do have that already, and it's kind of odd that people who see no point in thinking hard about risks to humans from this particular type of inhuman optimization process would readily agree that other types of inhuman optimization processes (with much better understood risk profiles and limitations) can be very harmful. :)
  - mercer 9y ago
    Perhaps the 'singularity' type stuff makes a lot of people disengage? So far I'm inclined to think that the whole 'sentient AI mucking things up' is jumping the gun (to put it mildly). I might very well be wrong though. However, it's only because of HN that I even bother to make the distinction between that particular fear (realistic or otherwise) and the much more directly useful discussion of how AI-type stuff (even just facebook feed algorithms) can be dangerous or harmful.
  - circlefavshape 9y ago
    I don't think it's odd. I agree that optimising for the wrong thing (or, probably, for any particular thing) can be harmful, especially when it's done on a very large scale. AI-as-we-know-it doesn't introduce a new class of risk, it just makes some of the optimisations faster. I don't see any point in thinking about AGI risk until we get some experimental evidence that AGI is possible
    - convexfunction 9y ago
      Maybe. I think the idea is that it's preferable to have wasted a lot of effort on coming up with irrelevant abstractions addressing a nonexistent problem, than to be surprised by something weird and poorly-understood presenting novel and not-clearly-bounded risk and not have effective tools at hand to address it. I won't claim to know what the right amount of effort directed toward this is though, nor do I really think AI safety research has a >epsilon chance of accomplishing its goal if it turns out to be relevant at all. ¯\_(ツ)_/¯
- fossuser 9y ago
  It's the paper clip maximizer idea [1]. "The goal of maximizing paperclips is chosen for illustrative purposes because it is very unlikely to be implemented, and has little apparent danger or emotional load (in contrast to, for example, curing cancer or winning wars). This produces a thought experiment which shows the contingency of human values: An extremely powerful optimizer (a highly intelligent agent) could seek goals that are completely alien to ours (orthogonality thesis), and as a side-effect destroy us by consuming resources essential to our survival." [1] https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer
  - shpx 9y ago
    OpenAI did a thing[0] where they used a human to help train a machine learning model by picking the best of 2 models. > Our algorithm’s performance is only as good as the human evaluator’s intuition about what behaviors look correct, so if the human doesn’t have a good grasp of the task they may not offer as much helpful feedback. Relatedly, in some domains our system can result in agents adopting policies that trick the evaluators. For example, a robot which was supposed to grasp items instead positioned its manipulator in between the camera and the object so that it only appeared to be grasping it, as shown below. [0] https://blog.openai.com/deep-reinforcement-learning-from-human-preferences/ https://blog.openai.com/deep-reinforcement-learning-from-hum... https://news.ycombinator.com/item?id=14545298 https://news.ycombinator.com/item?id=14545298 78 upvotes, 7 comments

circlefavshape 9y ago

Oh! Good insight! ... but we already have that, right? Undesirable outcomes from rulesets that govern the behaviour of a large number of (conscious or unconscious) actors are a pretty standard feature of civilisation

convexfunction 9y ago
Yes, we do have that already, and it's kind of odd that people who see no point in thinking hard about risks to humans from this particular type of inhuman optimization process would readily agree that other types of inhuman optimization processes (with much better understood risk profiles and limitations) can be very harmful. :)
- mercer 9y ago
  Perhaps the 'singularity' type stuff makes a lot of people disengage? So far I'm inclined to think that the whole 'sentient AI mucking things up' is jumping the gun (to put it mildly). I might very well be wrong though. However, it's only because of HN that I even bother to make the distinction between that particular fear (realistic or otherwise) and the much more directly useful discussion of how AI-type stuff (even just facebook feed algorithms) can be dangerous or harmful.
- circlefavshape 9y ago
  I don't think it's odd. I agree that optimising for the wrong thing (or, probably, for any particular thing) can be harmful, especially when it's done on a very large scale. AI-as-we-know-it doesn't introduce a new class of risk, it just makes some of the optimisations faster. I don't see any point in thinking about AGI risk until we get some experimental evidence that AGI is possible
  - convexfunction 9y ago
    Maybe. I think the idea is that it's preferable to have wasted a lot of effort on coming up with irrelevant abstractions addressing a nonexistent problem, than to be surprised by something weird and poorly-understood presenting novel and not-clearly-bounded risk and not have effective tools at hand to address it. I won't claim to know what the right amount of effort directed toward this is though, nor do I really think AI safety research has a >epsilon chance of accomplishing its goal if it turns out to be relevant at all. ¯\_(ツ)_/¯
fossuser 9y ago
It's the paper clip maximizer idea [1]. "The goal of maximizing paperclips is chosen for illustrative purposes because it is very unlikely to be implemented, and has little apparent danger or emotional load (in contrast to, for example, curing cancer or winning wars). This produces a thought experiment which shows the contingency of human values: An extremely powerful optimizer (a highly intelligent agent) could seek goals that are completely alien to ours (orthogonality thesis), and as a side-effect destroy us by consuming resources essential to our survival." [1] https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer
- shpx 9y ago
  OpenAI did a thing[0] where they used a human to help train a machine learning model by picking the best of 2 models. > Our algorithm’s performance is only as good as the human evaluator’s intuition about what behaviors look correct, so if the human doesn’t have a good grasp of the task they may not offer as much helpful feedback. Relatedly, in some domains our system can result in agents adopting policies that trick the evaluators. For example, a robot which was supposed to grasp items instead positioned its manipulator in between the camera and the object so that it only appeared to be grasping it, as shown below. [0] https://blog.openai.com/deep-reinforcement-learning-from-human-preferences/ https://blog.openai.com/deep-reinforcement-learning-from-hum... https://news.ycombinator.com/item?id=14545298 https://news.ycombinator.com/item?id=14545298 78 upvotes, 7 comments