12 ms·

Interesting thoughts. I was having a very similar chat with a friend about this recently, including exactly the question of "what should the reward function be"

Interesting thoughts. I was having a very similar chat with a friend about this recently, including exactly the question of "what should the reward function be", or at least the most minimal one. Some aspects we thought of: - pain (minimal reward): this should probably be hardwired straight to the artificial brain, though it can't be enough as otherwise no activity would take place. The agent would learn that the best course of action is to sit still - so we also came up with curiosity being a necessity. Encountering an unseen or hard to predict state leads to positive reward - although I am not sure whether pain is not sufficient by itself. E.g. in nature, actions are still necessary. Otherwise, other pain signals (hunger and thirst) start showing up. - what is tricky to figure out is how this works for more complicated intelligence such as humans. Let's be honest, most babies are fed whenever necessary by their caretakers. What causes them to learn? What causes grown ups to learn? So something important that we'll have to figure out is what needs to be hardwired versus what emerges from 2nd order things such as chemistry, hormons, gut bacteria, upbringing, role of parents etc., and whether there's a difference between non-conscious or simple intelligence versus complex ones w.r.t. the necessity of these aspects. E.g. you might talk about aspects such as "love" (towards your partner, children, born or not, future generations) but it is much more unclear how necessary this is, and how to quantify it. Perhaps indeed this emerges from the basic reward function but only after a meta-simulation: > I don’t think there’s a way to learn it aside from millennia of multi agent survival competition.

PartiallyTyped 4y ago
Curiosity is not a necessity. While curiosity is integral part of what makes a human well, human, it doesn't have to be hardwired. In us it is hardwired onto dopamine circuits (cf. Molecule of more, great read). However, I'd argue that in us, it is simply a form of inductive bias, i.e. an existing part of our hardware that makes it cheap. The keyword here is cheap. Any sufficiently powerful maximizer with an infinite horizon __has__ to develop curiosity otherwise they will not be able to maximise their reward function. In-fact, I'd argue that this is true for most mammalian functions such as taking care of our pack, exhibiting pro-social behaviour and so on, but there is a caveat. For this to happen, there needs to be an actual benefit in the behaviour. Due to evolution operating mostly linearly with small changes through genetic and epigenetic information passing, there seems to be relatively little variation between generations, which then implies that it is difficult for some candidates to overwhelm everyone else in a winner takes all fashion, hence maximization of replication will eventually result in cooperation simply because that allows genes in support of it to continue replicating, effectively self-selecting for itself. We saw this in the OpenAI video where agents eventually learned to cooperate in what was effectively a prisoners dilemma. In the video, there were two teams, the hiders and the seekers, in an environment that could be manipulated. Eventually the two teams learned strategies. From the perspective of seekers, their utility function involved observing the hiders. For the hiders, their utility function was to minimize their exposure to the seekers. For all intents and purposes, and for each team, the other agents were part of the environment. So given two hiders, one could hide behind the other to minimize their exposure to the seekers. This is effectively a defect. Eventually however, the hiders learn to cooperate and instead cooperatively manipulate the environment through strategies. --- Sorry if I got a little bit off topic there. Regardless, what causes us to learn is neurotransmitters getting released because certain circuits activate, the neurotransmitters charge a neuron which causes it to fire. Connections between neurons get reinforced if they are frequently used, which reinforces them, makes them cheaper, and that inevitably reinforces certain patterns of behaviour. --- What i propose is that we should instead look into analogies as a means of learning. Humans seem to be great at using analogies. Mathematically speaking, an analogy is a functor between categories. A category is a collection of objects and morphisms (directed relationships) between the objects, this is as abstract and simple as it gets. A functor between categories essentially maps the objects and the morphisms of one category to the respective of the other. When we use an analogy, we do the same. > A is to B as X is to Y This then allows us to learn the morphism in the category with {A,B} using just known relationships. I think I got off topic again, but this is something that I have been recycling in my head for a while and needed to eventually get out.
- mistermann 4y ago
  Could you possibly post a link to the video you refer to above?
  - PartiallyTyped 4y ago
    Here [1]. They called it "emergent" property, but I only see that as inevitable behaviour. [1] https://www.youtube.com/watch?v=kopoLzvh5jY https://www.youtube.com/watch?v=kopoLzvh5jY
- Macuyiko 4y ago
  > Sorry if I got a little bit off topic there. Not at all. Though forgive me for not agreeing with some key points you raise. > Curiosity is not a necessity. While curiosity is integral part of what makes a human well, human, it doesn't have to be hardwired. > The keyword here is cheap. Any sufficiently powerful maximizer with an infinite horizon __has__ to develop curiosity otherwise they will not be able to maximise their reward function. With that I do in fact agree. I think curiosity was a quick solution to fix some immediate problems I was seeing from the pain-slash-survival angle, perhaps from a belief there should be more to humanity. I also feel it is rather emergent and should (must!) emerge pretty fast, even, in order to survive. Actually typing out the last sentence made me realize another meta-meta-level of intelligence. Whereas the basic reward function is level 0, the chemistry surrounding and interactions with out body it might be level 1 (might be, because they are probably emergent as well), evolution is definitely level 2 (or 1) - the multi-simulations of agents. On top of that, there's the fact that initialisation is cheap: meaning that even if some emergent properties are highly necessary on top of the basic reward function, and might lead to very complex aspects later on, a designer (and this is a very badly chosen word mayhaps) would be prepared to deal with those given the fact that there are many chances. Many one-cell organism striving to do better in the "soup". The more I think about this, the more I start becoming convinced that computational biology should have been a serious field (and many are saying this). > In-fact, I'd argue that this is true for most mammalian functions such as taking care of our pack, exhibiting pro-social behaviour and so on, but there is a caveat. For this to happen, there needs to be an actual benefit in the behaviour. See, this is where I respectfully disagree. The benefit can be emerging from a longer-term simulation rather than immediately. You might say: sure but what are the chances of this happening? Well how many intelligent species like ours have we encountered so far? On this planet, in this universe? > Due to evolution operating mostly linearly with small changes through genetic and epigenetic information passing, there seems to be relatively little variation between generations, which then implies that it is difficult for some candidates to overwhelm everyone else in a winner takes all fashion, hence maximization of replication will eventually result in cooperation simply because that allows genes in support of it to continue replicating, effectively self-selecting for itself. Yes and no. From a gut feeling I agree though I also think small changes tend to take over very rapidly in a population pool once they show up. The waiting is mainly for the showing up part. > We saw this in the OpenAI video where agents eventually learned to cooperate in what was effectively a prisoners dilemma. In the video, there were two teams, the hiders and the seekers, in an environment that could be manipulated. I saw that as well, and this is why although I believe at some point this will be possible, the main reason why I agree with hotz is because our simulations suck and always allow for exploitation. Unless it doesn't, of course. The on-device part, hence, for me, is not that necessary. But it means we should have a very robust simulation (which so far we don't have in any area of RL and associated topics; digital twins are a joke; people making them care more about dataviz; and so on). > Regardless, what causes us to learn is neurotransmitters getting released because certain circuits activate, the neurotransmitters charge a neuron which causes it to fire. Connections between neurons get reinforced if they are frequently used, which reinforces them, makes them cheaper, and that inevitably reinforces certain patterns of behaviour. Too tired to go into this but you touch upon some key differences between gradient descent and biological neuron learning, though we are getting closer to that: spiking neurons, memory cells, even real-neuron cell chips. I am not sure electronics wouldn't be able to emulate it correctly in the end. If after all, P <> NP, then what does the "real" difference of a computational time step make? > What i propose is that we should instead look into analogies as a means of learning. Humans seem to be great at using analogies. Mathematically speaking, an analogy is a functor between categories. Agree, and surprising that this is still such an open question in AI even given the one-shot and zero-shot learning research. Though it seems like this has been put on the backburner yet again. It amazes me even today how young humans are so good at that. Like someone said: show a cartoon tomato and a real tomato to a toddler. Next time show a cartoon of an elephant and wait until they see a real elephant. They will shout: elephant. Though on the other hand, the solution for this might be very close to use. A small architectural or multi-modal change. I was more pessimistic about this a few years ago, but less sure today. I think the main piece missing of the puzzle is stepping away from supervised learning, self-supervised learning, and going for a continuous self-supervised reinforcement learning, where predictions for t+1 are continuously matched with reality, like a human brain does. The only problem is that you need to have a continuous reality. But we have that.
simon_000666 4y ago
I’ve been thinking deeply about this over the last couple of weeks. So sure, the obvious answer is the overriding human reward function is survival, propagation of our dna. Except I think it’s more complex than that, take War for example where people sacrifice themselves for an idea of nationality or culture, that goes against the ‘selfish gene’ theory. I think the answer is there are multiple reward functions that compete for dominance. Perhaps each of the different ‘brains’ has it’s own reward function.
- aaaaaaaaaaab 4y ago
  >take War for example where people sacrifice themselves for an idea of nationality or culture, that goes against the ‘selfish gene’ theory People sacrifice themselves in wars, because they think it's going to improve the chances of survival for their offspring, their extended family, their tribe, their kin, their nation, etc.
  - Macuyiko 4y ago
    Exactly. This is the crux of the question: what is necessary as a starting point. What is not. And given what is: what emerges over time (like nationalism).
- jlpom 4y ago
  > that goes against the ‘selfish gene’ theory On the contrary, it confirms it, it means some individuals for the overall replication success of the genes they are constituted of, will sacrifice themselves. In the book from which this theory have been popularised, the exemple of bees stinging like kamikazes is given.
a-dub 4y ago
> I don’t think there’s a way to learn it aside from millennia of multi agent survival competition. i think that millienia of multi agent survival competition in a sufficiently high fidelity world will you get there, and certainly all our best evidence indicates that's where we come from. but i really don't think at all that precludes any drastically effective shortcuts being taken by beings whom are capable of introspection to guide an engineering effort.
wavesquid 4y ago
I think you could get pretty far with avoiding pain and avoiding boredom. The avoiding boredom can be seen in humans studies like https://www.science.org/doi/10.1126/science.1250830 https://www.science.org/doi/10.1126/science.1250830

PartiallyTyped 4y ago

Curiosity is not a necessity. While curiosity is integral part of what makes a human well, human, it doesn't have to be hardwired. In us it is hardwired onto dopamine circuits (cf. Molecule of more, great read). However, I'd argue that in us, it is simply a form of inductive bias, i.e. an existing part of our hardware that makes it cheap. The keyword here is cheap. Any sufficiently powerful maximizer with an infinite horizon __has__ to develop curiosity otherwise they will not be able to maximise their reward function. In-fact, I'd argue that this is true for most mammalian functions such as taking care of our pack, exhibiting pro-social behaviour and so on, but there is a caveat. For this to happen, there needs to be an actual benefit in the behaviour. Due to evolution operating mostly linearly with small changes through genetic and epigenetic information passing, there seems to be relatively little variation between generations, which then implies that it is difficult for some candidates to overwhelm everyone else in a winner takes all fashion, hence maximization of replication will eventually result in cooperation simply because that allows genes in support of it to continue replicating, effectively self-selecting for itself. We saw this in the OpenAI video where agents eventually learned to cooperate in what was effectively a prisoners dilemma. In the video, there were two teams, the hiders and the seekers, in an environment that could be manipulated. Eventually the two teams learned strategies. From the perspective of seekers, their utility function involved observing the hiders. For the hiders, their utility function was to minimize their exposure to the seekers. For all intents and purposes, and for each team, the other agents were part of the environment. So given two hiders, one could hide behind the other to minimize their exposure to the seekers. This is effectively a defect. Eventually however, the hiders learn to cooperate and instead cooperatively manipulate the environment through strategies. --- Sorry if I got a little bit off topic there. Regardless, what causes us to learn is neurotransmitters getting released because certain circuits activate, the neurotransmitters charge a neuron which causes it to fire. Connections between neurons get reinforced if they are frequently used, which reinforces them, makes them cheaper, and that inevitably reinforces certain patterns of behaviour. --- What i propose is that we should instead look into analogies as a means of learning. Humans seem to be great at using analogies. Mathematically speaking, an analogy is a functor between categories. A category is a collection of objects and morphisms (directed relationships) between the objects, this is as abstract and simple as it gets. A functor between categories essentially maps the objects and the morphisms of one category to the respective of the other. When we use an analogy, we do the same. > A is to B as X is to Y This then allows us to learn the morphism in the category with {A,B} using just known relationships. I think I got off topic again, but this is something that I have been recycling in my head for a while and needed to eventually get out.

mistermann 4y ago
Could you possibly post a link to the video you refer to above?
- PartiallyTyped 4y ago
  Here [1]. They called it "emergent" property, but I only see that as inevitable behaviour. [1] https://www.youtube.com/watch?v=kopoLzvh5jY https://www.youtube.com/watch?v=kopoLzvh5jY
Macuyiko 4y ago
> Sorry if I got a little bit off topic there. Not at all. Though forgive me for not agreeing with some key points you raise. > Curiosity is not a necessity. While curiosity is integral part of what makes a human well, human, it doesn't have to be hardwired. > The keyword here is cheap. Any sufficiently powerful maximizer with an infinite horizon __has__ to develop curiosity otherwise they will not be able to maximise their reward function. With that I do in fact agree. I think curiosity was a quick solution to fix some immediate problems I was seeing from the pain-slash-survival angle, perhaps from a belief there should be more to humanity. I also feel it is rather emergent and should (must!) emerge pretty fast, even, in order to survive. Actually typing out the last sentence made me realize another meta-meta-level of intelligence. Whereas the basic reward function is level 0, the chemistry surrounding and interactions with out body it might be level 1 (might be, because they are probably emergent as well), evolution is definitely level 2 (or 1) - the multi-simulations of agents. On top of that, there's the fact that initialisation is cheap: meaning that even if some emergent properties are highly necessary on top of the basic reward function, and might lead to very complex aspects later on, a designer (and this is a very badly chosen word mayhaps) would be prepared to deal with those given the fact that there are many chances. Many one-cell organism striving to do better in the "soup". The more I think about this, the more I start becoming convinced that computational biology should have been a serious field (and many are saying this). > In-fact, I'd argue that this is true for most mammalian functions such as taking care of our pack, exhibiting pro-social behaviour and so on, but there is a caveat. For this to happen, there needs to be an actual benefit in the behaviour. See, this is where I respectfully disagree. The benefit can be emerging from a longer-term simulation rather than immediately. You might say: sure but what are the chances of this happening? Well how many intelligent species like ours have we encountered so far? On this planet, in this universe? > Due to evolution operating mostly linearly with small changes through genetic and epigenetic information passing, there seems to be relatively little variation between generations, which then implies that it is difficult for some candidates to overwhelm everyone else in a winner takes all fashion, hence maximization of replication will eventually result in cooperation simply because that allows genes in support of it to continue replicating, effectively self-selecting for itself. Yes and no. From a gut feeling I agree though I also think small changes tend to take over very rapidly in a population pool once they show up. The waiting is mainly for the showing up part. > We saw this in the OpenAI video where agents eventually learned to cooperate in what was effectively a prisoners dilemma. In the video, there were two teams, the hiders and the seekers, in an environment that could be manipulated. I saw that as well, and this is why although I believe at some point this will be possible, the main reason why I agree with hotz is because our simulations suck and always allow for exploitation. Unless it doesn't, of course. The on-device part, hence, for me, is not that necessary. But it means we should have a very robust simulation (which so far we don't have in any area of RL and associated topics; digital twins are a joke; people making them care more about dataviz; and so on). > Regardless, what causes us to learn is neurotransmitters getting released because certain circuits activate, the neurotransmitters charge a neuron which causes it to fire. Connections between neurons get reinforced if they are frequently used, which reinforces them, makes them cheaper, and that inevitably reinforces certain patterns of behaviour. Too tired to go into this but you touch upon some key differences between gradient descent and biological neuron learning, though we are getting closer to that: spiking neurons, memory cells, even real-neuron cell chips. I am not sure electronics wouldn't be able to emulate it correctly in the end. If after all, P <> NP, then what does the "real" difference of a computational time step make? > What i propose is that we should instead look into analogies as a means of learning. Humans seem to be great at using analogies. Mathematically speaking, an analogy is a functor between categories. Agree, and surprising that this is still such an open question in AI even given the one-shot and zero-shot learning research. Though it seems like this has been put on the backburner yet again. It amazes me even today how young humans are so good at that. Like someone said: show a cartoon tomato and a real tomato to a toddler. Next time show a cartoon of an elephant and wait until they see a real elephant. They will shout: elephant. Though on the other hand, the solution for this might be very close to use. A small architectural or multi-modal change. I was more pessimistic about this a few years ago, but less sure today. I think the main piece missing of the puzzle is stepping away from supervised learning, self-supervised learning, and going for a continuous self-supervised reinforcement learning, where predictions for t+1 are continuously matched with reality, like a human brain does. The only problem is that you need to have a continuous reality. But we have that.

simon_000666 4y ago

I’ve been thinking deeply about this over the last couple of weeks. So sure, the obvious answer is the overriding human reward function is survival, propagation of our dna. Except I think it’s more complex than that, take War for example where people sacrifice themselves for an idea of nationality or culture, that goes against the ‘selfish gene’ theory. I think the answer is there are multiple reward functions that compete for dominance. Perhaps each of the different ‘brains’ has it’s own reward function.

aaaaaaaaaaab 4y ago
>take War for example where people sacrifice themselves for an idea of nationality or culture, that goes against the ‘selfish gene’ theory People sacrifice themselves in wars, because they think it's going to improve the chances of survival for their offspring, their extended family, their tribe, their kin, their nation, etc.
- Macuyiko 4y ago
  Exactly. This is the crux of the question: what is necessary as a starting point. What is not. And given what is: what emerges over time (like nationalism).
jlpom 4y ago
> that goes against the ‘selfish gene’ theory On the contrary, it confirms it, it means some individuals for the overall replication success of the genes they are constituted of, will sacrifice themselves. In the book from which this theory have been popularised, the exemple of bees stinging like kamikazes is given.

a-dub 4y ago

> I don’t think there’s a way to learn it aside from millennia of multi agent survival competition. i think that millienia of multi agent survival competition in a sufficiently high fidelity world will you get there, and certainly all our best evidence indicates that's where we come from. but i really don't think at all that precludes any drastically effective shortcuts being taken by beings whom are capable of introspection to guide an engineering effort.

wavesquid 4y ago

I think you could get pretty far with avoiding pain and avoiding boredom. The avoiding boredom can be seen in humans studies like https://www.science.org/doi/10.1126/science.1250830 https://www.science.org/doi/10.1126/science.1250830