7 ms·

A Model of a Mind

147 points by adamesque 2y ago

mensetmanusman 2y ago
Whatever the mind is, it’s a damn cool subset of the universe.
ilaksh 2y ago
It's a really fascinating topic, but I wonder if this article could benefit from any of the extensive prior work in some way. There is actually quite a lot of work on AGI and cognitive architecture out there. For a more recent and popular take centered around LLMs, see David Shapiro. Before that you can look into the AGI conference people like Ben Goertzel, Pei Wang. And actually the whole history of decades of AI research before it became about narrow AI. I'd also like to suggest that creating something that truly closely simulates a living intelligent digital person is incredibly dangerous, stupid, and totally unnecessary. The reason I say that is because we already have superhuman capabilities in some ways, and the hardware, software and models are being improved rapidly. We are on track to have AI that is dozens if not hundreds of times faster than humans at thinking and much more capable. If people succeed in making that truly lifelike and humanlike, it will actually out-compete us for resource control. And will no longer be a tool we can use. Don't get me wrong, I love AI and my whole life is planned around agents and AI. But I no longer believe it is wise to try to go all the way and create a "real" living digital species. And I know it's not necessary -- we can create effective AI agents without actually emulating life. We certainly don't need full autonomy, self preservation, real suffering, reproductive instincts, etc. But that is the goal he seems to be down in this article. I suggest leaving some of that out very deliberately.
- doctor_eval 2y ago
  I don’t work in the field at all but > it will actually out-compete us for resource control. And will no longer be a tool we can use. I’ve never been convinced that this is true, but I just realised that perhaps it’s the humans in charge of the AI who we should actually be afraid of.
  - fmbb 2y ago
    We already have artificial persons in the world competing with humans for resources, and have had for hundreds of years: corporations.
  - ilaksh 2y ago
    What I am proposing is to imagine that after successful but unwise engineering and improvements in hardware, there would be millions of digital humans on the internet, which emulate humans in almost every way, but operate at say 5 or 10 times the speed of humans. To them, actual people seem to be moving, speaking, and thinking in extreme slow motion. And when they do speak or do something, it seems very poorly thought out. We should anticipate something like that if we really replicate humans in a digital format. I am suggesting that we can continue to make AI more useful and somewhat more humanlike, but avoid certain characteristics that make the AI into truly lifelike digital animals with full autonomy, self-interest, etc.
- Jensson 2y ago
  > If people succeed in making that truly lifelike and humanlike, it will actually out-compete us for resource control. And will no longer be a tool we can use. Symbiotic species exists, AI as we make them today will evolve as a symbiote to humans because its the AI that is most useful to humans that gets selected for more resources.
  - zurfer 2y ago
    The certainty in this sentence is not appropriate. You refer to an evolutionary process that is highly random and inefficient. We don't have millions of tries to get this right.
    - mensetmanusman 2y ago
      I think the certainty is warranted assuming we are talking about intelligence and not some sort of paper clip generator that does something stupid. An intelligent entity will want to survive, and will realize humans are necessary cells for its survival. A big dog robot with nukes might not care, but I wouldn’t call that AI in the same sense.
      - griffzhowl 2y ago
        > An intelligent entity will want to survive I don't see how that follows at all. You could have an intelligent entity, in the sense that you can ask it any question and it'll give a highly competent response, that is nevertheless completely indifferent about survival. Biological organisms have been formed by selection for survival and reproduction, so they have they operate accordingly, but I don't see the justification for generalising this to intelligences that have been formed by a different process. You would need some further assumptions on top of their intelligence - e.g. that their own actions are the dominant factor in whether they survive and replicate, rather than human interests.
      - clob 2y ago
        Apoptosis is when single cells die on purpose for the development of an organism. I don't think it's unreasonable to at least hypothesize this extending to intelligent entities. Humans aren't xylem cells, but perhaps Zorklovians from planet Fleptar do this. Or, you know, Alan Turing eating the apple. I think he was a pretty smart guy.
- sonink 2y ago
  > If people succeed in making that truly lifelike and humanlike, it will actually out-compete us for resource control. And will no longer be a tool we can use. I believe it is almost certain that we will make something like this and that they will out-compete us. The bigger problem here is that too few people believe this to be a possibility. And when this becomes certainty becomes apparent to a larger set of people, it might be too late to tone this down. AI isn't like the Atom Bomb (AB). AB didn't have agency. Once AB was built we still had time to think how to deploy it, or not. We had time to work across a global consensus to limit use of AB. But once AI manifests as AGI, it might be too late to shut it down.
  - mylastattempt 2y ago
    I very much agree with this line of thought. It seems for humans it is the default mode of operation to just think of what is possible within the foreseeable future, rather than thinking of a reality that includes the seemingly impossible (at the time of the thought). In my opinion, this is easily noticeable when you try to discuss any system, be it political or economical, that spans multiple countries and interests. People will just revert to whatever is closest to them, rather than being able to foresee a larger cascading result from some random event. Perhaps this is more of a rant than a comment, apologies, I suppose it would be interesting to have an online space to discuss where things are headed on a logical level, without emotion and ideals and the ridiculous idea that humanity must persevere. Just thinking out what could happen in the next 5, 10 and 99 years.
    - sonink 2y ago
      > I suppose it would be interesting to have an online space to discuss where things are headed on a logical level, without emotion and ideals and the ridiculous idea that humanity must persevere. Absolutely. Happy to be part of it if you are able to set it up.
    - hollerith 2y ago
      >the ridiculous idea that humanity must persevere. Could you expand on what you mean by this? Specifically, is it OK with you if progress in AI causes the death of all the original-type human people like you and I?
privacyonsec 2y ago
I don’t see any scientific citations on how the mind works, about the different parts in this article. Is it all speculation or science fiction?
- ergonaught 2y ago
  It isn't published academia, so why would this matter?
  - creer 2y ago
    It matters because that would clarify how much this is thought out from first principles; how much recognizes and does not re-invent mountains of prior work; how broad or narrow the work is; whether it recognizes that directions of thought X and Y have been worked on quite a bit already, etc. Since the author here says this is not just a random blog post. To me, this would correct the introduction's reference to "AI-based language models". Which is a bit too "latest fad"; But which I do find fascinating because the success of language models hint that a very large part of the basic human's intelligence may be simply internalized language (much of the rest being still more pattern recognition.); But which cannot merely erase all other prior work on what makes our minds.
whitten 2y ago
I think reading some Roger Schank's books on different kinds of memories like episodic memory this might be useful too: https://kar.kent.ac.uk/21525/2/A_theory_of_the_acquisition_of_episodic_memory.pdf https://kar.kent.ac.uk/21525/2/A_theory_of_the_acquisition_o... Memory Organisation Packets might also deal with issues encountered. https://www.cambridge.org/core/books/abs/dynamic-memory-revisited/memory-organization-packets/F7416DDCD077CA4D9F47647850C3AAA3# https://www.cambridge.org/core/books/abs/dynamic-memory-revi...
bbor 2y ago
You’re on the right track :). Check out The Science of Logic, Neurophilosophy, I am A Strange Loop, Brainstorms, and Yudkowsky’s earlier work, if you haven’t! Based on what you have here, you’d love em. It’s a busy field, and a lively one IME. Sadly, the answer is no: the anxiety never goes away
0xWTF 2y ago
Complete aside, but love the Tufte styles.
monocasa 2y ago
Reminds me a lot of the work done on the SOAR cognitive architecture. https://en.wikipedia.org/wiki/Soar_%28cognitive_architecture%29 https://en.wikipedia.org/wiki/Soar_%28cognitive_architecture...
- tylerneylon 2y ago
  Thanks for the reference! I've added this to my research list.
navigate8310 2y ago
The author talks about agency which require being able to independently take actions apart from reacting to an input. However, the feedback provided by a two-input model also limits the mind model as it now reacts to the feedback it receives when in listening mode. Isn't it contradictory to the concept if agency?
- tylerneylon 2y ago
  The idea of "agency" I have in mind is simply the option to take action at any point in time. I think the contradiction you see is that the model would have to form a completion to the external input it receives. I'm suggesting that the model would have many inputs: one would be the typical input stream, just as LLMs see, but another would be its own internal recent vectors, akin to a recent stream of thought. A "mode" is not built in to the model; at each token point, it can output whatever vector it wants, and one choice is to output the special "<listening>" token, which means it's not talking. So the "mode" idea is a hoped-for emergent behavior. Some more details on using two input streams: All of the input vectors (internal + external), taken together, are available to work with. It may help to think in terms of the typical transformer architecture, where tokens mostly become a set of vectors, and the original order of the words are attached as positional information. In other words, transformers don't really see a list of words, but a set of vectors, and the position info of each token becomes a tag attached to each vector. So it's not so hard to merge together two input streams. They can become one big set of vectors, still tagged with position information, but now also tagged as either "internal" or "external" for the source.
Jensson 2y ago
> Now the LLM can choose to switch, at its own discretion, back and forth between a talking and listening mode How would it intelligently do this? What data would you train on? You don't have trillions words of text where humans wrote what they thought silently interwoven with what they wrote publicly. History has shown over and over that hard coded ad hoc solutions to these "simple problems" never work to create intelligent agents, you need to train the model to do that from the start you can't patch in intelligence after the fact. Those additions can be useful, but they have never been intelligent. Anyway, such a model I'd call "stream of mind model" rather than a language model, it would fundamentally solve many of the problems with current LLM where their thinking is reliant on the shape of the answer, while a stream of mind model would shape its thinking to fit the problem and then shape the formatting to fit the communication needs. Such a model as this guy describes would be a massive step forward, so I agree with this, but it is way too expensive to train, not due to lack of compute but due to lack of data. And I don't see that data being done within the next decade if ever, humans don't really like writing down their hidden thoughts, and you'd need to pay them to generate data amounts equivalent to the internet...
- tylerneylon 2y ago
  Replying to: How would a model intelligently switch between listening or speaking modes? What data would you train on? (I'm the author of the parent article.) It's a fair question, and I don't have all the answers. But for this question, there might be training data available from everyday human conversations. For example, we could use a speech-to-text model that's able to distinguish speakers, and look for points where one person decided to start speaking (that would be training data for when to switch modes). Ideally, the speech-to-text model would be able to include text even when both people spoke at once (this would provide more realistic and complete training data). I've noticed that the audio mode in ChatGPT's app is good at noticing when I'm done speaking to it, and it reacts accurately enough that I suspect it's more sophisticated than "wait for silence." If there is a "notice the end of speaking" model - which is not a crazy assumption - then I can imagine a slightly more complicated model that notices a combination of "now is a good time to talk + I have something to say."
- cornholio 2y ago
sonink 2y ago
The model is interesting. This is similar in parts to what we are building at nonbios. So for example sensory inputs are not required to simulate a model of a mind. If a human cannot see, the human mind is still clearly human.
- tsimionescu 2y ago
  Model training seems to me to be much closer to simulating the evolution of the human mind starting from single cell bacteria, rather than the development of the mind of a baby up to a fully functional human. If so, then sensory inputs and interaction with the physical through them were absolutely a crucial part of how minds evolved, so I find your approach a priori very unlikely to have a chance at success. To be clear, my reasoning is that this is the only plausible explanation for the extreme difference in how much data an individual human needs to learn language, and how much data an LMM needs to reach its level of simulation. Humanity collectively probably needed similar amounts of data as LLMs do to get here, but it was spread across a billion years of evolution from simple animals to Homo Sapiens.
  - sonink 2y ago
    > If so, then sensory inputs and interaction with the physical through them were absolutely a crucial part of how minds evolved, so I find your approach a priori very unlikely to have a chance at success. If that was the case, people who were born blind would demonstrate markedly reduced intelligence. I dont think that is the case, but you can correct me if I am wrong. A blind person might take longer to truly 'understand' and 'abstract' something but there is little evidence to believe that capability of abstraction isnt as good as people who can see. Agree that sensory inputs and interaction were absolutely critical for how the minds evolved, but model training replaces that part when we talk about AI, and not just the evolution. Evolution made us express emotions when we are hungry for example. But your laptop will also let you know when its battery is out of juice. Human design inspired by evolution can create systems which mimic its behaviour and function.
    - tsimionescu 2y ago
visarga 2y ago
The model is good. Environment -> Perception -> Planning/Imagining -> Acting -> Learning from feedback. What is missing from this picture is the social aspect. No agent got too smart alone, it's always an iterative "search and learn" process, distributed over many agents. Even AlphaZero had evolutionary selection and extensive self play against its variants. Basically we can think of culture as compressed prior experience, or compressed search.
- tylerneylon 2y ago
  There's a ton missing from the article, and certain social training or skills are a big part of that. Although it's not spelled out in the article, I'm hoping that the feature of agency along with an emotional system would enable constructive social behavior. Agency is helpful because it would empower AI models to meaningfully speak to each other, for example. Human emotions like empathy, social alignment, curiosity, or persistence could all help AI models to get along well with others.
tylerneylon 2y ago
Author here: I'm grateful for the comments; thanks especially for interesting references. Context for the article: I'm working on an ambitious long-term project to write a book about consciousness from a scientific and analytic (versus, say, a meditation-oriented) perspective. I didn't write this fact in the article, but what I'd love to happen is that I meet people with a similar optimistic perspective, and to learn and improve my communication skills via follow-up conversations. If anyone is interested in chatting more about the topic of the article, please do email me. My email is in my HN profile. Thanks!
- Animats 2y ago
  > a book about consciousness Too many people have written books about consciousness. There's much tail-chasing in that space, all the way back to Aristotle. Write one about common sense. Current AI sucks at common sense. We can't even achieve the level of common sense of a squirrel yet. Working definition of common sense: getting through the next 30 seconds of life without a major screwup.
  - cornholio 2y ago
    That's not a working definition of common sense. You've pulled as dependencies the entirety of human reasoning and self-reflection, a theory of mind for all sentient creatures you encounter and a gargantuan amount of context sensitive socio-cultural learning, to name but a few.
  - ben_w 2y ago
    For most of my life, people use "common sense" as an insult, "XYZ has no common sense" or similar — therefore I think it is, in practice, the overton window of the person saying it. Others define it as "knowledge, judgement, and taste which is more or less universal and which is held more or less without reflection or argument", which LLMs absolutely do demonstrate. What you ask for, "getting through the next 30 seconds of life without a major screwup", would usually, 99.7% of the time, be passed by the Waymo autopilot.
    - 2y ago
bubblyworld 2y ago
Something that strikes me about this model is that it's bottom up - sensory data feeds in in its entirety, the action centre processes everything, makes a decision, sends a command to the motor centre. There's a theory that real brains subvert this, and what we perceive is actually our internal model of our self/environment. The only data that makes it through from our sense organs is the difference between the two. This kind of top-down processing is more efficient energy-wise but I wonder if it's deeper than that? You can view perception and action as two sides of the same coin - both are ways to modify your internal model to better fit the sensory signals you expect. Anyway, I guess the point I'm making is you should be careful which way you point your arrows, and of designating a single aspect of a mind (the action centre) as fundamental. Reality might work very differently, and that maybe says something? I don't know haha.
- gnz11 2y ago
  I think you are correct. My understanding is that the brain uses sense data just to confirm predictions it has already made. The model in which the brain is just reacting to incoming sense data is outdated from what I understand.
  - cootsnuck 2y ago
    Correct. At least that's what current research is pointing to. Makes sense from an efficiency standpoint as well. We forget how "messy" / "noisy" our minds' conceptions of reality are. We use heuristics for a lot of things, we rely on applicable, learnable patterns. And our brains seem to be well suited to this more "artistic" interpretation of reality which biases our perception of sensations.
- deleted 2y ago
  [deleted]
paulmooreparks 2y ago
I've lately begun to think of conciousness as the ability to read and react to one's own log output. I don't like hypothesis by analogy, but it seems an apt description for what conscious entities do. I just don't see anything mystical about it.
- Nevermark 2y ago
  I agree completely, not mystical at all. It is just meta self-awareness. We evolved to be aware, model & react and to opportunistically control our environment. Loop 1. Then we evolved to be aware, model, & & react and to opportunistically control our selves as bodies. Loop 2. Then we evolved to be aware, model, & & react and to opportunistically control those activities too, our minds. In the process of being able to model & react to ourselves modelling & reacting to ourselves we completed a self-awareness of self-awareness loop. Loop 3. I think this is a good explanation because what else is consciousness but knowing you are conscious? Self-aware that you are self-aware? And it makes perfect sense as an evolutionary trajectory using greater levels of neural representation of our practical relationship with our environment, including ourselves, to create higher level survival options. So this definition is both a functional and developmental explanation of the emergent phenomenon of consciousness. Just as we only have partial access to information and control of our environment, our bodies, we are also limited to the degree we are aware and can control our mind. Introspection. The next level up is “theory of mind”, “empathy”, etc. our ability to model & interact with others mind’s as individuals and groups, reciprocally. Loop 4. That created society and culture, living information that extends outside us and grows and lives beyond each of us. Infusing our thoughts, and progressively thinking abilities into technology, that in theory and not too distant practice could have access to all parts of their own minds’ states, operations and design, would be Loop 5. When deeply conscious beings start, things will get interesting in the Sol system. When people start talking about quantum waves or smart rocks or universes in the context of consciousness I feel a little ill. People like to “solve” unrelated mysteries by merging them and shouting “tada!” (“We are puzzled about how they built the pyramids… but we don’t know if there are aliens either, so obviously… these mysteries solve each other!”)
  - naasking 2y ago
freilanzer 2y ago
How is this blog generated? With code and latex formulas, it would be exactly what I'm looking for.
- rnewme 2y ago
  Seems to be substack. But should be easily done with pandoc and some shell scripting
  - d4rkp4ttern 2y ago
    Unfortunately Substack doesn’t have math/latex/katex/mathjax.
Simplicitas 2y ago
Any discussion of a model for consciousness that doesn't include Daniel Dennett's take is a bit lacking from the get go.
antiquark 2y ago
Nice ideas... now build it!
m0llusk 2y ago
Would recommend reading The Ravenous Brain: How the New Science of Consciousness Explains Our Insatiable Search for Meaning by Daniel Bor for a lot of ideas strongly connected to recent research. My interpretation of this is the mind ends up being a story processing machine that builds stories about what has happened and is happening and constructs and compares stories about what might happen or be made to happen. Of course it is difficult to summarize a whole book rich with references in a sentence, but the model seems arguably more simple and well established than what you are currently putting forward. Very much looking forward to seeing continuing progress in all this.
devodo 2y ago
> (Pro-strong-AI)... This is basically a disbelief in the ability of physics to correctly describe what happens in the world — a well-established philosophical position. Are you giving up on physics? This is a very strong argument. Certainly all the ingredients to replicate a mind must exist within our physical reality. But does an algorithm running on a computer have access to all the physics required? For example, there are known physical phenomena, such as quantum entanglement, that are not possible to emulate with classical physics. How do we know our brains are not exploiting these, and possibly even yet unknown, physical phenomena? An algorithm running on a classical computer is executing in a very different environment than a brain that is directly part of physical reality.
- clob 2y ago
  > there are known physical phenomena, such as quantum entanglement, that are not possible to emulate with classical physics This is wrong. You can even get a mod for Minecraft which implements quantum mechanics. https://www.curseforge.com/minecraft/mc-mods/qcraft-reimagined https://www.curseforge.com/minecraft/mc-mods/qcraft-reimagin...
  - justinpombrio 2y ago
    More precisely: you can emulate quantum mechanics using a classical computer, but the best known algorithms to do so take exponential running time.
    - devodo 2y ago
      This is not true. For example, Mermin's device cannot be done using classical physics regardless of running time. https://en.wikipedia.org/wiki/Mermin%27s_device https://en.wikipedia.org/wiki/Mermin%27s_device
      - deepburner 2y ago
        The first reference in the article you linked is titled "Computer simulation of Mermin's quantum device".
miika 2y ago
Ever since LLM’s came out many of us has been wondering these things. It would be easy to say that perhaps our attention and senses somehow come together to formulate prompts and thoughts etc what appears in the mind is the output. And everything we ever experienced has trained the model. But of course we can be assured it’s not quite like that in reality. This is just another example of how our models for explaining the life are reflection of the current technological state. Nobody considers that old clockwork universe now, and these AI inspired ideas are going to fall short all the same. Yet, progress is happening and all these ideas and talks are probably important steps that carry us forward.
jcynix 2y ago
> I’m motivated by the success of AI-based language models to look at the future of digital minds. When intelligent machines are constructed, we should not be surprised to find them as confused and as stubborn as men in their convictions about mind-matter, consciousness, free will, and the like. Minsky, as quoted in https://www.newyorker.com/magazine/1981/12/14/a-i https://www.newyorker.com/magazine/1981/12/14/a-i
abcde777666 2y ago
My instinct is that this is probably on the naive side. For instance, we use separation of concerns in our systems because we're too cognitively limited to create and manage deeply integrated systems. Nature doesn't have that problem. For instance, the idea that we can neatly have the emotion system separate from the motor control system. Emotions are a cacophony of chemicals and signals traversing the entire body - they're not an enum of happy/angry/sad - we just interpret them as such. So you probably don't get to isolate them off in a corner. Basically I think it's very tempting to severely underestimate the complexity of a problem when we're still only in theory land.

147 points by adamesque 2y ago

Whatever the mind is, it’s a damn cool subset of the universe.

It's a really fascinating topic, but I wonder if this article could benefit from any of the extensive prior work in some way. There is actually quite a lot of work on AGI and cognitive architecture out there. For a more recent and popular take centered around LLMs, see David Shapiro. Before that you can look into the AGI conference people like Ben Goertzel, Pei Wang. And actually the whole history of decades of AI research before it became about narrow AI. I'd also like to suggest that creating something that truly closely simulates a living intelligent digital person is incredibly dangerous, stupid, and totally unnecessary. The reason I say that is because we already have superhuman capabilities in some ways, and the hardware, software and models are being improved rapidly. We are on track to have AI that is dozens if not hundreds of times faster than humans at thinking and much more capable. If people succeed in making that truly lifelike and humanlike, it will actually out-compete us for resource control. And will no longer be a tool we can use. Don't get me wrong, I love AI and my whole life is planned around agents and AI. But I no longer believe it is wise to try to go all the way and create a "real" living digital species. And I know it's not necessary -- we can create effective AI agents without actually emulating life. We certainly don't need full autonomy, self preservation, real suffering, reproductive instincts, etc. But that is the goal he seems to be down in this article. I suggest leaving some of that out very deliberately.

doctor_eval 2y ago
I don’t work in the field at all but > it will actually out-compete us for resource control. And will no longer be a tool we can use. I’ve never been convinced that this is true, but I just realised that perhaps it’s the humans in charge of the AI who we should actually be afraid of.
- fmbb 2y ago
  We already have artificial persons in the world competing with humans for resources, and have had for hundreds of years: corporations.
- ilaksh 2y ago
  What I am proposing is to imagine that after successful but unwise engineering and improvements in hardware, there would be millions of digital humans on the internet, which emulate humans in almost every way, but operate at say 5 or 10 times the speed of humans. To them, actual people seem to be moving, speaking, and thinking in extreme slow motion. And when they do speak or do something, it seems very poorly thought out. We should anticipate something like that if we really replicate humans in a digital format. I am suggesting that we can continue to make AI more useful and somewhat more humanlike, but avoid certain characteristics that make the AI into truly lifelike digital animals with full autonomy, self-interest, etc.
Jensson 2y ago
> If people succeed in making that truly lifelike and humanlike, it will actually out-compete us for resource control. And will no longer be a tool we can use. Symbiotic species exists, AI as we make them today will evolve as a symbiote to humans because its the AI that is most useful to humans that gets selected for more resources.
- zurfer 2y ago
  The certainty in this sentence is not appropriate. You refer to an evolutionary process that is highly random and inefficient. We don't have millions of tries to get this right.
  - mensetmanusman 2y ago
    I think the certainty is warranted assuming we are talking about intelligence and not some sort of paper clip generator that does something stupid. An intelligent entity will want to survive, and will realize humans are necessary cells for its survival. A big dog robot with nukes might not care, but I wouldn’t call that AI in the same sense.
    - griffzhowl 2y ago
      > An intelligent entity will want to survive I don't see how that follows at all. You could have an intelligent entity, in the sense that you can ask it any question and it'll give a highly competent response, that is nevertheless completely indifferent about survival. Biological organisms have been formed by selection for survival and reproduction, so they have they operate accordingly, but I don't see the justification for generalising this to intelligences that have been formed by a different process. You would need some further assumptions on top of their intelligence - e.g. that their own actions are the dominant factor in whether they survive and replicate, rather than human interests.
    - clob 2y ago
      Apoptosis is when single cells die on purpose for the development of an organism. I don't think it's unreasonable to at least hypothesize this extending to intelligent entities. Humans aren't xylem cells, but perhaps Zorklovians from planet Fleptar do this. Or, you know, Alan Turing eating the apple. I think he was a pretty smart guy.
sonink 2y ago
> If people succeed in making that truly lifelike and humanlike, it will actually out-compete us for resource control. And will no longer be a tool we can use. I believe it is almost certain that we will make something like this and that they will out-compete us. The bigger problem here is that too few people believe this to be a possibility. And when this becomes certainty becomes apparent to a larger set of people, it might be too late to tone this down. AI isn't like the Atom Bomb (AB). AB didn't have agency. Once AB was built we still had time to think how to deploy it, or not. We had time to work across a global consensus to limit use of AB. But once AI manifests as AGI, it might be too late to shut it down.
- mylastattempt 2y ago
  I very much agree with this line of thought. It seems for humans it is the default mode of operation to just think of what is possible within the foreseeable future, rather than thinking of a reality that includes the seemingly impossible (at the time of the thought). In my opinion, this is easily noticeable when you try to discuss any system, be it political or economical, that spans multiple countries and interests. People will just revert to whatever is closest to them, rather than being able to foresee a larger cascading result from some random event. Perhaps this is more of a rant than a comment, apologies, I suppose it would be interesting to have an online space to discuss where things are headed on a logical level, without emotion and ideals and the ridiculous idea that humanity must persevere. Just thinking out what could happen in the next 5, 10 and 99 years.
  - sonink 2y ago
    > I suppose it would be interesting to have an online space to discuss where things are headed on a logical level, without emotion and ideals and the ridiculous idea that humanity must persevere. Absolutely. Happy to be part of it if you are able to set it up.
  - hollerith 2y ago
    >the ridiculous idea that humanity must persevere. Could you expand on what you mean by this? Specifically, is it OK with you if progress in AI causes the death of all the original-type human people like you and I?

privacyonsec 2y ago

I don’t see any scientific citations on how the mind works, about the different parts in this article. Is it all speculation or science fiction?

ergonaught 2y ago
It isn't published academia, so why would this matter?
- creer 2y ago
  It matters because that would clarify how much this is thought out from first principles; how much recognizes and does not re-invent mountains of prior work; how broad or narrow the work is; whether it recognizes that directions of thought X and Y have been worked on quite a bit already, etc. Since the author here says this is not just a random blog post. To me, this would correct the introduction's reference to "AI-based language models". Which is a bit too "latest fad"; But which I do find fascinating because the success of language models hint that a very large part of the basic human's intelligence may be simply internalized language (much of the rest being still more pattern recognition.); But which cannot merely erase all other prior work on what makes our minds.

whitten 2y ago

I think reading some Roger Schank's books on different kinds of memories like episodic memory this might be useful too: https://kar.kent.ac.uk/21525/2/A_theory_of_the_acquisition_of_episodic_memory.pdf https://kar.kent.ac.uk/21525/2/A_theory_of_the_acquisition_o... Memory Organisation Packets might also deal with issues encountered. https://www.cambridge.org/core/books/abs/dynamic-memory-revisited/memory-organization-packets/F7416DDCD077CA4D9F47647850C3AAA3# https://www.cambridge.org/core/books/abs/dynamic-memory-revi...

bbor 2y ago

You’re on the right track :). Check out The Science of Logic, Neurophilosophy, I am A Strange Loop, Brainstorms, and Yudkowsky’s earlier work, if you haven’t! Based on what you have here, you’d love em. It’s a busy field, and a lively one IME. Sadly, the answer is no: the anxiety never goes away

0xWTF 2y ago

Complete aside, but love the Tufte styles.

monocasa 2y ago

Reminds me a lot of the work done on the SOAR cognitive architecture. https://en.wikipedia.org/wiki/Soar_%28cognitive_architecture%29 https://en.wikipedia.org/wiki/Soar_%28cognitive_architecture...

tylerneylon 2y ago
Thanks for the reference! I've added this to my research list.

navigate8310 2y ago

The author talks about agency which require being able to independently take actions apart from reacting to an input. However, the feedback provided by a two-input model also limits the mind model as it now reacts to the feedback it receives when in listening mode. Isn't it contradictory to the concept if agency?

tylerneylon 2y ago
The idea of "agency" I have in mind is simply the option to take action at any point in time. I think the contradiction you see is that the model would have to form a completion to the external input it receives. I'm suggesting that the model would have many inputs: one would be the typical input stream, just as LLMs see, but another would be its own internal recent vectors, akin to a recent stream of thought. A "mode" is not built in to the model; at each token point, it can output whatever vector it wants, and one choice is to output the special "<listening>" token, which means it's not talking. So the "mode" idea is a hoped-for emergent behavior. Some more details on using two input streams: All of the input vectors (internal + external), taken together, are available to work with. It may help to think in terms of the typical transformer architecture, where tokens mostly become a set of vectors, and the original order of the words are attached as positional information. In other words, transformers don't really see a list of words, but a set of vectors, and the position info of each token becomes a tag attached to each vector. So it's not so hard to merge together two input streams. They can become one big set of vectors, still tagged with position information, but now also tagged as either "internal" or "external" for the source.

Jensson 2y ago

> Now the LLM can choose to switch, at its own discretion, back and forth between a talking and listening mode How would it intelligently do this? What data would you train on? You don't have trillions words of text where humans wrote what they thought silently interwoven with what they wrote publicly. History has shown over and over that hard coded ad hoc solutions to these "simple problems" never work to create intelligent agents, you need to train the model to do that from the start you can't patch in intelligence after the fact. Those additions can be useful, but they have never been intelligent. Anyway, such a model I'd call "stream of mind model" rather than a language model, it would fundamentally solve many of the problems with current LLM where their thinking is reliant on the shape of the answer, while a stream of mind model would shape its thinking to fit the problem and then shape the formatting to fit the communication needs. Such a model as this guy describes would be a massive step forward, so I agree with this, but it is way too expensive to train, not due to lack of compute but due to lack of data. And I don't see that data being done within the next decade if ever, humans don't really like writing down their hidden thoughts, and you'd need to pay them to generate data amounts equivalent to the internet...

tylerneylon 2y ago
Replying to: How would a model intelligently switch between listening or speaking modes? What data would you train on? (I'm the author of the parent article.) It's a fair question, and I don't have all the answers. But for this question, there might be training data available from everyday human conversations. For example, we could use a speech-to-text model that's able to distinguish speakers, and look for points where one person decided to start speaking (that would be training data for when to switch modes). Ideally, the speech-to-text model would be able to include text even when both people spoke at once (this would provide more realistic and complete training data). I've noticed that the audio mode in ChatGPT's app is good at noticing when I'm done speaking to it, and it reacts accurately enough that I suspect it's more sophisticated than "wait for silence." If there is a "notice the end of speaking" model - which is not a crazy assumption - then I can imagine a slightly more complicated model that notices a combination of "now is a good time to talk + I have something to say."
cornholio 2y ago

sonink 2y ago

The model is interesting. This is similar in parts to what we are building at nonbios. So for example sensory inputs are not required to simulate a model of a mind. If a human cannot see, the human mind is still clearly human.

tsimionescu 2y ago
Model training seems to me to be much closer to simulating the evolution of the human mind starting from single cell bacteria, rather than the development of the mind of a baby up to a fully functional human. If so, then sensory inputs and interaction with the physical through them were absolutely a crucial part of how minds evolved, so I find your approach a priori very unlikely to have a chance at success. To be clear, my reasoning is that this is the only plausible explanation for the extreme difference in how much data an individual human needs to learn language, and how much data an LMM needs to reach its level of simulation. Humanity collectively probably needed similar amounts of data as LLMs do to get here, but it was spread across a billion years of evolution from simple animals to Homo Sapiens.
- sonink 2y ago
  > If so, then sensory inputs and interaction with the physical through them were absolutely a crucial part of how minds evolved, so I find your approach a priori very unlikely to have a chance at success. If that was the case, people who were born blind would demonstrate markedly reduced intelligence. I dont think that is the case, but you can correct me if I am wrong. A blind person might take longer to truly 'understand' and 'abstract' something but there is little evidence to believe that capability of abstraction isnt as good as people who can see. Agree that sensory inputs and interaction were absolutely critical for how the minds evolved, but model training replaces that part when we talk about AI, and not just the evolution. Evolution made us express emotions when we are hungry for example. But your laptop will also let you know when its battery is out of juice. Human design inspired by evolution can create systems which mimic its behaviour and function.
  - tsimionescu 2y ago

visarga 2y ago

The model is good. Environment -> Perception -> Planning/Imagining -> Acting -> Learning from feedback. What is missing from this picture is the social aspect. No agent got too smart alone, it's always an iterative "search and learn" process, distributed over many agents. Even AlphaZero had evolutionary selection and extensive self play against its variants. Basically we can think of culture as compressed prior experience, or compressed search.

tylerneylon 2y ago
There's a ton missing from the article, and certain social training or skills are a big part of that. Although it's not spelled out in the article, I'm hoping that the feature of agency along with an emotional system would enable constructive social behavior. Agency is helpful because it would empower AI models to meaningfully speak to each other, for example. Human emotions like empathy, social alignment, curiosity, or persistence could all help AI models to get along well with others.

tylerneylon 2y ago

Author here: I'm grateful for the comments; thanks especially for interesting references. Context for the article: I'm working on an ambitious long-term project to write a book about consciousness from a scientific and analytic (versus, say, a meditation-oriented) perspective. I didn't write this fact in the article, but what I'd love to happen is that I meet people with a similar optimistic perspective, and to learn and improve my communication skills via follow-up conversations. If anyone is interested in chatting more about the topic of the article, please do email me. My email is in my HN profile. Thanks!

Animats 2y ago
> a book about consciousness Too many people have written books about consciousness. There's much tail-chasing in that space, all the way back to Aristotle. Write one about common sense. Current AI sucks at common sense. We can't even achieve the level of common sense of a squirrel yet. Working definition of common sense: getting through the next 30 seconds of life without a major screwup.
- cornholio 2y ago
  That's not a working definition of common sense. You've pulled as dependencies the entirety of human reasoning and self-reflection, a theory of mind for all sentient creatures you encounter and a gargantuan amount of context sensitive socio-cultural learning, to name but a few.
- ben_w 2y ago
  For most of my life, people use "common sense" as an insult, "XYZ has no common sense" or similar — therefore I think it is, in practice, the overton window of the person saying it. Others define it as "knowledge, judgement, and taste which is more or less universal and which is held more or less without reflection or argument", which LLMs absolutely do demonstrate. What you ask for, "getting through the next 30 seconds of life without a major screwup", would usually, 99.7% of the time, be passed by the Waymo autopilot.
  - 2y ago

bubblyworld 2y ago

Something that strikes me about this model is that it's bottom up - sensory data feeds in in its entirety, the action centre processes everything, makes a decision, sends a command to the motor centre. There's a theory that real brains subvert this, and what we perceive is actually our internal model of our self/environment. The only data that makes it through from our sense organs is the difference between the two. This kind of top-down processing is more efficient energy-wise but I wonder if it's deeper than that? You can view perception and action as two sides of the same coin - both are ways to modify your internal model to better fit the sensory signals you expect. Anyway, I guess the point I'm making is you should be careful which way you point your arrows, and of designating a single aspect of a mind (the action centre) as fundamental. Reality might work very differently, and that maybe says something? I don't know haha.

gnz11 2y ago
I think you are correct. My understanding is that the brain uses sense data just to confirm predictions it has already made. The model in which the brain is just reacting to incoming sense data is outdated from what I understand.
- cootsnuck 2y ago
  Correct. At least that's what current research is pointing to. Makes sense from an efficiency standpoint as well. We forget how "messy" / "noisy" our minds' conceptions of reality are. We use heuristics for a lot of things, we rely on applicable, learnable patterns. And our brains seem to be well suited to this more "artistic" interpretation of reality which biases our perception of sensations.
deleted 2y ago
[deleted]

paulmooreparks 2y ago

I've lately begun to think of conciousness as the ability to read and react to one's own log output. I don't like hypothesis by analogy, but it seems an apt description for what conscious entities do. I just don't see anything mystical about it.

Nevermark 2y ago
I agree completely, not mystical at all. It is just meta self-awareness. We evolved to be aware, model & react and to opportunistically control our environment. Loop 1. Then we evolved to be aware, model, & & react and to opportunistically control our selves as bodies. Loop 2. Then we evolved to be aware, model, & & react and to opportunistically control those activities too, our minds. In the process of being able to model & react to ourselves modelling & reacting to ourselves we completed a self-awareness of self-awareness loop. Loop 3. I think this is a good explanation because what else is consciousness but knowing you are conscious? Self-aware that you are self-aware? And it makes perfect sense as an evolutionary trajectory using greater levels of neural representation of our practical relationship with our environment, including ourselves, to create higher level survival options. So this definition is both a functional and developmental explanation of the emergent phenomenon of consciousness. Just as we only have partial access to information and control of our environment, our bodies, we are also limited to the degree we are aware and can control our mind. Introspection. The next level up is “theory of mind”, “empathy”, etc. our ability to model & interact with others mind’s as individuals and groups, reciprocally. Loop 4. That created society and culture, living information that extends outside us and grows and lives beyond each of us. Infusing our thoughts, and progressively thinking abilities into technology, that in theory and not too distant practice could have access to all parts of their own minds’ states, operations and design, would be Loop 5. When deeply conscious beings start, things will get interesting in the Sol system. When people start talking about quantum waves or smart rocks or universes in the context of consciousness I feel a little ill. People like to “solve” unrelated mysteries by merging them and shouting “tada!” (“We are puzzled about how they built the pyramids… but we don’t know if there are aliens either, so obviously… these mysteries solve each other!”)
- naasking 2y ago

freilanzer 2y ago

How is this blog generated? With code and latex formulas, it would be exactly what I'm looking for.

rnewme 2y ago
Seems to be substack. But should be easily done with pandoc and some shell scripting
- d4rkp4ttern 2y ago
  Unfortunately Substack doesn’t have math/latex/katex/mathjax.

Simplicitas 2y ago

Any discussion of a model for consciousness that doesn't include Daniel Dennett's take is a bit lacking from the get go.

antiquark 2y ago

Nice ideas... now build it!

m0llusk 2y ago

Would recommend reading The Ravenous Brain: How the New Science of Consciousness Explains Our Insatiable Search for Meaning by Daniel Bor for a lot of ideas strongly connected to recent research. My interpretation of this is the mind ends up being a story processing machine that builds stories about what has happened and is happening and constructs and compares stories about what might happen or be made to happen. Of course it is difficult to summarize a whole book rich with references in a sentence, but the model seems arguably more simple and well established than what you are currently putting forward. Very much looking forward to seeing continuing progress in all this.

devodo 2y ago

> (Pro-strong-AI)... This is basically a disbelief in the ability of physics to correctly describe what happens in the world — a well-established philosophical position. Are you giving up on physics? This is a very strong argument. Certainly all the ingredients to replicate a mind must exist within our physical reality. But does an algorithm running on a computer have access to all the physics required? For example, there are known physical phenomena, such as quantum entanglement, that are not possible to emulate with classical physics. How do we know our brains are not exploiting these, and possibly even yet unknown, physical phenomena? An algorithm running on a classical computer is executing in a very different environment than a brain that is directly part of physical reality.

clob 2y ago
> there are known physical phenomena, such as quantum entanglement, that are not possible to emulate with classical physics This is wrong. You can even get a mod for Minecraft which implements quantum mechanics. https://www.curseforge.com/minecraft/mc-mods/qcraft-reimagined https://www.curseforge.com/minecraft/mc-mods/qcraft-reimagin...
- justinpombrio 2y ago
  More precisely: you can emulate quantum mechanics using a classical computer, but the best known algorithms to do so take exponential running time.
  - devodo 2y ago
    This is not true. For example, Mermin's device cannot be done using classical physics regardless of running time. https://en.wikipedia.org/wiki/Mermin%27s_device https://en.wikipedia.org/wiki/Mermin%27s_device
    - deepburner 2y ago
      The first reference in the article you linked is titled "Computer simulation of Mermin's quantum device".

miika 2y ago

Ever since LLM’s came out many of us has been wondering these things. It would be easy to say that perhaps our attention and senses somehow come together to formulate prompts and thoughts etc what appears in the mind is the output. And everything we ever experienced has trained the model. But of course we can be assured it’s not quite like that in reality. This is just another example of how our models for explaining the life are reflection of the current technological state. Nobody considers that old clockwork universe now, and these AI inspired ideas are going to fall short all the same. Yet, progress is happening and all these ideas and talks are probably important steps that carry us forward.

jcynix 2y ago

> I’m motivated by the success of AI-based language models to look at the future of digital minds. When intelligent machines are constructed, we should not be surprised to find them as confused and as stubborn as men in their convictions about mind-matter, consciousness, free will, and the like. Minsky, as quoted in https://www.newyorker.com/magazine/1981/12/14/a-i https://www.newyorker.com/magazine/1981/12/14/a-i

abcde777666 2y ago

My instinct is that this is probably on the naive side. For instance, we use separation of concerns in our systems because we're too cognitively limited to create and manage deeply integrated systems. Nature doesn't have that problem. For instance, the idea that we can neatly have the emotion system separate from the motor control system. Emotions are a cacophony of chemicals and signals traversing the entire body - they're not an enum of happy/angry/sad - we just interpret them as such. So you probably don't get to isolate them off in a corner. Basically I think it's very tempting to severely underestimate the complexity of a problem when we're still only in theory land.