7 ms·

AI focused on brain regions recreates what you're looking at (2024)

79 points by openquery 1y ago

neonate 1y ago
https://archive.ph/650Az https://archive.ph/650Az
averageRoyalty 1y ago
Maybe I missed this, but isn't the underlying concept here big news? Am I understanding this right? It seems that by reading areas of the brain, a machine can effectively act as a rendering engine with knowledge on colour, brightness etc per pixel based on an image the person is seeing? And AI is being used to help because this method is lossy? This seems huge, is there other terminology around this I can kagi to understand more?
- walterbell 1y ago
  This requires intrusive electrodes, "fMRI visual recognition", https://scholar.google.com/scholar?q=fmri+visual+recognition https://scholar.google.com/scholar?q=fmri+visual+recognition There are startups working on less intrusive (e.g. headset) brain-computer interfaces (BCI).
  - Legend2440 1y ago
    fMRI isn't the one with the electrodes, it's the one with the giant scanner and no metal objects in the room.
- Legend2440 1y ago
  >And AI is being used to help because this method is lossy? AI is the method. They put somebody in a brain scanner and flash images on a screen in front of them. Then they train a neural network on the correlations between their brain activity and the known images. To test it, you display unknown images on the screen and have the neural network predict the image from the brain activity.
  - hwillis 1y ago
    > Then they train a neural network on the correlations between their brain activity and the known images. Not onto known images, onto latent spaces of existing image networks. The recognition network is getting a very approximate representation which it is then mapping onto latent spaces (which may or may not be equivalent) and then the image network is filling in the blanks. When you're using single-subject, well-framed images like this they're obviously very predictable. If you showed something unexpected, like a teddy bear with blue skin, the network probably would just show you a normal-ish teddy bear. It's also screwy if it doesn't have a well-correlated input, which is how you get those weird distortions. It will also be very off for things that require precision like seeing the actual outlines of an object, because the network is creating all that detail from nothing. At least the stuff using a Utah array (a square implanted electrode array) is not transferrable between subjects, and the fmri stuff also might not be transferrable. These models are not able to see enough detail to know what is happening- they only see glimpses of a small section of the process (Utah array) or very vague indirect processes (fmri). They're all very overfitted.
- ianarchy 1y ago
  Big blocker I believe, besides giant expensive fMRI machine, is each person is different, so model trained on Bob won’t work on Jane.
- ivape 1y ago
  Yeah, it's pretty crazy. This seems like it's inputting an image to the Monkeys eyes and then figuring out how that particular input maps to brain activity. Someone would have to fight me here, but with enough input, we should be able to mostly figure out how things map. As in, we can perfect this ...
cheschire 1y ago
I hope one day we can turn this on for coma patients and see if they're dreaming or otherwise processing the world.
- hwillis 1y ago
  Using these techniques, never. The electrode methods can only see a tiny section of processing and are missing all the information elsewhere. fMRI is very low resolution. Because of this they are all very overfitted- they cue off very particular subject-specific quirks that will not generalize well. More importantly, these techniques operate on the V1, V4 and inferior temporal cortex areas of the brain. These areas will fire in response to retina stimulation regardless of what's happening in the rest of your brain. V1 in particular is connected directly to your retinas. While deeper areas may be sympathetically activated by hallucinations etc, they aren't really related to your conception of things. In general if you want to read someone's thoughts you would look elsewhere in the brain.
  - khazhoux 1y ago
    He did say “one day”
    - hwillis 1y ago
      One day using different techniques. Not these techniques.
aitchnyu 1y ago
I want to see a cats POV when its startled by a cucumber (Youtube has lots of examples). A theory is that part of the brain mistook it for a snake. Also research on "constant bearing, decreasing range (CBDR)" where drivers may not notice another car/cycle in a perfectly clear crossroads till its too late.'
- explodes 1y ago
  For something like these kinds of reflexes, my understanding is that the response comes from the central nervous system, even before the brain has had the chance to fully process the input. This shortcut makes one avoid, say, burns or snakes, quicker than if it required the brain. Still, I agree with you that seeing what a cat sees (here or anywhere) would be awesome.
  - abeppu 1y ago
    I think the distinction you're drawing between "the central nervous system" and "the brain" is mistaken here -- the brain is part of the CNS. This kind of reflex basically has to involve the brain b/c it involves both the visual system and the motor system i.e. there's not a fast path from the retina to moving your appendages etc that doesn't include the brain. The "fully process" part is part of the story though -- e.g. perhaps some reactions use the dorsal stream based on peripheral vision while ventral stream is still waiting on a saccade and focus to get higher resolution foveal signals. But though these different pathways in the brain operate at different speeds, they're both still very much in the brain.
    - ljsprague 1y ago
      Some touch-based reflexes might avoid the higher parts of the brain though no?
      - abeppu 1y ago
        Yeah I think there are multiple documented cases of this, where especially well-practiced motor-plans seem to be 'pushed down', and if they're interrupted, correction can start faster than a round trip to the brain.
smusamashah 1y ago
It reminds of this research where faces monkey's were seeing were recreated almost identically. https://www.bbc.co.uk/news/science-environment-40131242 https://www.bbc.co.uk/news/science-environment-40131242 https://www.cell.com/cell/fulltext/S0092-8674(17)30538-X https://www.cell.com/cell/fulltext/S0092-8674(17)30538-X
abeppu 1y ago
I think it would be interesting to know if the viewer's familiarity with the object informs how accurate the reconstruction is. This shows presumably lab-raised macaques looking at boats and tarantulas and goldfish -- and that's cool. But presumably a macaque especially whose life has been indoors in confinement has no mental concepts for these things, so they're basically seeing still images of unfamiliar objects. If the animal has e.g. some favorite toys, or has eaten a range of foods, do they perceive these things with a higher detail and fidelity?
Animats 1y ago
The paper, at least as shown here, [1] is vague about which results came from implanted electrodes and which came from functional MRI data. Functional MRI data is showing blood flow. It's like looking at an IC with a thermal imager and trying to figure out what it is doing. [1] https://archive.is/650Az https://archive.is/650Az
- buildbot 1y ago
  That could be an interesting project in itself, take a simple 8 but microcontroller, a thermal camera, and some code that does different kinds of operations, see if you can train a classification model at least, or even generate the code running via an image to text llm.
  - moffkalast 1y ago
    Ah yes, yet another attack vector chip manufacturers will have to protect against now.
  - tharant 1y ago
    Please stop giving me project ideas. :)
- vo2maxer 1y ago
  Just to clarify, the paper [0] does use both implanted electrodes and fMRI data, but it is actually quite transparent about which data came from which source. The authors worked with two datasets: the B2G dataset, which includes multi-unit activity from implanted Utah arrays in macaques, and the Shen-19 dataset, which uses noninvasive fMRI from human participants. You’re right that fMRI measures blood flow rather than direct neural activity, and the authors acknowledge that limitation. But the study doesn’t treat it as a direct window into brain function. Instead, it proposes a predictive attention mechanism (PAM) that learns to selectively weigh signals from different brain areas, depending on the task of reconstructing perceived images from those signals. The “thermal imager” analogy might make sense in a different context, but in this case, the model is explicitly designed to deal with those signal differences and works across both modalities. If you’re curious, the paper is available here: [0] https://www.biorxiv.org/content/10.1101/2024.06.04.596589v2.full.pdf https://www.biorxiv.org/content/10.1101/2024.06.04.596589v2....
EasyMarion 1y ago
Big jump when we go from decoding what you’re seeing to what you’re thinking.
- Hoasi 1y ago
  In that case, you'll need the equivalent of ad-blockers for the brain, to prevent eavesdropping and intrusions by commercial and state actors.
- w_for_wumbo 1y ago
  This is a big jump ethically, but technically it feels like it's a hop away. If we can do this for visual images, we could use the same strategy on patterns of thought - especially if the person is a skilled at visualisation.
  - ivape 1y ago
    There are no ethics in China.
    - Gigachad 1y ago
      There are no ethics in America
  - pona-a 1y ago
    "Feels" and "is" are quite different in these domains. Self-driving feels like it's 5 years away for 10 years straight, and biology is infinitely more complex than automation. See this comment from this thread [0]. > These techniques operate on the V1, V4 and inferior temporal cortex areas of the brain. These areas will fire in response to retina stimulation regardless of what's happening in the rest of your brain. V1 in particular is connected directly to your retinas. While deeper areas may be sympathetically activated by hallucinations etc, they aren't really related to your conception of things. In general if you want to read someone's thoughts you would look elsewhere in the brain. [0] https://news.ycombinator.com/item?id=43910953 https://news.ycombinator.com/item?id=43910953
g0db1t 1y ago
[dead]
gitroom 1y ago
big leap for tech but honestly i get kinda uneasy thinking how far we're gonna take this brain decoding stuff
StefanBatory 1y ago
Ideally, every person working on that should be tried for crimes against humanity and locked away forever.

79 points by openquery 1y ago

https://archive.ph/650Az https://archive.ph/650Az

Maybe I missed this, but isn't the underlying concept here big news? Am I understanding this right? It seems that by reading areas of the brain, a machine can effectively act as a rendering engine with knowledge on colour, brightness etc per pixel based on an image the person is seeing? And AI is being used to help because this method is lossy? This seems huge, is there other terminology around this I can kagi to understand more?

walterbell 1y ago
This requires intrusive electrodes, "fMRI visual recognition", https://scholar.google.com/scholar?q=fmri+visual+recognition https://scholar.google.com/scholar?q=fmri+visual+recognition There are startups working on less intrusive (e.g. headset) brain-computer interfaces (BCI).
- Legend2440 1y ago
  fMRI isn't the one with the electrodes, it's the one with the giant scanner and no metal objects in the room.
Legend2440 1y ago
>And AI is being used to help because this method is lossy? AI is the method. They put somebody in a brain scanner and flash images on a screen in front of them. Then they train a neural network on the correlations between their brain activity and the known images. To test it, you display unknown images on the screen and have the neural network predict the image from the brain activity.
- hwillis 1y ago
  > Then they train a neural network on the correlations between their brain activity and the known images. Not onto known images, onto latent spaces of existing image networks. The recognition network is getting a very approximate representation which it is then mapping onto latent spaces (which may or may not be equivalent) and then the image network is filling in the blanks. When you're using single-subject, well-framed images like this they're obviously very predictable. If you showed something unexpected, like a teddy bear with blue skin, the network probably would just show you a normal-ish teddy bear. It's also screwy if it doesn't have a well-correlated input, which is how you get those weird distortions. It will also be very off for things that require precision like seeing the actual outlines of an object, because the network is creating all that detail from nothing. At least the stuff using a Utah array (a square implanted electrode array) is not transferrable between subjects, and the fmri stuff also might not be transferrable. These models are not able to see enough detail to know what is happening- they only see glimpses of a small section of the process (Utah array) or very vague indirect processes (fmri). They're all very overfitted.
ianarchy 1y ago
Big blocker I believe, besides giant expensive fMRI machine, is each person is different, so model trained on Bob won’t work on Jane.
ivape 1y ago
Yeah, it's pretty crazy. This seems like it's inputting an image to the Monkeys eyes and then figuring out how that particular input maps to brain activity. Someone would have to fight me here, but with enough input, we should be able to mostly figure out how things map. As in, we can perfect this ...

cheschire 1y ago

I hope one day we can turn this on for coma patients and see if they're dreaming or otherwise processing the world.

hwillis 1y ago
Using these techniques, never. The electrode methods can only see a tiny section of processing and are missing all the information elsewhere. fMRI is very low resolution. Because of this they are all very overfitted- they cue off very particular subject-specific quirks that will not generalize well. More importantly, these techniques operate on the V1, V4 and inferior temporal cortex areas of the brain. These areas will fire in response to retina stimulation regardless of what's happening in the rest of your brain. V1 in particular is connected directly to your retinas. While deeper areas may be sympathetically activated by hallucinations etc, they aren't really related to your conception of things. In general if you want to read someone's thoughts you would look elsewhere in the brain.
- khazhoux 1y ago
  He did say “one day”
  - hwillis 1y ago
    One day using different techniques. Not these techniques.

aitchnyu 1y ago

I want to see a cats POV when its startled by a cucumber (Youtube has lots of examples). A theory is that part of the brain mistook it for a snake. Also research on "constant bearing, decreasing range (CBDR)" where drivers may not notice another car/cycle in a perfectly clear crossroads till its too late.'

explodes 1y ago
For something like these kinds of reflexes, my understanding is that the response comes from the central nervous system, even before the brain has had the chance to fully process the input. This shortcut makes one avoid, say, burns or snakes, quicker than if it required the brain. Still, I agree with you that seeing what a cat sees (here or anywhere) would be awesome.
- abeppu 1y ago
  I think the distinction you're drawing between "the central nervous system" and "the brain" is mistaken here -- the brain is part of the CNS. This kind of reflex basically has to involve the brain b/c it involves both the visual system and the motor system i.e. there's not a fast path from the retina to moving your appendages etc that doesn't include the brain. The "fully process" part is part of the story though -- e.g. perhaps some reactions use the dorsal stream based on peripheral vision while ventral stream is still waiting on a saccade and focus to get higher resolution foveal signals. But though these different pathways in the brain operate at different speeds, they're both still very much in the brain.
  - ljsprague 1y ago
    Some touch-based reflexes might avoid the higher parts of the brain though no?
    - abeppu 1y ago
      Yeah I think there are multiple documented cases of this, where especially well-practiced motor-plans seem to be 'pushed down', and if they're interrupted, correction can start faster than a round trip to the brain.

smusamashah 1y ago

It reminds of this research where faces monkey's were seeing were recreated almost identically. https://www.bbc.co.uk/news/science-environment-40131242 https://www.bbc.co.uk/news/science-environment-40131242 https://www.cell.com/cell/fulltext/S0092-8674(17)30538-X https://www.cell.com/cell/fulltext/S0092-8674(17)30538-X

abeppu 1y ago

I think it would be interesting to know if the viewer's familiarity with the object informs how accurate the reconstruction is. This shows presumably lab-raised macaques looking at boats and tarantulas and goldfish -- and that's cool. But presumably a macaque especially whose life has been indoors in confinement has no mental concepts for these things, so they're basically seeing still images of unfamiliar objects. If the animal has e.g. some favorite toys, or has eaten a range of foods, do they perceive these things with a higher detail and fidelity?

Animats 1y ago

The paper, at least as shown here, [1] is vague about which results came from implanted electrodes and which came from functional MRI data. Functional MRI data is showing blood flow. It's like looking at an IC with a thermal imager and trying to figure out what it is doing. [1] https://archive.is/650Az https://archive.is/650Az

buildbot 1y ago
That could be an interesting project in itself, take a simple 8 but microcontroller, a thermal camera, and some code that does different kinds of operations, see if you can train a classification model at least, or even generate the code running via an image to text llm.
- moffkalast 1y ago
  Ah yes, yet another attack vector chip manufacturers will have to protect against now.
- tharant 1y ago
  Please stop giving me project ideas. :)
vo2maxer 1y ago
Just to clarify, the paper [0] does use both implanted electrodes and fMRI data, but it is actually quite transparent about which data came from which source. The authors worked with two datasets: the B2G dataset, which includes multi-unit activity from implanted Utah arrays in macaques, and the Shen-19 dataset, which uses noninvasive fMRI from human participants. You’re right that fMRI measures blood flow rather than direct neural activity, and the authors acknowledge that limitation. But the study doesn’t treat it as a direct window into brain function. Instead, it proposes a predictive attention mechanism (PAM) that learns to selectively weigh signals from different brain areas, depending on the task of reconstructing perceived images from those signals. The “thermal imager” analogy might make sense in a different context, but in this case, the model is explicitly designed to deal with those signal differences and works across both modalities. If you’re curious, the paper is available here: [0] https://www.biorxiv.org/content/10.1101/2024.06.04.596589v2.full.pdf https://www.biorxiv.org/content/10.1101/2024.06.04.596589v2....

EasyMarion 1y ago

Big jump when we go from decoding what you’re seeing to what you’re thinking.

Hoasi 1y ago
In that case, you'll need the equivalent of ad-blockers for the brain, to prevent eavesdropping and intrusions by commercial and state actors.
w_for_wumbo 1y ago
This is a big jump ethically, but technically it feels like it's a hop away. If we can do this for visual images, we could use the same strategy on patterns of thought - especially if the person is a skilled at visualisation.
- ivape 1y ago
  There are no ethics in China.
  - Gigachad 1y ago
    There are no ethics in America
- pona-a 1y ago
  "Feels" and "is" are quite different in these domains. Self-driving feels like it's 5 years away for 10 years straight, and biology is infinitely more complex than automation. See this comment from this thread [0]. > These techniques operate on the V1, V4 and inferior temporal cortex areas of the brain. These areas will fire in response to retina stimulation regardless of what's happening in the rest of your brain. V1 in particular is connected directly to your retinas. While deeper areas may be sympathetically activated by hallucinations etc, they aren't really related to your conception of things. In general if you want to read someone's thoughts you would look elsewhere in the brain. [0] https://news.ycombinator.com/item?id=43910953 https://news.ycombinator.com/item?id=43910953

g0db1t 1y ago

[dead]

gitroom 1y ago

big leap for tech but honestly i get kinda uneasy thinking how far we're gonna take this brain decoding stuff

StefanBatory 1y ago

Ideally, every person working on that should be tried for crimes against humanity and locked away forever.