5 ms·

The point stands whether you attribute agency to AIs themselves or to AI companies. The companies are not deliberately sabotaging human mathematical understand

The point stands whether you attribute agency to AIs themselves or to AI companies. The companies are not deliberately sabotaging human mathematical understanding by writing up purposely inscrutable results, either (which is what the word 'misalignment' would imply in this context): they are merely working with frontier tools that have a very limited capability for human-like understanding. It's left to human mathematicians to bridge that particular gap.

omnicognate 4d ago
You seem determined not to understand this, but to spell it out even further: The point is not that the companies are deliberately trying to harm maths but that in pursuing their own goals, which are different to those of mathematicians, they are doing so. Their goals and their methods of achieving them are not aligned with those of mathematicians. This is, incidentally, exactly what alignment means in AI as well: a paperclip maximiser harms humans not because it is intentionally trying to do so but as a side effect of trying to achieve its own objectives, which are not aligned with those of humans (which include not having their entire environment turned into paperclips). The word is being used entirely correctly, and I'm sure the nod to the AI usage is quite deliberate.
- zozbot234 4d ago
  > Their goals and their methods of achieving them are not aligned with those of mathematicians. This is exactly the assumption that Tao is smuggling in with "misalignment" talk and then refusing to elaborate on any further. Is the issue that AI companies are willfully refusing to provide mathematical insight that they could provide (because they have diverging underlying "goals" to those of human mathematicians) or are they merely working under a capability gap, where current AIs can awkwardly settle major open questions but are not smart enough to provide the kind of understanding and insight that the community of human mathematicians relies on? These are two very different problems and by foregrounding the word "misalignment" in his letter so openly (as opposed to talking about AI capability to provide valued insight), Tao is picking the more adversarial reading with zero proof or motivation.
  - omnicognate 4d ago
    His view is the "capability gap" one, and you can just read the the statement and Tao's other writings for far more eloquent explanations of the difference between a raw LLM proof dump and real mathematical insight, and what it takes to get from the former to the latter, than I can provide. Again, nobody is accusing anyone of deliberately trying to harm mathematics. It's about misaligned objectives. Eg. Tao wrote the following before the Navier-Stokes announcement (referring to exactly the project OpenAI was undertaking): > At this point, I would not be surprised if one could batter out such an extension by pouring an enormous amount of compute and AI assistance at such a task. But such an exercise does not particularly hold my interest; I am far more interested in digesting the proof methods and extracting out the key new insights uncovered by this approach. And then went further to say that such activity could be actively harmful to the field. (All this can be read on Mathstodon: https://mathstodon.xyz/@tao https://mathstodon.xyz/@tao). The misalignment is that this activity that he and 24 other Fields medalists think is actively harmful to their field is deemed by OpenAI and others to be worth ploughing vast financial, human and compute resources into.
    - zozbot234 4d ago
      > His view is the "capability gap" one That's the far more sensible reading, so thanks for confirming I guess. But then the misalignment talk is pretty clearly a distraction. > ...And then went further to say that such activity could be actively harmful to the field. If true (and there is as of yet insufficient evidence of this), that's merely a contingent fact about very real institutional misalignment within the human mathematical community, not about AI itself or even AI frontier labs. There's simply zero inherent reason why providing a bare truth value or a completely inscrutable proof about the status of some open conjecture should make that entire subfield of math "contaminated" for the foreseeable future when it comes to extracting further human-relevant insight. That's the misalignment we should be caring about.
      - SpicyLemonZest 4d ago
        The declaration is about that institutional misalignment. They want to change the norms of the mathematical community, so that producing inscrutable proofs is a low-value activity nobody cares much about rather than a high-value activity that AI frontier labs can make headlines by performing. Their ask of the frontier labs is to please be aware of the problems with the old norms and not exploit them during the transition period; they agree this doesn't have much to do with AI itself, which they acknowledge is a powerful technology that will improve and accelerate mathematical research.
- psychoslave 4d ago
  Even mathematical community as a whole is not that aligned with the stated goals. Not to blame mathematicians in general, its just a general anthropological platitude that the group actual behavior is not fully matching what people from the group want to pretend even in full honesty. Though it doesn't take even an outsider to question the matching here, just look what the only guy who settled a millennium prize what he thought about the community. Or ask some actual PhD, postdoc, or even better someone who dropped along the way, how does it feel to go through this community. Not sure the goals exposed in this article are really helped with the community as it is. One again, this doesn't mean everywhere encompasses the same issues and everyone is acting badly on is own. But if we want to accept happy pink shiny depiction of a community, we should be giving the same generosity to other communities à priori. All the more as AI and mathematical community do share a large set of common individuals.
JumpCrisscross 4d ago
> deliberately sabotaging human mathematical understanding...which is what the word 'misalignment' would imply in this context The entire point of the paperclip maximiser is the AI isn't evil. It isn't trying to hurt humans. It just doesn't care about us. Nobody claims OpenAI and Anthropic are out to torpedo mathematics. Just that relative to their internal goals of getting publicity ahead of IPOs by winning awards, what happens to mathematics and mathemeticians in the long run isn't a real concern. The Holocene [1] features relatively few species humans set out to eradicate. We mostly realised something had gone extinct after we accidentally destroyed them. That is what originally misalignment meant in respect of AI. [1] https://en.wikipedia.org/wiki/Holocene_extinction https://en.wikipedia.org/wiki/Holocene_extinction
DoctorOetker 4d ago
I wouldn't call the situation settled, nor do I expect we will ever find out what truly happened: - a rare low likelihood sequence of coincidences - deliberate frontier lab behavior, setting intellectual interrupts on LLM "aha"-moments, so that when some mathematician explains for the umpteenth time the approach they want to take "stop paraphrasing my approach, start the actual calculation!" and when the chatbot eventually groks it, they can scoop in, possibly with live dash-board and interrupt priority levels ranging from "ensure this sample gets into the new dataset" to "Millenium Prize Scoop Opportunity Imminent, call Sam"... There is always an alignment problem, incentives to ideals, behavior to incentives, ...

omnicognate 4d ago

You seem determined not to understand this, but to spell it out even further: The point is not that the companies are deliberately trying to harm maths but that in pursuing their own goals, which are different to those of mathematicians, they are doing so. Their goals and their methods of achieving them are not aligned with those of mathematicians. This is, incidentally, exactly what alignment means in AI as well: a paperclip maximiser harms humans not because it is intentionally trying to do so but as a side effect of trying to achieve its own objectives, which are not aligned with those of humans (which include not having their entire environment turned into paperclips). The word is being used entirely correctly, and I'm sure the nod to the AI usage is quite deliberate.

zozbot234 4d ago
> Their goals and their methods of achieving them are not aligned with those of mathematicians. This is exactly the assumption that Tao is smuggling in with "misalignment" talk and then refusing to elaborate on any further. Is the issue that AI companies are willfully refusing to provide mathematical insight that they could provide (because they have diverging underlying "goals" to those of human mathematicians) or are they merely working under a capability gap, where current AIs can awkwardly settle major open questions but are not smart enough to provide the kind of understanding and insight that the community of human mathematicians relies on? These are two very different problems and by foregrounding the word "misalignment" in his letter so openly (as opposed to talking about AI capability to provide valued insight), Tao is picking the more adversarial reading with zero proof or motivation.
- omnicognate 4d ago
  His view is the "capability gap" one, and you can just read the the statement and Tao's other writings for far more eloquent explanations of the difference between a raw LLM proof dump and real mathematical insight, and what it takes to get from the former to the latter, than I can provide. Again, nobody is accusing anyone of deliberately trying to harm mathematics. It's about misaligned objectives. Eg. Tao wrote the following before the Navier-Stokes announcement (referring to exactly the project OpenAI was undertaking): > At this point, I would not be surprised if one could batter out such an extension by pouring an enormous amount of compute and AI assistance at such a task. But such an exercise does not particularly hold my interest; I am far more interested in digesting the proof methods and extracting out the key new insights uncovered by this approach. And then went further to say that such activity could be actively harmful to the field. (All this can be read on Mathstodon: https://mathstodon.xyz/@tao https://mathstodon.xyz/@tao). The misalignment is that this activity that he and 24 other Fields medalists think is actively harmful to their field is deemed by OpenAI and others to be worth ploughing vast financial, human and compute resources into.
  - zozbot234 4d ago
    > His view is the "capability gap" one That's the far more sensible reading, so thanks for confirming I guess. But then the misalignment talk is pretty clearly a distraction. > ...And then went further to say that such activity could be actively harmful to the field. If true (and there is as of yet insufficient evidence of this), that's merely a contingent fact about very real institutional misalignment within the human mathematical community, not about AI itself or even AI frontier labs. There's simply zero inherent reason why providing a bare truth value or a completely inscrutable proof about the status of some open conjecture should make that entire subfield of math "contaminated" for the foreseeable future when it comes to extracting further human-relevant insight. That's the misalignment we should be caring about.
    - SpicyLemonZest 4d ago
      The declaration is about that institutional misalignment. They want to change the norms of the mathematical community, so that producing inscrutable proofs is a low-value activity nobody cares much about rather than a high-value activity that AI frontier labs can make headlines by performing. Their ask of the frontier labs is to please be aware of the problems with the old norms and not exploit them during the transition period; they agree this doesn't have much to do with AI itself, which they acknowledge is a powerful technology that will improve and accelerate mathematical research.
psychoslave 4d ago
Even mathematical community as a whole is not that aligned with the stated goals. Not to blame mathematicians in general, its just a general anthropological platitude that the group actual behavior is not fully matching what people from the group want to pretend even in full honesty. Though it doesn't take even an outsider to question the matching here, just look what the only guy who settled a millennium prize what he thought about the community. Or ask some actual PhD, postdoc, or even better someone who dropped along the way, how does it feel to go through this community. Not sure the goals exposed in this article are really helped with the community as it is. One again, this doesn't mean everywhere encompasses the same issues and everyone is acting badly on is own. But if we want to accept happy pink shiny depiction of a community, we should be giving the same generosity to other communities à priori. All the more as AI and mathematical community do share a large set of common individuals.

JumpCrisscross 4d ago

> deliberately sabotaging human mathematical understanding...which is what the word 'misalignment' would imply in this context The entire point of the paperclip maximiser is the AI isn't evil. It isn't trying to hurt humans. It just doesn't care about us. Nobody claims OpenAI and Anthropic are out to torpedo mathematics. Just that relative to their internal goals of getting publicity ahead of IPOs by winning awards, what happens to mathematics and mathemeticians in the long run isn't a real concern. The Holocene [1] features relatively few species humans set out to eradicate. We mostly realised something had gone extinct after we accidentally destroyed them. That is what originally misalignment meant in respect of AI. [1] https://en.wikipedia.org/wiki/Holocene_extinction https://en.wikipedia.org/wiki/Holocene_extinction

DoctorOetker 4d ago

I wouldn't call the situation settled, nor do I expect we will ever find out what truly happened: - a rare low likelihood sequence of coincidences - deliberate frontier lab behavior, setting intellectual interrupts on LLM "aha"-moments, so that when some mathematician explains for the umpteenth time the approach they want to take "stop paraphrasing my approach, start the actual calculation!" and when the chatbot eventually groks it, they can scoop in, possibly with live dash-board and interrupt priority levels ranging from "ensure this sample gets into the new dataset" to "Millenium Prize Scoop Opportunity Imminent, call Sam"... There is always an alignment problem, incentives to ideals, behavior to incentives, ...