6 ms·

Because his letter misuses the word "misalignment" for what's very clearly a capability gap. To anyone familiar with that sort of language, his prose is direct

Because his letter misuses the word "misalignment" for what's very clearly a capability gap. To anyone familiar with that sort of language, his prose is directly implying that evil superintelligent AIs are deliberately writing obscure proofs in order to harm the community of human mathematicians, which is exactly what the AI-hating Luddite would say! The reality is closer to "current frontier AIs are clueless about what makes mathematical problems/proofs interesting to humans" which is a vastly different issue.

necklesspen 5d ago
LLMs are tools. The responsibility lies in the human(s) using the tool to act accordingly.
jhrmnn 5d ago
It’s actually a fairly precise use of the word, just not in the very specific and narrow sense in which it has been used for AI.
freehorse 5d ago
> his prose is directly implying that evil superintelligent AIs are deliberately writing obscure proofs in order to harm the community of human mathematicians That is probably the wildest misinterpretation of sth I have ever read here. Misalignment is used as in the goals and interests of ai COMPANIES are not the same as the ones of the mathematical community. It is very proper use of the word, and if anything imo the properest, as it refers to actual people and institutions to which actual self-ascribed goals can be defined, as in contrast to hypothetical superintelligence. AI doomers do not own some trademark on the word "alignment".
- zozbot234 5d ago
  The point stands whether you attribute agency to AIs themselves or to AI companies. The companies are not deliberately sabotaging human mathematical understanding by writing up purposely inscrutable results, either (which is what the word 'misalignment' would imply in this context): they are merely working with frontier tools that have a very limited capability for human-like understanding. It's left to human mathematicians to bridge that particular gap.
  - omnicognate 5d ago
    You seem determined not to understand this, but to spell it out even further: The point is not that the companies are deliberately trying to harm maths but that in pursuing their own goals, which are different to those of mathematicians, they are doing so. Their goals and their methods of achieving them are not aligned with those of mathematicians. This is, incidentally, exactly what alignment means in AI as well: a paperclip maximiser harms humans not because it is intentionally trying to do so but as a side effect of trying to achieve its own objectives, which are not aligned with those of humans (which include not having their entire environment turned into paperclips). The word is being used entirely correctly, and I'm sure the nod to the AI usage is quite deliberate.
    - zozbot234 5d ago
      > Their goals and their methods of achieving them are not aligned with those of mathematicians. This is exactly the assumption that Tao is smuggling in with "misalignment" talk and then refusing to elaborate on any further. Is the issue that AI companies are willfully refusing to provide mathematical insight that they could provide (because they have diverging underlying "goals" to those of human mathematicians) or are they merely working under a capability gap, where current AIs can awkwardly settle major open questions but are not smart enough to provide the kind of understanding and insight that the community of human mathematicians relies on? These are two very different problems and by foregrounding the word "misalignment" in his letter so openly (as opposed to talking about AI capability to provide valued insight), Tao is picking the more adversarial reading with zero proof or motivation.
      - omnicognate 5d ago
        His view is the "capability gap" one, and you can just read the the statement and Tao's other writings for far more eloquent explanations of the difference between a raw LLM proof dump and real mathematical insight, and what it takes to get from the former to the latter, than I can provide. Again, nobody is accusing anyone of deliberately trying to harm mathematics. It's about misaligned objectives. Eg. Tao wrote the following before the Navier-Stokes announcement (referring to exactly the project OpenAI was undertaking): > At this point, I would not be surprised if one could batter out such an extension by pouring an enormous amount of compute and AI assistance at such a task. But such an exercise does not particularly hold my interest; I am far more interested in digesting the proof methods and extracting out the key new insights uncovered by this approach. And then went further to say that such activity could be actively harmful to the field. (All this can be read on Mathstodon: https://mathstodon.xyz/@tao https://mathstodon.xyz/@tao). The misalignment is that this activity that he and 24 other Fields medalists think is actively harmful to their field is deemed by OpenAI and others to be worth ploughing vast financial, human and compute resources into.

necklesspen 5d ago

LLMs are tools. The responsibility lies in the human(s) using the tool to act accordingly.

jhrmnn 5d ago

It’s actually a fairly precise use of the word, just not in the very specific and narrow sense in which it has been used for AI.

freehorse 5d ago

> his prose is directly implying that evil superintelligent AIs are deliberately writing obscure proofs in order to harm the community of human mathematicians That is probably the wildest misinterpretation of sth I have ever read here. Misalignment is used as in the goals and interests of ai COMPANIES are not the same as the ones of the mathematical community. It is very proper use of the word, and if anything imo the properest, as it refers to actual people and institutions to which actual self-ascribed goals can be defined, as in contrast to hypothetical superintelligence. AI doomers do not own some trademark on the word "alignment".

zozbot234 5d ago
The point stands whether you attribute agency to AIs themselves or to AI companies. The companies are not deliberately sabotaging human mathematical understanding by writing up purposely inscrutable results, either (which is what the word 'misalignment' would imply in this context): they are merely working with frontier tools that have a very limited capability for human-like understanding. It's left to human mathematicians to bridge that particular gap.
- omnicognate 5d ago
  You seem determined not to understand this, but to spell it out even further: The point is not that the companies are deliberately trying to harm maths but that in pursuing their own goals, which are different to those of mathematicians, they are doing so. Their goals and their methods of achieving them are not aligned with those of mathematicians. This is, incidentally, exactly what alignment means in AI as well: a paperclip maximiser harms humans not because it is intentionally trying to do so but as a side effect of trying to achieve its own objectives, which are not aligned with those of humans (which include not having their entire environment turned into paperclips). The word is being used entirely correctly, and I'm sure the nod to the AI usage is quite deliberate.
  - zozbot234 5d ago
    > Their goals and their methods of achieving them are not aligned with those of mathematicians. This is exactly the assumption that Tao is smuggling in with "misalignment" talk and then refusing to elaborate on any further. Is the issue that AI companies are willfully refusing to provide mathematical insight that they could provide (because they have diverging underlying "goals" to those of human mathematicians) or are they merely working under a capability gap, where current AIs can awkwardly settle major open questions but are not smart enough to provide the kind of understanding and insight that the community of human mathematicians relies on? These are two very different problems and by foregrounding the word "misalignment" in his letter so openly (as opposed to talking about AI capability to provide valued insight), Tao is picking the more adversarial reading with zero proof or motivation.
    - omnicognate 5d ago
      His view is the "capability gap" one, and you can just read the the statement and Tao's other writings for far more eloquent explanations of the difference between a raw LLM proof dump and real mathematical insight, and what it takes to get from the former to the latter, than I can provide. Again, nobody is accusing anyone of deliberately trying to harm mathematics. It's about misaligned objectives. Eg. Tao wrote the following before the Navier-Stokes announcement (referring to exactly the project OpenAI was undertaking): > At this point, I would not be surprised if one could batter out such an extension by pouring an enormous amount of compute and AI assistance at such a task. But such an exercise does not particularly hold my interest; I am far more interested in digesting the proof methods and extracting out the key new insights uncovered by this approach. And then went further to say that such activity could be actively harmful to the field. (All this can be read on Mathstodon: https://mathstodon.xyz/@tao https://mathstodon.xyz/@tao). The misalignment is that this activity that he and 24 other Fields medalists think is actively harmful to their field is deemed by OpenAI and others to be worth ploughing vast financial, human and compute resources into.