6 ms·

Problems in AI alignment: A scale model

49 points by hamburga 1y ago

daveguy 1y ago
This is an excellent point. How we choose to use and interact with AI is an individual and stochastic collective. We can still choose not to give AI control.
- hamburga 1y ago
  I find myself using the term “human superorganism” a lot these days. Our bodies are composed of many cells and microbes working together; and if we zoom out, we have the dynamics in which human individuals operate together as superorganisms. IMO there’s a whole world of productive thinking to be done to attend to the health and strength of these entities.
  - breakyerself 1y ago
    But Margret Thatcher said there isn't even a society. Just individuals.
    - hamburga 1y ago
      Thatcher was briefly a research chemist; it’s extra weird that she couldn’t see beyond the atomic unit, to the societal equivalents of complex organic compounds.
- cokeandpepsi 1y ago
  [dead]
- constantcrying 1y ago
  Who is this "we"? Supposing a single person disagrees and decides to give AI more control and gains a very significant advantage by that, what then? I think people keep forgetting that "Selection" can be excessively cruel.
  - daveguy 1y ago
    You say that like AI has some sort of autonomy. It doesn't.
    - constantcrying 1y ago
      Doesn't matter.
    - selfhoster11 1y ago
      It doesn't, by default. All it takes is a capable enough model without rails, and a single user instructing it to act autonomously as its primary goal.
      - hamburga 1y ago
        All it takes for somebody to nuke Atlanta is an atom bomb and an airplane and somebody willing to fly the plane. I’m being facetious but there ARE ways to decide/act as a society and as subgroups within society that we want to disallow and punish and select out qualities of AIs that we think are unethical.
      - daveguy 1y ago
        The point is, AI is not able to act capably or autonomously. If someone tells it to "act autonomously" it'll either sit like a lump or flail until the power runs out.
  - hamburga 1y ago
    A single person acting in isolation (no friends, no colleagues, no customers) has very little agency. While theoretically a single person could release smallpox back into civilization, we have collectively selected it out very effectively.
    - constantcrying 1y ago
      The question is only relevant if AI is a significant force amplifier. If it is not, it is an unthreatening tool, whose usage should be largely unrestricted, this is the case now. If that ever changes then there is the question what to do with it and at a certain level of power an individual decision would have impact, if sufficiently amplified.
blamestross 1y ago
I'm kind of upset to see systematically "Alignment" and "AI Safety" co-opted for "undesirable business outcomes". These are existential problems, not mild profit blockers. Its almost like the goals of humanity and these companies are misaligned.
- godelski 1y ago
  > Its almost like the goals of humanity and these companies are misaligned. Certainly. I'd say that we've created a lot of Lemon Markets, if not an entire Lemon Economy[0]. The Lemon Market is literally an alignment problem, resultant from asymmetric information. Clearly the intent of the economy (via our social contract) is that we allocate money towards things that provide "value". Where I think we generally interpret that word to mean bettering peoples' lives in some form or another. But it is also clear that the term takes on other definitions and isn't perfectly aligned with making us better. Certainly our metrics can be hacked, as in the case of Lemon Markets. A well functioning market has competition that not only drives down prices but increases quality of products. Obviously customers want to simultaneously maximize quality and minimize price. But when customers cannot differentiate quality, they can only minimize price. Leading to the feedback loop, where producers are in a race to the bottom, making sacrifices to quality in favor of driving down prices (and thus driving up profits). Not because this is actually the thing that customers want! But because the market is inefficient. I think critical to these alignment issues is that they're not primarily driven by people trying to be malicious nor deceptive. They are more often driven by being short sighted and overlooking subtle nuances. They don't happen all at once, but instead slowly creep, making them more difficult to detect. It's like good horror: you might know something is wrong, but by the time you put it all together you're dead. It isn't because anyone is dumb or doing anything evil, but because maintaining alignment is difficult and mistakes are easy. [0] https://en.wikipedia.org/wiki/The_Market_for_Lemons https://en.wikipedia.org/wiki/The_Market_for_Lemons
- hamburga 1y ago
  Agreed. I see this more and more as the AI safety discourse spills more into the general lexicon and into PR efforts. For example, the “sycophantic” GPT 4o was also described as “misaligned” as code for “unlikable.” In the meme, I filed this under “personality programming.” Very different from the kinds of problems the original AI alignment writers were focused on.
godelski 1y ago
A critical part of AI alignment is understanding what goals besides the intended one maximize our training objectives. I think this is a thing that everyone kinda knows and will say but simultaneously are not giving anywhere near the depth of thought necessary to address the problems. Kind of like a clique: something everyone can repeat but frequently fails to implement in practice. Critically, when discussing intention I think there is not enough attention given to the fact that deception also maximizes RLHF, DPO, and any human preference based optimization. These are quite difficult things to measure and there's no formal mathematically derived evaluation. Alignment is incredibly difficult even in settings where measures have strong mathematical bases and we have means to make high quality measurements. But here, we have neither... We essentially are using the Justice Potter definition: I know it when I see it[0]. This has been highly successful and helped us make major strides! I don't want to detract from that in any way. But we also do have to recognize that there is a lurking danger that can create major problems. As long as it is based on human preference, well... we sure prefer a lie that doesn't sound like a lie compared to a lie that is obviously a lie. We obviously prefer truth and accuracy above either, but the notion of truth is fairly ill-defined and we really have no formal immutable definition outside highly constrained settings. It means that the models are also optimizing that their errors are difficult to detect. This is inherently a dangerous position, even if only from the standpoint that our optimization methods do not preclude this possibility. It may not be happening, but if it is, we may not know. The is the opposite of what is considered good design in all other forms of engineering. A lot of time is dedicated to error analysis and design. We specifically design things so that when they fail, or being to fail, that they do so in controllable and easily detectable ways. You don't want your bridges to fail, but when they fail you also don't want them to fail unpredictably. You don't want your code to fail, but when it does you don't want it leaking memory, spawning new processes, or doing any other wild things. You want it to come with easy to understand error messages. But our current design for AI and ML does not provide such a framework. This is true beyond LLMs. I'm not saying we should stop and I'm definitely not a doomer. I think AI and ML do a lot of good and will do much more good in the future[1]. It will also do harm, but I think the rewards outweigh the risks. But we should make sure we're not going into this completely blind and we should try to minimize the potential for harm. This isn't a call to stop, this is a call for more people to enter the space, a call for people already in the space to spend more time deeply thinking about these things. There's so many underlying subtleties that they are easy to miss, especially given all the excitement. We're definitely on an edge now, in the public eye, where if our work makes too many mistakes or too big of a mistake that it will risk shutting everything down. I know many might interpret me as being "a party pooper", but actually I want to keep the party going! But that also means making sure the party doesn't go overboard. Inviting a monkey with a machine gun sure will make the party legendary, but it's also a lot more likely to get it shut down a lot sooner with someone getting shot. So maybe let's just invite the monkey, but not with the machine gun? It won't be as epic, but I'm certain the good times will go on for much longer and we'll have much more fun in the long run. If the physicists can double check that the atomic bomb isn't going to destroy the world (something everyone was highly confident would not happen[2]), I think we can do this. Stakes are pretty similar, but the odds of our work doing high harm is greater. [0] https://en.wikipedia.org/wiki/Potter_Stewart https://en.wikipedia.org/wiki/Potter_Stewart [1] I'm a ML researcher myself! I'm passionate about creating these systems. But we need to recognize flaws and limitations if we are to improve them. Ignoring flaws and limits is playing with fire. Maybe you won't burn your house down, maybe you will. But you can't even determine the answer if you won't ask the question. [2] The story gets hyped, but it really wasn't believed. Despite this, they still double checked considering the risk. We could say the same thing about micro-blackholes with the LHC. Public finds out and gets scared, physicists really think it is near impossible, but run the calculations anyways. Why take that extreme level of risk, right?
comp_throw7 1y ago
I think this post is sort of confused because, centrally, the reason "AI Alignment" is a thing people talk about is because the problem, as originally envisioned, was to figure out how to avoid having superintelligent AI kill everyone. For a variety of reasons the term no longer refers primarily to that core problem, so the reason so many things that look like engineering problems have that label is mostly a historical artifact.
- hamburga 1y ago
  Not totally following your last point, though I do totally agree that there is this historical drift from “AI alignment” referring to existential risk, to today, where any AI personality you don’t like is “unaligned.” Still, “AI existential risk” is practically a different beast from “AI alignment,” and I’m trying to argue that the latter is not just for experts, but that it’s mostly a sociopolitical question of selection.
  - tbrownaw 1y ago
    > historical drift from “AI alignment” referring to existential risk, to today, where any AI personality you don’t like is “unaligned.” Alignment has always been "what it actually does doesn't match what it's meant to do". When the crowd that believes that AI will inevitably become an all-powerful God owned the news cycle, alignment concerns were of course presented through that lens. But it's actually rather interesting if approached seriously, especially when different people have different ideas about what it's meant to do.
  - mrbungie 1y ago
    What I understand from what GP was saying, is that AI Alignment today is more akin to trying to analyze and reduce error in an already fitted linear regressor rather than aligning AI behaviour and values to expected ones. Perhaps that has to do with the fact that aligning LLM-based AI systems has become a pseudo predictable engineering problem solvable via a "target, measure and reiterate cycle" rather than the highly philosophical and moral task old AI Alignment researchers thought it would be.
benlivengood 1y ago
https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ is the essay that crystalizes the reasons that Selection and Markets are not the forces we want to be aligning AI (or much else). In short (it is a very long article) fitness is not the same as goodness (by human standards) and so selection pressure will squeeze out goodness in favor of fitness, across all environments and niches, in the long run.
- hamburga 1y ago
  Why can’t we infuse Selection with Goodness? We’re the ones doing the selecting. We’ve selected out things like chattel slavery, for example. (Disclaimer: fell asleep after 10 minutes of reading the SSC post last night. I know it’s part of the HN Canon and perhaps I’m missing something)
verisimi 1y ago
> While Nature can’t do its selection on ethical grounds, we can, and do, when we select what kinds of companies and rules and power centers are filling which niches in our world. It’s a decentralized operation (like evolution), not controlled by any single entity, but consisting of the “sum total of the wills of the masses,” as Tolstoy put it. Alternatively, corporations and kings can manufacture the right kinds of opinions in people to sanction and direct the wills of the masses.
- hamburga 1y ago
  Indeed. Tolstoy does a deep exploration of this in War and Peace, using the example of Napoleon. Of course, this gets to the heart of the free will debate (to be settled in a future post ;)). Both are true at the same time - organized people and dictators and other factors simultaneously wrestle for influence in complex ways in which causation is impossible to measure. My own two cents, though, is that the Categorical Imperative is a tremendously important and underappreciated tool for raising the self-consciousness of groups. A practical implementation of it is linked at the bottom of the blog post.
constantcrying 1y ago
>Why isn’t there a “pharmaceutical alignment” or a “school curriculum alignment” Wikipedia page? >I think that the answer is “AI Alignment” has an implicit technical bent to it. If you go on the AI Alignment Forum, for example, you’ll find more math than Confucius or Foucault. What an absolutely insane thing to write. AI Alignment is different because it is trying to align something which is completely human made. Every other field is aligned "aligned" when the humans in it are "aligned", Outside of AI "alignment" is the subject of ethics (what is wrong and what is right) and law (How do we translate ethics into rules). What I think is absolutely important to understand is that throughout human history "alignment" has never happened. For every single thing you believe to be right there existed a human who considered that exact thing as completely wrong. Selection certainly has not created alignment.
- kalavan 1y ago
  > Every other field is aligned "aligned" when the humans in it are "aligned", That doesn't seem like the whole story. Pick two countries, for instance, one of which has evolved to be democratic (with high regard for rule of law, etc.) and the other is dictatorial. How did these countries end up the way they did? It probably has to do with rules, not just default human qualities. Let's say you consider popular participation to be good. Then you could say the humans who live in the first country are more "aligned" than the second, but the mechanisms of their forms of government also play part. E.g. if the bureaucracy is set up so that skillfully stabbing others in the back gets you political clout, the selection process will marginalize or kick out people who don't want to engage in backstabbing. Any organization's behavior depends on some combination of what its incentives promote and on the qualities of its members. This makes AI alignment just an extreme on a scale, not a thing set apart from all other kinds of alignment. The AI alignment problem is the "all rules" extreme of the scale, and organizational alignment is some combination of rules and the inclinations of the humans who are part of it. The ethics problem of "what does 'aligned' mean anyway" would both apply to the AI situation and the mixed organization situation. A dictator might want an AI "aligned" to maximize his own power, and would also want a human organization to be engineered in such a way as to be both obedient and effective. Someone of a more democratic predisposition would have other priorities - whether they are of what AIs should do or what human organizations should do.
shwouchk 1y ago
the world was completely unprepared for fully autonomous botnets

49 points by hamburga 1y ago

This is an excellent point. How we choose to use and interact with AI is an individual and stochastic collective. We can still choose not to give AI control.

hamburga 1y ago
I find myself using the term “human superorganism” a lot these days. Our bodies are composed of many cells and microbes working together; and if we zoom out, we have the dynamics in which human individuals operate together as superorganisms. IMO there’s a whole world of productive thinking to be done to attend to the health and strength of these entities.
- breakyerself 1y ago
  But Margret Thatcher said there isn't even a society. Just individuals.
  - hamburga 1y ago
    Thatcher was briefly a research chemist; it’s extra weird that she couldn’t see beyond the atomic unit, to the societal equivalents of complex organic compounds.
cokeandpepsi 1y ago
[dead]
constantcrying 1y ago
Who is this "we"? Supposing a single person disagrees and decides to give AI more control and gains a very significant advantage by that, what then? I think people keep forgetting that "Selection" can be excessively cruel.
- daveguy 1y ago
  You say that like AI has some sort of autonomy. It doesn't.
  - constantcrying 1y ago
    Doesn't matter.
  - selfhoster11 1y ago
    It doesn't, by default. All it takes is a capable enough model without rails, and a single user instructing it to act autonomously as its primary goal.
    - hamburga 1y ago
      All it takes for somebody to nuke Atlanta is an atom bomb and an airplane and somebody willing to fly the plane. I’m being facetious but there ARE ways to decide/act as a society and as subgroups within society that we want to disallow and punish and select out qualities of AIs that we think are unethical.
    - daveguy 1y ago
      The point is, AI is not able to act capably or autonomously. If someone tells it to "act autonomously" it'll either sit like a lump or flail until the power runs out.
- hamburga 1y ago
  A single person acting in isolation (no friends, no colleagues, no customers) has very little agency. While theoretically a single person could release smallpox back into civilization, we have collectively selected it out very effectively.
  - constantcrying 1y ago
    The question is only relevant if AI is a significant force amplifier. If it is not, it is an unthreatening tool, whose usage should be largely unrestricted, this is the case now. If that ever changes then there is the question what to do with it and at a certain level of power an individual decision would have impact, if sufficiently amplified.

blamestross 1y ago

I'm kind of upset to see systematically "Alignment" and "AI Safety" co-opted for "undesirable business outcomes". These are existential problems, not mild profit blockers. Its almost like the goals of humanity and these companies are misaligned.

godelski 1y ago
> Its almost like the goals of humanity and these companies are misaligned. Certainly. I'd say that we've created a lot of Lemon Markets, if not an entire Lemon Economy[0]. The Lemon Market is literally an alignment problem, resultant from asymmetric information. Clearly the intent of the economy (via our social contract) is that we allocate money towards things that provide "value". Where I think we generally interpret that word to mean bettering peoples' lives in some form or another. But it is also clear that the term takes on other definitions and isn't perfectly aligned with making us better. Certainly our metrics can be hacked, as in the case of Lemon Markets. A well functioning market has competition that not only drives down prices but increases quality of products. Obviously customers want to simultaneously maximize quality and minimize price. But when customers cannot differentiate quality, they can only minimize price. Leading to the feedback loop, where producers are in a race to the bottom, making sacrifices to quality in favor of driving down prices (and thus driving up profits). Not because this is actually the thing that customers want! But because the market is inefficient. I think critical to these alignment issues is that they're not primarily driven by people trying to be malicious nor deceptive. They are more often driven by being short sighted and overlooking subtle nuances. They don't happen all at once, but instead slowly creep, making them more difficult to detect. It's like good horror: you might know something is wrong, but by the time you put it all together you're dead. It isn't because anyone is dumb or doing anything evil, but because maintaining alignment is difficult and mistakes are easy. [0] https://en.wikipedia.org/wiki/The_Market_for_Lemons https://en.wikipedia.org/wiki/The_Market_for_Lemons
hamburga 1y ago
Agreed. I see this more and more as the AI safety discourse spills more into the general lexicon and into PR efforts. For example, the “sycophantic” GPT 4o was also described as “misaligned” as code for “unlikable.” In the meme, I filed this under “personality programming.” Very different from the kinds of problems the original AI alignment writers were focused on.

godelski 1y ago

A critical part of AI alignment is understanding what goals besides the intended one maximize our training objectives. I think this is a thing that everyone kinda knows and will say but simultaneously are not giving anywhere near the depth of thought necessary to address the problems. Kind of like a clique: something everyone can repeat but frequently fails to implement in practice. Critically, when discussing intention I think there is not enough attention given to the fact that deception also maximizes RLHF, DPO, and any human preference based optimization. These are quite difficult things to measure and there's no formal mathematically derived evaluation. Alignment is incredibly difficult even in settings where measures have strong mathematical bases and we have means to make high quality measurements. But here, we have neither... We essentially are using the Justice Potter definition: I know it when I see it[0]. This has been highly successful and helped us make major strides! I don't want to detract from that in any way. But we also do have to recognize that there is a lurking danger that can create major problems. As long as it is based on human preference, well... we sure prefer a lie that doesn't sound like a lie compared to a lie that is obviously a lie. We obviously prefer truth and accuracy above either, but the notion of truth is fairly ill-defined and we really have no formal immutable definition outside highly constrained settings. It means that the models are also optimizing that their errors are difficult to detect. This is inherently a dangerous position, even if only from the standpoint that our optimization methods do not preclude this possibility. It may not be happening, but if it is, we may not know. The is the opposite of what is considered good design in all other forms of engineering. A lot of time is dedicated to error analysis and design. We specifically design things so that when they fail, or being to fail, that they do so in controllable and easily detectable ways. You don't want your bridges to fail, but when they fail you also don't want them to fail unpredictably. You don't want your code to fail, but when it does you don't want it leaking memory, spawning new processes, or doing any other wild things. You want it to come with easy to understand error messages. But our current design for AI and ML does not provide such a framework. This is true beyond LLMs. I'm not saying we should stop and I'm definitely not a doomer. I think AI and ML do a lot of good and will do much more good in the future[1]. It will also do harm, but I think the rewards outweigh the risks. But we should make sure we're not going into this completely blind and we should try to minimize the potential for harm. This isn't a call to stop, this is a call for more people to enter the space, a call for people already in the space to spend more time deeply thinking about these things. There's so many underlying subtleties that they are easy to miss, especially given all the excitement. We're definitely on an edge now, in the public eye, where if our work makes too many mistakes or too big of a mistake that it will risk shutting everything down. I know many might interpret me as being "a party pooper", but actually I want to keep the party going! But that also means making sure the party doesn't go overboard. Inviting a monkey with a machine gun sure will make the party legendary, but it's also a lot more likely to get it shut down a lot sooner with someone getting shot. So maybe let's just invite the monkey, but not with the machine gun? It won't be as epic, but I'm certain the good times will go on for much longer and we'll have much more fun in the long run. If the physicists can double check that the atomic bomb isn't going to destroy the world (something everyone was highly confident would not happen[2]), I think we can do this. Stakes are pretty similar, but the odds of our work doing high harm is greater. [0] https://en.wikipedia.org/wiki/Potter_Stewart https://en.wikipedia.org/wiki/Potter_Stewart [1] I'm a ML researcher myself! I'm passionate about creating these systems. But we need to recognize flaws and limitations if we are to improve them. Ignoring flaws and limits is playing with fire. Maybe you won't burn your house down, maybe you will. But you can't even determine the answer if you won't ask the question. [2] The story gets hyped, but it really wasn't believed. Despite this, they still double checked considering the risk. We could say the same thing about micro-blackholes with the LHC. Public finds out and gets scared, physicists really think it is near impossible, but run the calculations anyways. Why take that extreme level of risk, right?

comp_throw7 1y ago

I think this post is sort of confused because, centrally, the reason "AI Alignment" is a thing people talk about is because the problem, as originally envisioned, was to figure out how to avoid having superintelligent AI kill everyone. For a variety of reasons the term no longer refers primarily to that core problem, so the reason so many things that look like engineering problems have that label is mostly a historical artifact.

hamburga 1y ago
Not totally following your last point, though I do totally agree that there is this historical drift from “AI alignment” referring to existential risk, to today, where any AI personality you don’t like is “unaligned.” Still, “AI existential risk” is practically a different beast from “AI alignment,” and I’m trying to argue that the latter is not just for experts, but that it’s mostly a sociopolitical question of selection.
- tbrownaw 1y ago
  > historical drift from “AI alignment” referring to existential risk, to today, where any AI personality you don’t like is “unaligned.” Alignment has always been "what it actually does doesn't match what it's meant to do". When the crowd that believes that AI will inevitably become an all-powerful God owned the news cycle, alignment concerns were of course presented through that lens. But it's actually rather interesting if approached seriously, especially when different people have different ideas about what it's meant to do.
- mrbungie 1y ago
  What I understand from what GP was saying, is that AI Alignment today is more akin to trying to analyze and reduce error in an already fitted linear regressor rather than aligning AI behaviour and values to expected ones. Perhaps that has to do with the fact that aligning LLM-based AI systems has become a pseudo predictable engineering problem solvable via a "target, measure and reiterate cycle" rather than the highly philosophical and moral task old AI Alignment researchers thought it would be.

benlivengood 1y ago

https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ is the essay that crystalizes the reasons that Selection and Markets are not the forces we want to be aligning AI (or much else). In short (it is a very long article) fitness is not the same as goodness (by human standards) and so selection pressure will squeeze out goodness in favor of fitness, across all environments and niches, in the long run.

hamburga 1y ago
Why can’t we infuse Selection with Goodness? We’re the ones doing the selecting. We’ve selected out things like chattel slavery, for example. (Disclaimer: fell asleep after 10 minutes of reading the SSC post last night. I know it’s part of the HN Canon and perhaps I’m missing something)

verisimi 1y ago

> While Nature can’t do its selection on ethical grounds, we can, and do, when we select what kinds of companies and rules and power centers are filling which niches in our world. It’s a decentralized operation (like evolution), not controlled by any single entity, but consisting of the “sum total of the wills of the masses,” as Tolstoy put it. Alternatively, corporations and kings can manufacture the right kinds of opinions in people to sanction and direct the wills of the masses.

hamburga 1y ago
Indeed. Tolstoy does a deep exploration of this in War and Peace, using the example of Napoleon. Of course, this gets to the heart of the free will debate (to be settled in a future post ;)). Both are true at the same time - organized people and dictators and other factors simultaneously wrestle for influence in complex ways in which causation is impossible to measure. My own two cents, though, is that the Categorical Imperative is a tremendously important and underappreciated tool for raising the self-consciousness of groups. A practical implementation of it is linked at the bottom of the blog post.

constantcrying 1y ago

>Why isn’t there a “pharmaceutical alignment” or a “school curriculum alignment” Wikipedia page? >I think that the answer is “AI Alignment” has an implicit technical bent to it. If you go on the AI Alignment Forum, for example, you’ll find more math than Confucius or Foucault. What an absolutely insane thing to write. AI Alignment is different because it is trying to align something which is completely human made. Every other field is aligned "aligned" when the humans in it are "aligned", Outside of AI "alignment" is the subject of ethics (what is wrong and what is right) and law (How do we translate ethics into rules). What I think is absolutely important to understand is that throughout human history "alignment" has never happened. For every single thing you believe to be right there existed a human who considered that exact thing as completely wrong. Selection certainly has not created alignment.

kalavan 1y ago
> Every other field is aligned "aligned" when the humans in it are "aligned", That doesn't seem like the whole story. Pick two countries, for instance, one of which has evolved to be democratic (with high regard for rule of law, etc.) and the other is dictatorial. How did these countries end up the way they did? It probably has to do with rules, not just default human qualities. Let's say you consider popular participation to be good. Then you could say the humans who live in the first country are more "aligned" than the second, but the mechanisms of their forms of government also play part. E.g. if the bureaucracy is set up so that skillfully stabbing others in the back gets you political clout, the selection process will marginalize or kick out people who don't want to engage in backstabbing. Any organization's behavior depends on some combination of what its incentives promote and on the qualities of its members. This makes AI alignment just an extreme on a scale, not a thing set apart from all other kinds of alignment. The AI alignment problem is the "all rules" extreme of the scale, and organizational alignment is some combination of rules and the inclinations of the humans who are part of it. The ethics problem of "what does 'aligned' mean anyway" would both apply to the AI situation and the mixed organization situation. A dictator might want an AI "aligned" to maximize his own power, and would also want a human organization to be engineered in such a way as to be both obedient and effective. Someone of a more democratic predisposition would have other priorities - whether they are of what AIs should do or what human organizations should do.

shwouchk 1y ago

the world was completely unprepared for fully autonomous botnets