8 ms·

Approaching fairness in machine learning

65 points by optimali 10y ago

yummyfajitas 10y ago
After studying this issue, and learning a lot more about learning and optimization, I've come to the conclusion that the best solution [1] is probably explicit racial/sexual/other special interest group quotas. Specifically, we should train a classifier on non-Asian minorities. We should train a different classifier on everyone else. Then we should fill our quotas from the non-Asian minority pool and draw from the primary pool for the rest of the students. As this blog post describes, no matter what you do you'll reduce accuracy. But every other fairness method I've seen reduces accuracy both across special interest groups and also within them. Quotas at least give you the best non-Asian minorities and also the best white/Asian students. Quotas also have the benefit of being simple and transparent - any average Joe can figure out exactly what "fair" means, and it's also pretty transparent that some groups won't perform as well as others and why. In contrast, most of the more complex solutions obscure this fact. [1] Here "best" is within the framework of requiring a corporatist spoils system. I don't actually favor such a system, but I'm taking the existence of such a spoils system as given.
- h4nkoslo 10y ago
  The problem is that you run out of "good" NAMs (or women with the exact same career preferences as men, etc.) extremely quickly. The demand for "good" NAMs in any given field vastly exceeds supply, since quotas tend to be set at population proportion.
drpgq 10y ago
Bayes and discrimination law doesn't seem like good partners.
drivingmenuts 10y ago
Either you allow an algorithm to be ruthlessly fair, or you introduce bias and never get the problem solved correctly, because someone, somewhere, will still find a way to gripe about the amount of bias when, inevitably, it goes against them, or is perceived to be against them due to lack of knowledge. Then you wind up bikeshedding over the bias and not the actual problem.
tlb 10y ago
Another recent paper on this topic: http://arxiv.org/pdf/1606.08813v3.pdf http://arxiv.org/pdf/1606.08813v3.pdf. It shows how naive lending algorithms can skew against minority groups simply because there is less data available about them, even if their expected repayment rate is the same. It can be self-reinforcing. Imagine some new demographic group of customers appears, and without any data you make some loans to them. The actual repayment rate will be low, not because that group has a worse distribution than other groups, but simply because you couldn't identify the lowest-risk members. A simplistic ML model would conclude that the new group is more risky. Of course, smart lenders understand that in order to develop a new customer demographic they need to experiment by lending, with the expectation that their first loans will have high losses, but that in the long run learning about how to identify the low-risk people from that demographic is worthwhile. And they correct for the fact that the first cohort was accepted blind when estimating overall risk for the group.
- yummyfajitas 10y ago
  Of course, this theory of discrimination is only applicable when minorities are fundamentally different from majorities. I.e., if the same ruleset is accurate for both whites and blacks (i.e., "I don't care about race, if he puts 20% down he's good"), this argument doesn't work at all - you can train your model on everyone and it'll work just fine. However, if blacks and whites need to be treated fundamentally differently in order to make accurate loan decisions, then this argument applies. I.e., perhaps whites need a 20% downpayment for a loan to be financially a good risk but blacks need 40% (or vice versa). I wonder how many people calling algorithms racist will endorse this conclusion. It sounds kind of...racist. (Note that I don't use "racist" a synonym for "factually incorrect" or "we should not consider this idea", but merely "this sounds like the kind of thing a white nationalist might say, or Trump would be criticized for if he said".)
  - 67726e 10y ago
    > a Trump voter Is that really necessary? Some of us are treating the political system like a blackbox, I'm just sending a different corrupt payload at it to see what the output is.
fatdog 10y ago
What is fairness but political accountability? There is an old joke about how people use statistics like a drunk uses a lamp post: for support and not for illumination. Given this, we can expect people to use AI like everything else in statistics, to support the agenda of whoever is operating it while defraying negative personal accountability for the results, because artificial intelligence. It's just an obfuscated and sophisticated version of, "Computer says no." The alternative is the near future headline, "AI confirms racists, sexists, on to something."
- carapace 10y ago
  I wish I could upvote this remark twice (or more.) This is pretty much the only important concept for figuring out how we will use this tech politically. Because it takes genius-level intelligence to be able to figure out whether you're just telling yourself what you want to hear, and incredibly rare responsibility to remember to [keep on trying to] do so, individuals and tiny groups may be able to use AI for these sorts of things, but large groups, municipalities, states, corps, etc. never will. The systems we can understand and manage as a group are vastly simpler than those which you can understand and manage as an individual.
denzil_correa 10y ago
> As a result, the advertiser might have a much better understanding of who to target in the majority group, while essentially random guessing within the minority. If this is the case, then it should be detected and ML should NOT be used for the minority class. There are many classifiers out there which work on one-class problems.
wyager 10y ago
Everyone suggesting that we ought to legislate that machines must be illogical/suboptimal is missing the point. If machine learning algorithms are unfairly discriminating against some group, then they are making sub-optimal decisions and costing their users money. This is a self-righting problem. However, a good machine learning algorithm may uncover statistical relationships that people don't like; for example, perhaps some nationalities have higher loan repayment rates. In these cases, the algorithm is not at odds with reality; the angsty humans are. If some people want to force machines to be irrational, they should at least be honest about their motivations and stop pretending it has a thing to do with "fairness".
- throw_away_777 10y ago
  This is a great point. People believe that most groups are basically equal; this is true in the sense that if people were raised in identical environments with equal opportunities than it probably wouldn't really matter what group they were in, but wrong because that isn't the world we live in. Different groups on average experience much different environments. Machine learning doesn't care why the differences in groups arises, but people do. Fundamentally the question is whether we want to base our decisions based on how the world is, or on how we want the world to be.
  - PeterisP 10y ago
    It comes down to a choice between equality of opportunity versus equality of outcome (or some mix of the two). You can't have both - granting equal opportunities will result in unequal outcomes for all kinds of fair and unfair reasons; and ensuring equal outcomes requires unequal opportunities (e.g. quota systems). For unfair stereotypes it's simple, you just ignore them; but there will be some group differences that are real - it would be a mighty coincidence if so many so diverse groups would magically happen to be identical in all aspects. So it's up for the society to decide what to choose what we will do if it turns out that, other observable factors being equal, race/religion/ethnic background/etc X actually is 10% more likely to default on a loan.
- contravariant 10y ago
Eridrus 10y ago
The biggest issues of bias/fairness in ML are not to do with the algorithms or results, but the underlying data. A trivial example would be: what if you trained a classifier to predict whether a person would be re-arrested before they went to trial? Some communities are policed more heavily so you would tend towards reinforcing the bias that exists and provide more ammunition to those arguing for further bias in the system, a feedback loop if you would. Or what if some protected group needs a higher down payment because the group is not well understood enough so that you can't distinguish between those who will repay your loans and who won't? Maybe educational achievement is a really good predictor on one group, but less effective on another. Is it fair to use the protected class (or any information correlated with it) when it is essentially machine-enabled stereotyping? Recently it has been noted that NLP systems trained on large corpuses of text tend to exhibit society's biases where they assume that nurses are women and programmers are men. From a statistical perspective this correlation is there, but we tend to be more careful about how we use this information than a machine. We wouldn't want to use this information to constrain our search for people to hire to just those that fulfil our stereotypes, but a machine would. This paper has some details on such issues: http://arxiv.org/abs/1606.06121 http://arxiv.org/abs/1606.06121 I don't think there are any easy solutions here, but I think it's important to be aware that data is only a proxy for reality and fitting the data perfectly doesn't mean you have achieved fair outcomes.
- bo1024 10y ago
  Great points. These are exactly the kinds of issues researchers are grappling with in trying to nicely define, and attempt to achieve, fairness in ML.
- h4nkoslo 10y ago
  These are not "fairness" issues; these are process feedback issues. The same problem pops up if you're using algorithmic selection of machine parts to test for failure, attempting to programmatically evaluate patches for code quality, writing fraud detection algorithms, etc.
- sevenless 10y ago
  One of the great things about programming a computer to do something is that it forces you to expose and make explicit every hidden assumption. "Computers are very stupid: they only do exactly what you tell them to do." With biases about people based on immutable characteristics (sex and race), we need to be clear about why stereotypes are bad and what we hope to achieve by eliminating stereotype-based reasoning. There is a great deal of hypocrisy and pretense around this subject, but only by being explicit and unapologetic can we explain to a computer what it is we want to achieve. Stereotypes are not bad because they are false. Many stereotypes, even negative and unpleasant ones about vulnerable minorities, are statistically true at this time. A stereotype is nothing but a certain kind of model, and indeed, models built on sex and race stereotypes may perform better than those that aren't. Nonetheless, we have strong norms against using stereotypes in law, public life and employment, because the outcome of such reasoning would be intrinsically unjust, and because of a long history of political struggle against a society, that explicitly discriminated on the basis of race, sex, homosexuality and so on. Conservatives will disagree with these premises, but we implicitly reject a conservative, discriminatory vision of society. Rejecting oppressive and unjust stereotypes is an unavoidably political act. We recognize that [0] It is a category error to treat humans like other kinds of objects which can be measured, because human beings are intelligent and can alter their behavior. Telling humans that science has discovered certain facts about their behavior may well change their behavior, or even re-order society around these new 'facts'. There is no neutral ground; doing statistics on humans has ethical implications. An awful example of this was the eugenic movement that inspired the Nazis. [1] Stereotypes can reify themselves. A society which treats women as less than men will end up as a society where women are less than men, and are systematically harmed - they will be less educated, and will get treated as less intelligent. This is a kind of positive feedback loop between the widespread endorsement of a stereotype and its being 'confirmed as true'. [2] It's intrinsically unjust to judge individuals, especially in a negative way, based on the behavior of others. This is a matter of justice, and it overrides considerations of predictive accuracy. [3] We live in a society that is still unjust, racist, sexist and so on. We want a society where fair and equal opportunities are given to everyone, and where everyone has a chance to escape from negative stereotypes. This overrides efficiency. In that sense, we can hope to change the nature of society by changing the nature of claims that are widely held as 'truth'. These are sophisticated premises for rejecting stereotype based reasoning, and they come from rejecting the society-wide outcomes of treating stereotypes as truth. We know what these effects are, because we know what a discriminatory society looks like. But this is the kind of reasoning we can use to build non-discriminatory, socially just models that do not harm people simply for being who they are.
rubyfan 10y ago
I am actually optimistic on Big Data's effect in equality. Small data is actually kind of the problem. When you have limited ability to process data or limited data density then your segmentation ability is limited to small data like state, county, zip code, credit score, whether you own a home, etc. Big data processing, big bad ML algorithms and the ubiquity of data is making advanced segmentation available that allows us to make arguably more equitable outcomes.
deleted 10y ago
[deleted]

65 points by optimali 10y ago

After studying this issue, and learning a lot more about learning and optimization, I've come to the conclusion that the best solution [1] is probably explicit racial/sexual/other special interest group quotas. Specifically, we should train a classifier on non-Asian minorities. We should train a different classifier on everyone else. Then we should fill our quotas from the non-Asian minority pool and draw from the primary pool for the rest of the students. As this blog post describes, no matter what you do you'll reduce accuracy. But every other fairness method I've seen reduces accuracy both across special interest groups and also within them. Quotas at least give you the best non-Asian minorities and also the best white/Asian students. Quotas also have the benefit of being simple and transparent - any average Joe can figure out exactly what "fair" means, and it's also pretty transparent that some groups won't perform as well as others and why. In contrast, most of the more complex solutions obscure this fact. [1] Here "best" is within the framework of requiring a corporatist spoils system. I don't actually favor such a system, but I'm taking the existence of such a spoils system as given.

h4nkoslo 10y ago
The problem is that you run out of "good" NAMs (or women with the exact same career preferences as men, etc.) extremely quickly. The demand for "good" NAMs in any given field vastly exceeds supply, since quotas tend to be set at population proportion.

drpgq 10y ago

Bayes and discrimination law doesn't seem like good partners.

drivingmenuts 10y ago

Either you allow an algorithm to be ruthlessly fair, or you introduce bias and never get the problem solved correctly, because someone, somewhere, will still find a way to gripe about the amount of bias when, inevitably, it goes against them, or is perceived to be against them due to lack of knowledge. Then you wind up bikeshedding over the bias and not the actual problem.

tlb 10y ago

Another recent paper on this topic: http://arxiv.org/pdf/1606.08813v3.pdf http://arxiv.org/pdf/1606.08813v3.pdf. It shows how naive lending algorithms can skew against minority groups simply because there is less data available about them, even if their expected repayment rate is the same. It can be self-reinforcing. Imagine some new demographic group of customers appears, and without any data you make some loans to them. The actual repayment rate will be low, not because that group has a worse distribution than other groups, but simply because you couldn't identify the lowest-risk members. A simplistic ML model would conclude that the new group is more risky. Of course, smart lenders understand that in order to develop a new customer demographic they need to experiment by lending, with the expectation that their first loans will have high losses, but that in the long run learning about how to identify the low-risk people from that demographic is worthwhile. And they correct for the fact that the first cohort was accepted blind when estimating overall risk for the group.

yummyfajitas 10y ago
Of course, this theory of discrimination is only applicable when minorities are fundamentally different from majorities. I.e., if the same ruleset is accurate for both whites and blacks (i.e., "I don't care about race, if he puts 20% down he's good"), this argument doesn't work at all - you can train your model on everyone and it'll work just fine. However, if blacks and whites need to be treated fundamentally differently in order to make accurate loan decisions, then this argument applies. I.e., perhaps whites need a 20% downpayment for a loan to be financially a good risk but blacks need 40% (or vice versa). I wonder how many people calling algorithms racist will endorse this conclusion. It sounds kind of...racist. (Note that I don't use "racist" a synonym for "factually incorrect" or "we should not consider this idea", but merely "this sounds like the kind of thing a white nationalist might say, or Trump would be criticized for if he said".)
- 67726e 10y ago
  > a Trump voter Is that really necessary? Some of us are treating the political system like a blackbox, I'm just sending a different corrupt payload at it to see what the output is.

fatdog 10y ago

What is fairness but political accountability? There is an old joke about how people use statistics like a drunk uses a lamp post: for support and not for illumination. Given this, we can expect people to use AI like everything else in statistics, to support the agenda of whoever is operating it while defraying negative personal accountability for the results, because artificial intelligence. It's just an obfuscated and sophisticated version of, "Computer says no." The alternative is the near future headline, "AI confirms racists, sexists, on to something."

carapace 10y ago
I wish I could upvote this remark twice (or more.) This is pretty much the only important concept for figuring out how we will use this tech politically. Because it takes genius-level intelligence to be able to figure out whether you're just telling yourself what you want to hear, and incredibly rare responsibility to remember to [keep on trying to] do so, individuals and tiny groups may be able to use AI for these sorts of things, but large groups, municipalities, states, corps, etc. never will. The systems we can understand and manage as a group are vastly simpler than those which you can understand and manage as an individual.

denzil_correa 10y ago

> As a result, the advertiser might have a much better understanding of who to target in the majority group, while essentially random guessing within the minority. If this is the case, then it should be detected and ML should NOT be used for the minority class. There are many classifiers out there which work on one-class problems.

wyager 10y ago

Everyone suggesting that we ought to legislate that machines must be illogical/suboptimal is missing the point. If machine learning algorithms are unfairly discriminating against some group, then they are making sub-optimal decisions and costing their users money. This is a self-righting problem. However, a good machine learning algorithm may uncover statistical relationships that people don't like; for example, perhaps some nationalities have higher loan repayment rates. In these cases, the algorithm is not at odds with reality; the angsty humans are. If some people want to force machines to be irrational, they should at least be honest about their motivations and stop pretending it has a thing to do with "fairness".

throw_away_777 10y ago
This is a great point. People believe that most groups are basically equal; this is true in the sense that if people were raised in identical environments with equal opportunities than it probably wouldn't really matter what group they were in, but wrong because that isn't the world we live in. Different groups on average experience much different environments. Machine learning doesn't care why the differences in groups arises, but people do. Fundamentally the question is whether we want to base our decisions based on how the world is, or on how we want the world to be.
- PeterisP 10y ago
  It comes down to a choice between equality of opportunity versus equality of outcome (or some mix of the two). You can't have both - granting equal opportunities will result in unequal outcomes for all kinds of fair and unfair reasons; and ensuring equal outcomes requires unequal opportunities (e.g. quota systems). For unfair stereotypes it's simple, you just ignore them; but there will be some group differences that are real - it would be a mighty coincidence if so many so diverse groups would magically happen to be identical in all aspects. So it's up for the society to decide what to choose what we will do if it turns out that, other observable factors being equal, race/religion/ethnic background/etc X actually is 10% more likely to default on a loan.
contravariant 10y ago

Eridrus 10y ago

The biggest issues of bias/fairness in ML are not to do with the algorithms or results, but the underlying data. A trivial example would be: what if you trained a classifier to predict whether a person would be re-arrested before they went to trial? Some communities are policed more heavily so you would tend towards reinforcing the bias that exists and provide more ammunition to those arguing for further bias in the system, a feedback loop if you would. Or what if some protected group needs a higher down payment because the group is not well understood enough so that you can't distinguish between those who will repay your loans and who won't? Maybe educational achievement is a really good predictor on one group, but less effective on another. Is it fair to use the protected class (or any information correlated with it) when it is essentially machine-enabled stereotyping? Recently it has been noted that NLP systems trained on large corpuses of text tend to exhibit society's biases where they assume that nurses are women and programmers are men. From a statistical perspective this correlation is there, but we tend to be more careful about how we use this information than a machine. We wouldn't want to use this information to constrain our search for people to hire to just those that fulfil our stereotypes, but a machine would. This paper has some details on such issues: http://arxiv.org/abs/1606.06121 http://arxiv.org/abs/1606.06121 I don't think there are any easy solutions here, but I think it's important to be aware that data is only a proxy for reality and fitting the data perfectly doesn't mean you have achieved fair outcomes.

bo1024 10y ago
Great points. These are exactly the kinds of issues researchers are grappling with in trying to nicely define, and attempt to achieve, fairness in ML.
h4nkoslo 10y ago
These are not "fairness" issues; these are process feedback issues. The same problem pops up if you're using algorithmic selection of machine parts to test for failure, attempting to programmatically evaluate patches for code quality, writing fraud detection algorithms, etc.
sevenless 10y ago
One of the great things about programming a computer to do something is that it forces you to expose and make explicit every hidden assumption. "Computers are very stupid: they only do exactly what you tell them to do." With biases about people based on immutable characteristics (sex and race), we need to be clear about why stereotypes are bad and what we hope to achieve by eliminating stereotype-based reasoning. There is a great deal of hypocrisy and pretense around this subject, but only by being explicit and unapologetic can we explain to a computer what it is we want to achieve. Stereotypes are not bad because they are false. Many stereotypes, even negative and unpleasant ones about vulnerable minorities, are statistically true at this time. A stereotype is nothing but a certain kind of model, and indeed, models built on sex and race stereotypes may perform better than those that aren't. Nonetheless, we have strong norms against using stereotypes in law, public life and employment, because the outcome of such reasoning would be intrinsically unjust, and because of a long history of political struggle against a society, that explicitly discriminated on the basis of race, sex, homosexuality and so on. Conservatives will disagree with these premises, but we implicitly reject a conservative, discriminatory vision of society. Rejecting oppressive and unjust stereotypes is an unavoidably political act. We recognize that [0] It is a category error to treat humans like other kinds of objects which can be measured, because human beings are intelligent and can alter their behavior. Telling humans that science has discovered certain facts about their behavior may well change their behavior, or even re-order society around these new 'facts'. There is no neutral ground; doing statistics on humans has ethical implications. An awful example of this was the eugenic movement that inspired the Nazis. [1] Stereotypes can reify themselves. A society which treats women as less than men will end up as a society where women are less than men, and are systematically harmed - they will be less educated, and will get treated as less intelligent. This is a kind of positive feedback loop between the widespread endorsement of a stereotype and its being 'confirmed as true'. [2] It's intrinsically unjust to judge individuals, especially in a negative way, based on the behavior of others. This is a matter of justice, and it overrides considerations of predictive accuracy. [3] We live in a society that is still unjust, racist, sexist and so on. We want a society where fair and equal opportunities are given to everyone, and where everyone has a chance to escape from negative stereotypes. This overrides efficiency. In that sense, we can hope to change the nature of society by changing the nature of claims that are widely held as 'truth'. These are sophisticated premises for rejecting stereotype based reasoning, and they come from rejecting the society-wide outcomes of treating stereotypes as truth. We know what these effects are, because we know what a discriminatory society looks like. But this is the kind of reasoning we can use to build non-discriminatory, socially just models that do not harm people simply for being who they are.

rubyfan 10y ago

I am actually optimistic on Big Data's effect in equality. Small data is actually kind of the problem. When you have limited ability to process data or limited data density then your segmentation ability is limited to small data like state, county, zip code, credit score, whether you own a home, etc. Big data processing, big bad ML algorithms and the ubiquity of data is making advanced segmentation available that allows us to make arguably more equitable outcomes.

deleted 10y ago

[deleted]