10 ms·

Notes on AI Bias

146 points by andrevoget 7y ago

fvdessen 7y ago
> Since Amazon’s current employee base skews male, the examples of ‘successful hires’ also, mechanistically, skewed male and so, therefore, did this system’s selection of resumés. Amazon spotted this and the system was never put into production. Couldn't they have retrained the system with a 50/50 mix of males / females resumes ? Or restrict the use of the algorithm to sort male resumes ? Or maybe resumes don't actually correlate at all with success in Amazon ...
- deogeo 7y ago
  Sounds like an extraordinarily poor AI system if it depends on absolute numbers, and not per capita. And wouldn't the number of unsuccessful hires also skew male?
  - theoh 7y ago
    Consider the possibility that the (pre-AI system) probability of success for a female applicant is the same as the probability of success of a male applicant. You could make a "per capita" quota as a kind of goal. That's not a problem, but how would you make sure the quota was met? The typical AI system doesn't work on the basis of selecting candidates entirely at random, pro rata, in order to meet a quota. It works on the basis of criteria for success. One thing it might learn (unfortunately) is that most posts at the company are filled by men.
    - cygaril 7y ago
      From a machine learning point of view, one can just add the constraint that the probability of being in the "yes" bucket is that same for both male and female candidates. Doing this will give a worse fit than an unconstrained optimization, but it is fairer. More sophisticated approaches are possible.
      - theoh 7y ago
        There's no "just" to any aspect of this topic. I think what you are talking about is what is sometimes called "classification parity", and there are problems with it, and with everything else we've come up with to combat bias. https://arxiv.org/abs/1808.00023 https://arxiv.org/abs/1808.00023
  - jerf 7y ago
    "Sounds like an extraordinarily poor AI system if it depends on absolute numbers, and not per capita." To some extent, you're bringing in your human bias to prefer human biases when you make that statement. We humans have a hierarchy of important attributes, and for various reasons believe race and gender are more important than eye color or height. But the machine learning algorithm just gets a multidimensional point in hyperspace. It doesn't, a priori, "know" that it needs to do a "per capita" adjustment based on FIELD_1 any more than it knows it needs to do a per capita adjustment on FIELD_2. And you can't "adjust" on all the fields because that'll just cancel out. We are also in the weird position of wanting the machine to do adjustments based on FIELD_1, but without us having to actually admit to ourselves that we're doing it. From a technical perspective, probably the best answer is to do a straight-up training based on the data, then have an cleanly-separated after-the-fact cleanup process to perform whatever social adjustments it is we want on the outcome. But nobody is willing to admit that's what we want, and to put those adjustments down on paper in the form of code, because the instant they're concrete, pretty much everybody is going to decide they're wrong, and no two people are going to agree on the manner in which they are wrong, and an epic, national-front-page-news shitstorm will ensue. So here we are, trying to make adjustments without making adjustments, or, alternatively, trying to make adjustments in a place where we can blame the AI rather than humans. (The ironic thing is that because we can't admit what we're trying to do, we're going to end up doing a really poor job of it. Tools will be applied haphazardly, the results can't be measured except very grossly at the very end of the process, and the goals won't be obtained and the system is always going to be quirky and weird. If we could clearly declare what it is we actually wanted, it would be fairly easy to get it from the AIs.)
  - Bartweiss 7y ago
    The basic "resumes skewed male so the algorithm did too" explanation appears to be incorrect. But it's found in the original Reuters story and most derived stories, and finding it here implies it's reached the level of urban legend. Going by the details of the Reuters story and several others, it appears that what actually happened was a training/task mismatch. Amazon wanted an algorithm to do resume discovery, which recruiters would run and get quality predictions as they viewed resumes. But they trained it on resume results, giving it past resumes which had been submitted to Amazon and telling it to seek similar resumes. None of the stories make it clear if there even was negative training data; it looks like the tool was simply told to compute degree-of-similarity to past inputs, and possibly told to prioritize resumes which were ultimately hired. As a result, the tool was trying to convert a relatively gender-neutral pool (resumes found online) to a skewed one (Amazon applicant resumes), and did so by weighting gendered terms. It also seems to have underweighted technical terms, failing to appreciate them as mandatory or strictly position-specific. The developers were sufficiently aware of that to catch and correct the known gender biases (e.g. devaluing women's colleges or the literal word "women's"), but were scared there were other uncaught biases. And the results were apparently terrible all around, so the tool was scrapped. Which is pretty much what you'd expect from something trained on exclusively positive, sample-biased examples. The story has been seriously distorted, but the real plan also seems terrible...
- duxup 7y ago
  It does make you wonder how much AI will be .. AI and how much guidance for desired outcomes humans will give it. Humans are pretty happy to create nonsensical results if it fits their goals... especially if it befits them. I wonder if with AI we do that to the point that it is somewhat irrelevant.
- TheRealPomax 7y ago
  maybe, but this might also just be someone unwilling to commit to the sunk cost fallacy. You can spend time and money fixing it, or you can cut your losses and just stop trying to automate something that probably didn't need full automation to begin with.
- FakeComments 7y ago
  If the only problem is that it mixes up male and female resumes, why not have it sort each into two piles and have a human recruiter perform the merge by picking the better applicant from the pair on top, repeatedly? I have the suspicion that Amazons AI was factually correct, but politically wrong.
  - TheCoelacanth 7y ago
    I highly doubt that the preference for male candidates was the only problem with the AI. The preference for candidates with ice hockey on their resume almost certainly also would have resulted in a preference for white candidates.
    - FakeComments 7y ago
      Sure, my point is why does that matter? The correlation with ice hockey could be a career relevant detail because of correlation with discipline and pain tolerance. If it tracks all such signals, I don’t see the problem. That it also has a circumstantial correlation with races isn’t inherently problematic — it could be that different groups of people are differently qualified. There seems to be a position that culture is uncorrelated to job success, and that’s just nonsense. When people can show me these biases are correlated to race or sex once controlling for culture, then we have a problem.
      - TheCoelacanth 7y ago
        Good luck with that argument in court.
      - michaelmior 7y ago
        Race and culture are two very different things. There certainly can be some correlation, but drawing conclusions of suitability for employment based on race is a problem.
        FakeComments 7y ago
- eanzenberg 7y ago
  That wouldn't matter if the KPI (worker performance) being predicted, which is inherently biased as well, was distributed differently among the balanced pool of applicants.
- munchbunny 7y ago
  I think this retelling of the story is over-simplified. It's a compelling story, but I don't know any competent engineers who give up on a whole project because of one setback. If this system never saw production use, it was because it's still not ready, or there were many other issues that aren't mentioned that led the team to give up, or because political winds shifted. Amazon is famous for killing projects quickly.
- manfredo 7y ago
  The whole problem with allegations of AI bias is that people often point to disparities of outcome as proof of bias. The reality is that there are plenty of disparities on outcome regardless of bias, and the allegations of bias and attempt to rectify the alleged bias is another vector for the introduction of bias.
- DuskStar 7y ago
  One situation I could see leading to this result (Amazon cancelling their resume filtering software with the excuse that it 'skewed male') is that 1. The AI system accurately predicted employee success across both genders AND 2. The AI system predicted that women would do worse than men That's politically embarrassing and something that you can't necessarily 'fix' by improving the system. (see: all the 'will this person commit a crime if let out on parole' systems that end up accurately discriminating based on race) This isn't to say that women are worse engineers than men, or anything of that sort - only that the applicant pool to Amazon was skewed, or women were treated worse in the workplace and thus performed worse, or a dozen other possible causes. (And only in this hypothetical scenario! I have no inside info from Amazon!)
  - gizmo686 7y ago
    Your example is quite possible, particularly at an organazation that would be embarrased by such a result. Assume that the ability curve of male applicants and female applicants are identical; that the majority of applicants are male; and that Amazon wants to hire more females then would be expected given the portion of applicants that are female. A natural way of accomplishing this goal is to give extra points to female applicants [0]. Due to selection bias, the ability curve of women within the population of Amazon engineers would skew lower then men within the population of Amazon engineers. This is a special case of a more general phenomona. If you have signal S that is positivly correlated with a desired trait in the general population, and over select for S, you will find that S is negativly correlated within your population. [0]. All proposals I have seen amount to either a good approximation of this or changing the applicant pool. And, by assumption, the latter is excluded.
  - lotu 7y ago
    This is feels like an elephant in the room when it comes to AI bias. We develop an AI that accurately predicts outcomes and discover it is biased, then instead of asking if maybe this means our current system is deeply biased and needs to be changed, we say, "don't use the AI; keep using the people who might or might not be biased but we don't know because we can't measure it in the way an AI can be measured." If it isn't acceptable to use an AI to create biased outcomes how is it acceptable to use people to create the the same outcomes. AI decision making can be examined and tuned in ways that people cannot.
- turtlecloud 7y ago
  Just remove the gender/sex as variables for the AI and maybe name too. Preprocess the resumes to remove them. Now you remove the majority of gender bias for the AI.
  - gizmo686 7y ago
    AI is really good at infering information. If gender is a real signal, it would be very difficult to filter the input such that it is not making a determination by what could be reffered to inferred gender.
- Bartweiss 7y ago
  This story has been constantly misrepresented, because Reuters absolutely botched their initial report. Amazon was never building a tool to decide which interviewed candidates to hire, they were building a tool for discovering candidates. It was biased, but that gender bias wasn't the proximate reason for scrapping the tool. As far as I can tell from later stories (e.g. 1, 2), what Amazon actually did was build a tool to show recruiters 'quality' predictions for all resumes, for instance as they scrolled LinkedIn. But they trained it on resumes submitted to Amazon for various positions, possibly also adding weight to resumes which produced hires. In which case the problem is painfully obvious; the system effectively had no negative training data, and its positive examples (submitted resumes) didn't actually match the desired output (qualified resumes). It was computing degree of similarity between a gender-neutral-ish pool (resumes posted online) and a gender-skewed pool (resumes submitted to Amazon), and tried to make that conversion with whatever data was available - like devaluing resumes that mentioned women's colleges. (This wasn't just a proxy-variable thing, the model essentially learned to weight on gender.) Amazon's team apparently caught this issue and did the usual things like blinding on those words. But they were scared of uncaught factors; reading between the lines, they were unable to "detrain" biases like neural nets do because their dataset and task didn't match. Ultimately, the tool was apparently scrapped because it made selections "almost at random". Which, again, isn't exactly surprising in light of the absolutely bonkers choice of training examples. [1] https://www.aclu.org/blog/womens-rights/womens-rights-workplace/why-amazons-automated-hiring-tool-discriminated-against https://www.aclu.org/blog/womens-rights/womens-rights-workpl... [2] https://www.ml.cmu.edu/news/news-archive/2018/october/amazon-scraps-secret-artificial-intelligence-recruiting-engine-that-showed-biases-against-women.html https://www.ml.cmu.edu/news/news-archive/2018/october/amazon...
- bumby 7y ago
  Or couldn't they provide data augmentation on the same samples to give the effect of a more diverse (and more populous) training set? Using the blog's skin cancer example, couldn't the labelled images be augmented by altering the skin tones and adding these new examples to the training set? It seems to me that some of the anomalous results discussed in the article are actually the result of poor model design or poor pre-processing data choices. We can't just throw anything to any ol' machine learning model and expect it to be magic
eanzenberg 7y ago
>>Now, suppose that 75% of the bad turbines use a Siemens sensor and only 12% of the good turbines use one (and suppose this has no connection to the failure). The system will build a model to spot turbines with Siemens sensors. Oops. Given a statistically large enough sample, 2 outcomes: 1) The Siemens sensor actually is at fault. 2) The Siemens sensor is a part of a larger system, which is different in non-Siemens turbines, and that system is failing. Either way, the model prediction on turbine failures is enhanced with that Siemens feature. But to even get to this granularity, you are diving into model explainability, or what features were important for each prediction. Here, you try to understand the black-box to find reasons for particular input->output.
- TheCoelacanth 7y ago
  Only if your test data is free of sample bias. Given how incredibly hard it is to avoid sample bias, you can't take it for granted that your training data doesn't have any sample bias.
  - DuskStar 7y ago
    If the sample is "all the gas turbines I own", I don't particularly CARE about the bias...
    - chobeat 7y ago
      you should, because you might make worse decisions for the business, for the system or for the people that are impacted by the system. If you don't have the right data to decide, don't decide using the data.
    - TheCoelacanth 7y ago
      If the training data is all gas turbines that you own, why do you care about having the ML model at all? You already have complete knowledge of the state of all your gas turbines. There's no point to having an ML model unless you are applying it to something outside of the training data. If you plan on applying the model to different turbines, then there is potential for sample bias in which turbines you selected. If you apply it to the same turbines at some point in the future, then you sampled points in time so there is a potential for sample bias based on which points in time you selected. There is no way of completely avoiding the potential for sample bias unless you completely abandon ML as a useful concept.
Zolomon 7y ago
There is a course on this at New York University: https://dataresponsibly.github.io/courses/spring19/ https://dataresponsibly.github.io/courses/spring19/
twa927 7y ago
> Until about 2013, If you wanted to make a software system that could, say, recognise a cat in a photo, you would write logical steps. You’d make something that looked for edges in an image, and an eye detector, and a texture analyser for fur, and try to count legs, and so on, and you’d bolt them all together... I'm doing a lot of such algorithms (well, not for images). Does someone know if such algorithms have a name? I'm calling it "heuristics" and I think it falls under "AI".
- layoutIfNeeded 7y ago
  I would call it “classical” machine learning.
  - twa927 7y ago
    Hmm, I think there's no "machine learning" here. There's a human hard-coding some thought process, using mostly some simple statistics/thresholds to e.g. define what a "fur texture" looks like.
    - pedrosorio 7y ago
      Machine learning was extensively used in image processing before 2013 / deep learning. The main difference is that you’d write code to extract features from the image and then learn a model using those features (as opposed to using the pixel data directly and learning a model from that as in CNNs). As an example, you wouldn’t necessarily write code for “fur texture” but instead would extract histograms of pixel brightness gradients and feed those (along with other things) to a machine learning algorithm. In this example, fur texture would generate a different histogram (to be used as a feature) than skin texture. https://en.m.wikipedia.org/wiki/Histogram_of_oriented_gradients https://en.m.wikipedia.org/wiki/Histogram_of_oriented_gradie...
      - twa927 7y ago
        Ok, so this depends on what algorithms are used for the feature detection ("edges in an image, and an eye detector, and a texture analyser for fur"). I'm guessing hand-coding an algorithm for detecting edges in an image can be done successfully, but it looks much harder for "an eye detector", so it needs "machine learning" What I meant when asking for a name of an algorithm class are algorithms where the feature extraction is done using hand-coded algorithms.
chobeat 7y ago
I've just added this post to my reading list. I share it if anybody is interested in this and similar topics: https://github.com/chobeat/awesome-critical-tech-reading-list/ https://github.com/chobeat/awesome-critical-tech-reading-lis...
Someone 7y ago
Short way to describe the problem: we want to build systems that detect causation, but statistical models can only detect correlation.
- wongarsu 7y ago
  That's not entirely true: it's hard to show causation, but with enough data you can. If A correlates with B you know that either A causes B, B causes A, some C causes both A and B, or the correlation is a coincidence. If you have the data to rule out 3 of those the remaining possibility is the causation.
  - Someone 7y ago
    So, how do you, for example, rule out “some C causes both A and B“, if you may not even know of the existence of C? More importantly, the only way to really show causation is by positing a mechanism.
gambler 7y ago
>The most obvious and immediately concerning place that this issue can be manifested is in human diversity. I swear, when someone starts building autonomous killer robots, the first set of concerned articles will probably be asking whether robots were properly trained to target all genders and races with equal accuracy. This is not a sensible way to approach AI ethics. >It was recently reported that Amazon had tried building a machine learning system to screen resumés for recruitment. Since Amazon’s current employee base skews male, the examples of ‘successful hires’ also, mechanistically, skewed male and so, therefore, did this system’s selection of resumés. There is nothing "mechanistic" about this. It depends on how you select sample resumes and how you split them between "good" and "bad" labels. I worked on a similar thing as an "encouraged" side-project at a certain company. Except I realized from day 1 that using AI on resumes is a bad idea and aimed to show this with data. My model was aiming to detect people who will quit or get fired within first 6 month (with the intent of lowering them in priority for interviews, supposedly). It miraculously achieved 85% accuracy... by figuring out how to detect summer interns. Framing this problem as "bias" and especially hyper-focusing everyone's attention on diversity aspect of it is extremely irresponsible. (I'm not saying that's what the author is doing, but that's definitely what's being done at large.) Fundamentally, there are significant higher-level problems with using statistical ML models for things like hiring or crime prediction.
- Bartweiss 7y ago
  That intern story is excellent; I'm adding it to my bank of "weird AI tricks" like pausing Tetris to avoid losing. More topically, you're quite right to object to that Amazon reference. As far as I can tell, the real story is even worse than mislabeling. Amazon devs wanted a system to spot candidates in resume banks, so they trained it to recognize resumes similar to the ones submitted to Amazon in the past. The entire dataset was 'positive', and output degrees of similarity instead of classifications. Amazon applicants are mostly male while the pool was presumably 50/50, so that was learned as an element of "Amazon-candidate-ness". That's also an interesting story, but from the first publication (in Reuters) it's been framed as an uneven base rate 'inevitably/predictably/mechanistically' producing a biased result. Which is not only untrue but downright backwards, since it implies that the rate in the general data is what matters, rather than the relative rate between samples and positive classifications. It's yet another variant of the mammogram base rates question, and I wish people would stop trying to reinforce the incorrect answer to that.
jgon 7y ago
This quote stands out to me: "just as a dog is much better at finding drugs than people, but you wouldn’t convict someone on a dog’s evidence. And dogs are much more intelligent than any machine learning." Because in my head I followed it with the sentence "but we're all confident that we will have dogs driving our cars in about 5 years." Food for thought for sure.
- dmix 7y ago
  So dogs are better than humans at detecting drugs because they have a better sense of smell than can penetrate packaging. What does that have to do with technology being better/worse than humans at driving, exactly? They didn't say dogs were better than technology at solving problems, in any sort of general sense.
killjoywashere 7y ago
I actually think this is where ML really shines. You can pick things apart. Sure, you might need carefully designed experiments, but you can subtract "female" from the resume and look for other data that cause some trained machine to activate, like patterns of word choice, etc. This is akin to the Go players learning from Alpha Go. It's actually a richly rewarding investigation for those of us who have done it. To discover a whole class of failure modes, that's success! And, unlike courts of law, the the process is much more efficient, because you don't have to contend with a defendant appealing to matters of intent or the emotions of a jury.

146 points by andrevoget 7y ago

> Since Amazon’s current employee base skews male, the examples of ‘successful hires’ also, mechanistically, skewed male and so, therefore, did this system’s selection of resumés. Amazon spotted this and the system was never put into production. Couldn't they have retrained the system with a 50/50 mix of males / females resumes ? Or restrict the use of the algorithm to sort male resumes ? Or maybe resumes don't actually correlate at all with success in Amazon ...

deogeo 7y ago
Sounds like an extraordinarily poor AI system if it depends on absolute numbers, and not per capita. And wouldn't the number of unsuccessful hires also skew male?
- theoh 7y ago
  Consider the possibility that the (pre-AI system) probability of success for a female applicant is the same as the probability of success of a male applicant. You could make a "per capita" quota as a kind of goal. That's not a problem, but how would you make sure the quota was met? The typical AI system doesn't work on the basis of selecting candidates entirely at random, pro rata, in order to meet a quota. It works on the basis of criteria for success. One thing it might learn (unfortunately) is that most posts at the company are filled by men.
  - cygaril 7y ago
    From a machine learning point of view, one can just add the constraint that the probability of being in the "yes" bucket is that same for both male and female candidates. Doing this will give a worse fit than an unconstrained optimization, but it is fairer. More sophisticated approaches are possible.
    - theoh 7y ago
      There's no "just" to any aspect of this topic. I think what you are talking about is what is sometimes called "classification parity", and there are problems with it, and with everything else we've come up with to combat bias. https://arxiv.org/abs/1808.00023 https://arxiv.org/abs/1808.00023
- jerf 7y ago
  "Sounds like an extraordinarily poor AI system if it depends on absolute numbers, and not per capita." To some extent, you're bringing in your human bias to prefer human biases when you make that statement. We humans have a hierarchy of important attributes, and for various reasons believe race and gender are more important than eye color or height. But the machine learning algorithm just gets a multidimensional point in hyperspace. It doesn't, a priori, "know" that it needs to do a "per capita" adjustment based on FIELD_1 any more than it knows it needs to do a per capita adjustment on FIELD_2. And you can't "adjust" on all the fields because that'll just cancel out. We are also in the weird position of wanting the machine to do adjustments based on FIELD_1, but without us having to actually admit to ourselves that we're doing it. From a technical perspective, probably the best answer is to do a straight-up training based on the data, then have an cleanly-separated after-the-fact cleanup process to perform whatever social adjustments it is we want on the outcome. But nobody is willing to admit that's what we want, and to put those adjustments down on paper in the form of code, because the instant they're concrete, pretty much everybody is going to decide they're wrong, and no two people are going to agree on the manner in which they are wrong, and an epic, national-front-page-news shitstorm will ensue. So here we are, trying to make adjustments without making adjustments, or, alternatively, trying to make adjustments in a place where we can blame the AI rather than humans. (The ironic thing is that because we can't admit what we're trying to do, we're going to end up doing a really poor job of it. Tools will be applied haphazardly, the results can't be measured except very grossly at the very end of the process, and the goals won't be obtained and the system is always going to be quirky and weird. If we could clearly declare what it is we actually wanted, it would be fairly easy to get it from the AIs.)
- Bartweiss 7y ago
  The basic "resumes skewed male so the algorithm did too" explanation appears to be incorrect. But it's found in the original Reuters story and most derived stories, and finding it here implies it's reached the level of urban legend. Going by the details of the Reuters story and several others, it appears that what actually happened was a training/task mismatch. Amazon wanted an algorithm to do resume discovery, which recruiters would run and get quality predictions as they viewed resumes. But they trained it on resume results, giving it past resumes which had been submitted to Amazon and telling it to seek similar resumes. None of the stories make it clear if there even was negative training data; it looks like the tool was simply told to compute degree-of-similarity to past inputs, and possibly told to prioritize resumes which were ultimately hired. As a result, the tool was trying to convert a relatively gender-neutral pool (resumes found online) to a skewed one (Amazon applicant resumes), and did so by weighting gendered terms. It also seems to have underweighted technical terms, failing to appreciate them as mandatory or strictly position-specific. The developers were sufficiently aware of that to catch and correct the known gender biases (e.g. devaluing women's colleges or the literal word "women's"), but were scared there were other uncaught biases. And the results were apparently terrible all around, so the tool was scrapped. Which is pretty much what you'd expect from something trained on exclusively positive, sample-biased examples. The story has been seriously distorted, but the real plan also seems terrible...
duxup 7y ago
It does make you wonder how much AI will be .. AI and how much guidance for desired outcomes humans will give it. Humans are pretty happy to create nonsensical results if it fits their goals... especially if it befits them. I wonder if with AI we do that to the point that it is somewhat irrelevant.
TheRealPomax 7y ago
maybe, but this might also just be someone unwilling to commit to the sunk cost fallacy. You can spend time and money fixing it, or you can cut your losses and just stop trying to automate something that probably didn't need full automation to begin with.
FakeComments 7y ago
If the only problem is that it mixes up male and female resumes, why not have it sort each into two piles and have a human recruiter perform the merge by picking the better applicant from the pair on top, repeatedly? I have the suspicion that Amazons AI was factually correct, but politically wrong.
- TheCoelacanth 7y ago
  I highly doubt that the preference for male candidates was the only problem with the AI. The preference for candidates with ice hockey on their resume almost certainly also would have resulted in a preference for white candidates.
  - FakeComments 7y ago
    Sure, my point is why does that matter? The correlation with ice hockey could be a career relevant detail because of correlation with discipline and pain tolerance. If it tracks all such signals, I don’t see the problem. That it also has a circumstantial correlation with races isn’t inherently problematic — it could be that different groups of people are differently qualified. There seems to be a position that culture is uncorrelated to job success, and that’s just nonsense. When people can show me these biases are correlated to race or sex once controlling for culture, then we have a problem.
    - TheCoelacanth 7y ago
      Good luck with that argument in court.
    - michaelmior 7y ago
      Race and culture are two very different things. There certainly can be some correlation, but drawing conclusions of suitability for employment based on race is a problem.
      - FakeComments 7y ago
eanzenberg 7y ago
That wouldn't matter if the KPI (worker performance) being predicted, which is inherently biased as well, was distributed differently among the balanced pool of applicants.
munchbunny 7y ago
I think this retelling of the story is over-simplified. It's a compelling story, but I don't know any competent engineers who give up on a whole project because of one setback. If this system never saw production use, it was because it's still not ready, or there were many other issues that aren't mentioned that led the team to give up, or because political winds shifted. Amazon is famous for killing projects quickly.
manfredo 7y ago
The whole problem with allegations of AI bias is that people often point to disparities of outcome as proof of bias. The reality is that there are plenty of disparities on outcome regardless of bias, and the allegations of bias and attempt to rectify the alleged bias is another vector for the introduction of bias.
DuskStar 7y ago
One situation I could see leading to this result (Amazon cancelling their resume filtering software with the excuse that it 'skewed male') is that 1. The AI system accurately predicted employee success across both genders AND 2. The AI system predicted that women would do worse than men That's politically embarrassing and something that you can't necessarily 'fix' by improving the system. (see: all the 'will this person commit a crime if let out on parole' systems that end up accurately discriminating based on race) This isn't to say that women are worse engineers than men, or anything of that sort - only that the applicant pool to Amazon was skewed, or women were treated worse in the workplace and thus performed worse, or a dozen other possible causes. (And only in this hypothetical scenario! I have no inside info from Amazon!)
- gizmo686 7y ago
  Your example is quite possible, particularly at an organazation that would be embarrased by such a result. Assume that the ability curve of male applicants and female applicants are identical; that the majority of applicants are male; and that Amazon wants to hire more females then would be expected given the portion of applicants that are female. A natural way of accomplishing this goal is to give extra points to female applicants [0]. Due to selection bias, the ability curve of women within the population of Amazon engineers would skew lower then men within the population of Amazon engineers. This is a special case of a more general phenomona. If you have signal S that is positivly correlated with a desired trait in the general population, and over select for S, you will find that S is negativly correlated within your population. [0]. All proposals I have seen amount to either a good approximation of this or changing the applicant pool. And, by assumption, the latter is excluded.
- lotu 7y ago
  This is feels like an elephant in the room when it comes to AI bias. We develop an AI that accurately predicts outcomes and discover it is biased, then instead of asking if maybe this means our current system is deeply biased and needs to be changed, we say, "don't use the AI; keep using the people who might or might not be biased but we don't know because we can't measure it in the way an AI can be measured." If it isn't acceptable to use an AI to create biased outcomes how is it acceptable to use people to create the the same outcomes. AI decision making can be examined and tuned in ways that people cannot.
turtlecloud 7y ago
Just remove the gender/sex as variables for the AI and maybe name too. Preprocess the resumes to remove them. Now you remove the majority of gender bias for the AI.
- gizmo686 7y ago
  AI is really good at infering information. If gender is a real signal, it would be very difficult to filter the input such that it is not making a determination by what could be reffered to inferred gender.
Bartweiss 7y ago
This story has been constantly misrepresented, because Reuters absolutely botched their initial report. Amazon was never building a tool to decide which interviewed candidates to hire, they were building a tool for discovering candidates. It was biased, but that gender bias wasn't the proximate reason for scrapping the tool. As far as I can tell from later stories (e.g. 1, 2), what Amazon actually did was build a tool to show recruiters 'quality' predictions for all resumes, for instance as they scrolled LinkedIn. But they trained it on resumes submitted to Amazon for various positions, possibly also adding weight to resumes which produced hires. In which case the problem is painfully obvious; the system effectively had no negative training data, and its positive examples (submitted resumes) didn't actually match the desired output (qualified resumes). It was computing degree of similarity between a gender-neutral-ish pool (resumes posted online) and a gender-skewed pool (resumes submitted to Amazon), and tried to make that conversion with whatever data was available - like devaluing resumes that mentioned women's colleges. (This wasn't just a proxy-variable thing, the model essentially learned to weight on gender.) Amazon's team apparently caught this issue and did the usual things like blinding on those words. But they were scared of uncaught factors; reading between the lines, they were unable to "detrain" biases like neural nets do because their dataset and task didn't match. Ultimately, the tool was apparently scrapped because it made selections "almost at random". Which, again, isn't exactly surprising in light of the absolutely bonkers choice of training examples. [1] https://www.aclu.org/blog/womens-rights/womens-rights-workplace/why-amazons-automated-hiring-tool-discriminated-against https://www.aclu.org/blog/womens-rights/womens-rights-workpl... [2] https://www.ml.cmu.edu/news/news-archive/2018/october/amazon-scraps-secret-artificial-intelligence-recruiting-engine-that-showed-biases-against-women.html https://www.ml.cmu.edu/news/news-archive/2018/october/amazon...
bumby 7y ago
Or couldn't they provide data augmentation on the same samples to give the effect of a more diverse (and more populous) training set? Using the blog's skin cancer example, couldn't the labelled images be augmented by altering the skin tones and adding these new examples to the training set? It seems to me that some of the anomalous results discussed in the article are actually the result of poor model design or poor pre-processing data choices. We can't just throw anything to any ol' machine learning model and expect it to be magic

eanzenberg 7y ago

>>Now, suppose that 75% of the bad turbines use a Siemens sensor and only 12% of the good turbines use one (and suppose this has no connection to the failure). The system will build a model to spot turbines with Siemens sensors. Oops. Given a statistically large enough sample, 2 outcomes: 1) The Siemens sensor actually is at fault. 2) The Siemens sensor is a part of a larger system, which is different in non-Siemens turbines, and that system is failing. Either way, the model prediction on turbine failures is enhanced with that Siemens feature. But to even get to this granularity, you are diving into model explainability, or what features were important for each prediction. Here, you try to understand the black-box to find reasons for particular input->output.

TheCoelacanth 7y ago
Only if your test data is free of sample bias. Given how incredibly hard it is to avoid sample bias, you can't take it for granted that your training data doesn't have any sample bias.
- DuskStar 7y ago
  If the sample is "all the gas turbines I own", I don't particularly CARE about the bias...
  - chobeat 7y ago
    you should, because you might make worse decisions for the business, for the system or for the people that are impacted by the system. If you don't have the right data to decide, don't decide using the data.
  - TheCoelacanth 7y ago
    If the training data is all gas turbines that you own, why do you care about having the ML model at all? You already have complete knowledge of the state of all your gas turbines. There's no point to having an ML model unless you are applying it to something outside of the training data. If you plan on applying the model to different turbines, then there is potential for sample bias in which turbines you selected. If you apply it to the same turbines at some point in the future, then you sampled points in time so there is a potential for sample bias based on which points in time you selected. There is no way of completely avoiding the potential for sample bias unless you completely abandon ML as a useful concept.

Zolomon 7y ago

There is a course on this at New York University: https://dataresponsibly.github.io/courses/spring19/ https://dataresponsibly.github.io/courses/spring19/

twa927 7y ago

> Until about 2013, If you wanted to make a software system that could, say, recognise a cat in a photo, you would write logical steps. You’d make something that looked for edges in an image, and an eye detector, and a texture analyser for fur, and try to count legs, and so on, and you’d bolt them all together... I'm doing a lot of such algorithms (well, not for images). Does someone know if such algorithms have a name? I'm calling it "heuristics" and I think it falls under "AI".

layoutIfNeeded 7y ago
I would call it “classical” machine learning.
- twa927 7y ago
  Hmm, I think there's no "machine learning" here. There's a human hard-coding some thought process, using mostly some simple statistics/thresholds to e.g. define what a "fur texture" looks like.
  - pedrosorio 7y ago
    Machine learning was extensively used in image processing before 2013 / deep learning. The main difference is that you’d write code to extract features from the image and then learn a model using those features (as opposed to using the pixel data directly and learning a model from that as in CNNs). As an example, you wouldn’t necessarily write code for “fur texture” but instead would extract histograms of pixel brightness gradients and feed those (along with other things) to a machine learning algorithm. In this example, fur texture would generate a different histogram (to be used as a feature) than skin texture. https://en.m.wikipedia.org/wiki/Histogram_of_oriented_gradients https://en.m.wikipedia.org/wiki/Histogram_of_oriented_gradie...
    - twa927 7y ago
      Ok, so this depends on what algorithms are used for the feature detection ("edges in an image, and an eye detector, and a texture analyser for fur"). I'm guessing hand-coding an algorithm for detecting edges in an image can be done successfully, but it looks much harder for "an eye detector", so it needs "machine learning" What I meant when asking for a name of an algorithm class are algorithms where the feature extraction is done using hand-coded algorithms.

chobeat 7y ago

I've just added this post to my reading list. I share it if anybody is interested in this and similar topics: https://github.com/chobeat/awesome-critical-tech-reading-list/ https://github.com/chobeat/awesome-critical-tech-reading-lis...

Someone 7y ago

Short way to describe the problem: we want to build systems that detect causation, but statistical models can only detect correlation.

wongarsu 7y ago
That's not entirely true: it's hard to show causation, but with enough data you can. If A correlates with B you know that either A causes B, B causes A, some C causes both A and B, or the correlation is a coincidence. If you have the data to rule out 3 of those the remaining possibility is the causation.
- Someone 7y ago
  So, how do you, for example, rule out “some C causes both A and B“, if you may not even know of the existence of C? More importantly, the only way to really show causation is by positing a mechanism.

gambler 7y ago

>The most obvious and immediately concerning place that this issue can be manifested is in human diversity. I swear, when someone starts building autonomous killer robots, the first set of concerned articles will probably be asking whether robots were properly trained to target all genders and races with equal accuracy. This is not a sensible way to approach AI ethics. >It was recently reported that Amazon had tried building a machine learning system to screen resumés for recruitment. Since Amazon’s current employee base skews male, the examples of ‘successful hires’ also, mechanistically, skewed male and so, therefore, did this system’s selection of resumés. There is nothing "mechanistic" about this. It depends on how you select sample resumes and how you split them between "good" and "bad" labels. I worked on a similar thing as an "encouraged" side-project at a certain company. Except I realized from day 1 that using AI on resumes is a bad idea and aimed to show this with data. My model was aiming to detect people who will quit or get fired within first 6 month (with the intent of lowering them in priority for interviews, supposedly). It miraculously achieved 85% accuracy... by figuring out how to detect summer interns. Framing this problem as "bias" and especially hyper-focusing everyone's attention on diversity aspect of it is extremely irresponsible. (I'm not saying that's what the author is doing, but that's definitely what's being done at large.) Fundamentally, there are significant higher-level problems with using statistical ML models for things like hiring or crime prediction.

Bartweiss 7y ago
That intern story is excellent; I'm adding it to my bank of "weird AI tricks" like pausing Tetris to avoid losing. More topically, you're quite right to object to that Amazon reference. As far as I can tell, the real story is even worse than mislabeling. Amazon devs wanted a system to spot candidates in resume banks, so they trained it to recognize resumes similar to the ones submitted to Amazon in the past. The entire dataset was 'positive', and output degrees of similarity instead of classifications. Amazon applicants are mostly male while the pool was presumably 50/50, so that was learned as an element of "Amazon-candidate-ness". That's also an interesting story, but from the first publication (in Reuters) it's been framed as an uneven base rate 'inevitably/predictably/mechanistically' producing a biased result. Which is not only untrue but downright backwards, since it implies that the rate in the general data is what matters, rather than the relative rate between samples and positive classifications. It's yet another variant of the mammogram base rates question, and I wish people would stop trying to reinforce the incorrect answer to that.

jgon 7y ago

This quote stands out to me: "just as a dog is much better at finding drugs than people, but you wouldn’t convict someone on a dog’s evidence. And dogs are much more intelligent than any machine learning." Because in my head I followed it with the sentence "but we're all confident that we will have dogs driving our cars in about 5 years." Food for thought for sure.

dmix 7y ago
So dogs are better than humans at detecting drugs because they have a better sense of smell than can penetrate packaging. What does that have to do with technology being better/worse than humans at driving, exactly? They didn't say dogs were better than technology at solving problems, in any sort of general sense.

killjoywashere 7y ago

I actually think this is where ML really shines. You can pick things apart. Sure, you might need carefully designed experiments, but you can subtract "female" from the resume and look for other data that cause some trained machine to activate, like patterns of word choice, etc. This is akin to the Go players learning from Alpha Go. It's actually a richly rewarding investigation for those of us who have done it. To discover a whole class of failure modes, that's success! And, unlike courts of law, the the process is much more efficient, because you don't have to contend with a defendant appealing to matters of intent or the emotions of a jury.