8 ms·

Algorithmic Monocultures in Hiring

154 points by sizzle 3mo ago

https://algorithmichiring.github.io/ https://algorithmichiring.github.io/ https://arxiv.org/abs/2605.27371 https://arxiv.org/abs/2605.27371

xdennis 3mo ago
It's surprising to me to hear that these systems are considered racist when they're the same ones that are so color blind that they generate pictures of SS soldiers as African American women.
roysting 3mo ago
There is no isolation of variables. This is not science. This is propaganda.
anonfunction 3mo ago
This is something I've been working on exposing to AI labs through my startup LatentEvals[1], and found similar results in other industries from lending to insurance claims. Happy to share some sample reports if anyone is interested! 1. https://www.latentevals.com/ https://www.latentevals.com/
- etchalon 3mo ago
  Don't have much to add beyond being grateful for everyone working to call this out, with a hope some lawsuits drop and our SCOTUS doesn't decide racial bias in AI is fine because we can't prove the AI is racist in its heart.
alain94040 3mo ago
The European Union passed The Artificial Intelligence Act, which classifies: High-risk – AI applications that are expected to pose significant threats to health, safety, or the fundamental rights of persons. Notably, AI systems used in health, education, recruitment, critical infrastructure management, law enforcement or justice. They are subject to quality, transparency, human oversight and safety obligations That's a pretty common sense legislation to me.
- anon373839 3mo ago
  The AI “safety” industry is lobbying for federal preemption so that states won’t have the power to enact these types of sensible regulations.
  - pc86 3mo ago
    > > European
    - anon373839 3mo ago
      Right, I'm saying that these are sensible regulations and Anthropic et al want to prevent US states from being able to pursue them.
- 72027372920 3mo ago
  [dead]
- pc86 3mo ago
  This is one of those things where the first sentence sounds completely fine and reasonable, maybe even objectively good. Of all the things listed "recruitment" doesn't belong to me. Is the argument that it is someone's fundamental human right to get someone else to pay them to do a job? Or is it strictly about human oversight?
  - tadfisher 3mo ago
    Friend, we are discussing a demonstration of systemic bias in hiring decisions made by AI models. The argument is that it is someone's fundamental right to be treated the same as someone else with the same qualifications but different skin color.
- tbrownaw 3mo ago
  > That's a pretty common sense legislation to me. There's no reason to single out AI vs any other approach to the same topics.
asdff 3mo ago
Some job application websites I've seen actually have a yes or no option to consent to AI review that they claim is to simply assist HR and not actually screen you. I always select no. There is no way that selecting yes would ever be in my interest. I'm sorry, I'm going to force a real human to look at my stuff if I still can.
- bluefirebrand 3mo ago
  My fear is that pressing "no" on stuff like that is going to become an auto-rejection in the vast majority of cases
  - simpaticoder 3mo ago
    It won't be rejected. Your resume will be meticulously placed into a human review queue pending the allocation of someone to look at the contents. Meanwhile the position will be filled, and so serving no purpose the review queue will be emptied.
    - bluefirebrand 3mo ago
      Oddly enough, being rejected by process versus being rejected by a person doesn't actually make me feel any better about the coming future :)
  - jcims 3mo ago
    It's probably not going to be an auto-rejection, it's just going to sit in a queue that looks like this Screened Applications [13] Unscreened Applications [39148]
    - bluefirebrand 3mo ago
      Yeah, I know My point is that this is effectively an auto rejection
- booleandilemma 3mo ago
  [dead]
everyone 3mo ago
Its fucking crazy that people are using these systems for important tasks like hiring. They have zero understanding about how these systems work. And LLMs are absolutely not designed to do those sorts of jobs, they're designed to be chatbots and to fool a human conversing them that they are responding intelligently. Of course they're gonna be useless at other tasks. (I assume they're just using a big LLM for this, it doesnt say, it just says "AI" when they say "AI like that they usually mean LLM".. A custom trained hiring ML system would be better)
- engineer_22 3mo ago
  Isn't HR basically just an LLM with ears and teeth?
anonreeeeplor 3mo ago
[dead]
bakugo 3mo ago
> To put this in perspective: If the AI had recommended Black and Asian candidates at the same rate as it recommended the most-favored group (typically white applicants) Some people just can't help but put their biases on display at every opportunity, even when it comes to the most minute details.
- moate 3mo ago
  Where do you think this sentence shows bias? The phrase "most-favored" means, "most recommended by the AI relative to the field". What did you think this sentence meant?
  - redsocksfan45 3mo ago
    [dead]
- gacgacgac 3mo ago
  Nothing in this has any bias in it? Which words are you suggesting are biased? This study measured constructed resumes where only names were changed, and observed the rate each group was favored (the percentage of resumes that passed). One group must be "most favored" because thats how math works. It's the group whose percentage was the highest. The resumes were fictional and equivalent across race, only the names were changed.
  - bakugo 3mo ago
    Look closer at the capitalization of the words in the quoted sentence.
x313 3mo ago
This study only looks at one specific vendor algorithmn (a job assesment given by a company called pymetrics)
- all2 3mo ago
  LLMs are trained on the Internet, which isn't exactly known for it's race agnostic opinions.
logicchains 3mo ago
Could the AI actually see the race of the applicants? Or was it just discriminating on the basis of some factor it found that was correlated with race, like SAT scores?
- gacgacgac 3mo ago
  Name. Other factors were controlled.
  - moate 3mo ago
    Where was this listed in the study? I can't find this anywhere in either the linked page or the Github https://algorithmichiring.github.io/ https://algorithmichiring.github.io/
  - deleted 3mo ago
    [deleted]
- foolserrandboy 3mo ago
  It rejected Asians more because of their higher SAT scores? If it’s not directly based on applicants disclosing their ethnicity then probably something more obvious like names.
- moate 3mo ago
  I'm going to assume that people aren't allowed to put "don't send me black applicants" into their process even if they do see race in the application as that's entirely illegal. The paper's conclusion, that we need to study this more, is showing the authors likely believe this to be a byproduct of inherent/invisible bias.
- runako 3mo ago
  > discriminating on the basis of some factor it found that was correlated with race, like SAT scores Hypothetical SAT score: 1060 How does that help you predict the race of an individual applicant? It's been a while since I took the SAT, but I didn't realize one's score provided so much information.
dash2 3mo ago
> To measure adverse impact, we apply the EEOC’s “four-fifths rule,” which flags a position when one group is recommended at less than 80% of the rate of the most-recommended group That seems like a nonsensical way to measure racial discrimination. What could justify it?
- nemomarx 3mo ago
  I guess it measures if there's more than one std deviation gap between highest and lowest? Assuming that's twenty percent here it sounds like how you'd get that kind of metric at least
- moate 3mo ago
  It's a starting point to flag. Here's some analysis of what it is and why it's useful as a canary in the coal mine: https://www.prevuehr.com/resources/insights/adverse-impact-analysis-four-fifths-rule/ https://www.prevuehr.com/resources/insights/adverse-impact-a...
  - dash2 3mo ago
    Thanks. I read the article: > Since the 80% test does not involve probability distributions to determine whether the disparity is a “beyond chance” occurrence, it is usually not regarded as a definitive test for adverse impact. Instead, other statistically significance tests, such as the standard deviation analysis, may be used for this purpose. But then my question recurs: isn’t this a ridiculous way to measure discrimination? It’s assuming that the only thing that differs between the different ethnic applicant pools is their ethnicity, which is essentially never going to be true.
    - gacgacgac 3mo ago
      It's not used to measure discrimination. It's used to identify outcomes that appear to be potentially discriminatory. You have to do the legwork afterwards. Like. If I am evaluating a developer on lines of code written, I am a bad manager. But if an engineer has 40% fewer lines of code than the team median, it's absolutely ok for me to go, "Interesting. What's the story there? Are they slower or is there some other factor?" Same idea -- this is purely a fast, first pass metric that can quickly assess if something warrants a deeper evaluation.
groundzeros2015 3mo ago
I don’t think AI screening is effective. But this study is just disparate impact.
engineer_22 3mo ago
> Using our large dataset of real hiring AI recommendations, we test our hypothesis. We find that people who submit multiple applications to positions screened by the same algorithmic hiring vendor are more likely to be rejected from every position to which they apply than would be true if the companies made decisions statistically independently from one another. I would be surprised if the results were different.
verteu 3mo ago
The paper is here: https://arxiv.org/pdf/2605.27371 https://arxiv.org/pdf/2605.27371 They find "disparate impact" of pymetrics across racial groups, but it doesn't seem like they controlled for anything.
- efavdb 3mo ago
  They also say that if they do the analysis globally the effect goes away. Curious, does that not imply that if one domain is biased against some group there would be another where the bias was in its favor?
  - zeroonetwothree 3mo ago
    Also has issues of random chance causing these differences. How many different positions are there that have the chance of a 80% effect?
    - verteu 3mo ago
      They're using a Benjamini–Hochberg correction (alpha=0.05) to account for multiple comparisons (see Table 2).
wand3r 3mo ago
Did I miss the part of the article where they break down how they determined race? Is the algorithm blind to race? It looks like they specifically looked at 83k people applying to ~100 companies which notably were Fortune 500 companies. Could there simply be candidate discrepancies here? Hard for me to follow the full methodology but it doesn't necessarily seem either malicious or that well structured. Don't you need to have a control group of applicants who are similar on paper? To allege DISCRIMINATION is quite bold. Definitely open to opposing or critical views
- gacgacgac 3mo ago
  Yes. You missed it. They are using a test dataset of 83k resumes generated in 2022 for this paper and comparing it as a baseline against their observational data: https://www.nber.org/papers/w29053 https://www.nber.org/papers/w29053 The dataset is constructed, deliberately, to hold candidate performance constant and vary the names of candidates to appear to be associated with a specific race.
  - AStrangeMorrow 3mo ago
    From looking at how that was done, it seems they (the paper you linked) used an older paper which looked at which names are frequent enough and more biased toward a certain demographic (90% of that name occurrence falls within that demographic). But they picked 9 family names per group. Which sounds quite low. And combined that with first names to reach 500 first+last names per group. I wonder how much of the bias we see has to do with the names actually picked versus it being racially motivated (absolutely not denying that this probably is a factor, but might not be the only one). For example, in France there is the national BAC end of high school exam. If you you at the names X grade distribution, and look at the higher “very good” bracket: some names are heavily under-represented (less than 5% of say “Jordan” get that grade) while some are over-represented (35% of “Josephine” get such a grade). The exam is for the most part anonymous, but some names are definitely heavily correlated with lower/higher income groups. So nothing surprising: Josephines tend to come from richer families, thus in average get better education/support, thus better grades. Same thing is true with family names to a smaller extent. So I wonder how much of the bias we see, be it from real persons or the AI has more to do with a class thing than a racial thing. Again those are not neatly separate things, but still
xrd 3mo ago
Would be very interested to see how this affects post-50 workers. That's a protected class and I would imagine an ambulance chasing lawyer would be excited for a class action lawsuit.
black6 3mo ago
I'm struggling to figure out what they're trying to say here in the linked (and very anemic) paper: > 30% of Black applicants apply to at least one position that demonstrates adverse impact against Black applicants. The whole thing reads like a tautology.
- gacgacgac 3mo ago
  You are reading a paper without understanding the language of the paper. Adverse Impact has a specific meaning, and in this case it's specifically meaning that Black candidates were selected only four fifths as often as white candidates when their qualifications were identical. The study is only suggesting that further investigation is warranted.
  - black6 3mo ago
    Thanks. It was unclear reading the article and linked paper. I didn't follow the citation/link trail far enough.
    - rayiner 3mo ago
      Your initial assessment was correct and this part of the post above is incorrect: “when their qualifications were identical.” The paper doesn’t control for identical qualifications.
huflungdung 3mo ago
[dead]
tamimio 3mo ago
You don’t need a complicated study to find out, do it yourself for science. Get a resume, make few different versions but keep the context the same, change the layout (one time education on top other on bottom etc etc), and use different names to signal different backgrounds, and you can extend it to schools too and gender, and send it to the same employers, you will see wonders!! I tried it before, and discrimination is there, I would get one resume rejected quickly and few days later the same company would invite another resume for a screening call. I tried this before and after AI hype, results weren’t that different btw, and that was tested in US and Canada employers only.
ApolloFortyNine 3mo ago
I truly don't doubt it's possible for the AI to be 'racist'. >If the AI had recommended Black and Asian candidates at the same rate as it recommended the most-favored group (typically white applicants), 40,000 more of their applications would have advanced to the next stage of hiring. I don't think this is the right benchmark here, or at least, it would be very interesting if the actual outcome, offer or rejected, was considered at the end.
- gacgacgac 3mo ago
  You are misreading this sentence. This sentence is saying: "Using a constructed dataset of resumes, whose only difference was a name change, we would anticipate a system evaluating on qualifications to produce an equal distribution of candidates across names. Our observed result was highly unequal, and that warrants further investigation."
  - _0ffh 3mo ago
    To me it appears as if the study using the constructed dataset was a completely different one than the one that was concerned with AI. For the AI study real data from "3.4 million people who submit 4 million job applications to 1,700 job postings across 150 employers and 11 industry sectors" was used.
petesergeant 3mo ago
I’m sure (really sure) there are real problems with AI and bias, but this is a weird study that isn’t looking at resumes or anything, it’s looking at how candidates did in some weird psychometric tests.
- gacgacgac 3mo ago
  Double check the link. The study clearly looked at resumes.
  - petesergeant 3mo ago
    I’ve rechecked it, and I still think I’m right. What am I missing? This is the paper under discussion: https://arxiv.org/pdf/2605.27371 https://arxiv.org/pdf/2605.27371
ericol 3mo ago
2 days ago there was another interesting article on the effects of AI in hiring[1] I guess this one just compounds. [1] https://news.ycombinator.com/item?id=48620142 https://news.ycombinator.com/item?id=48620142
jazz9k 3mo ago
We can't take blanket percentages as a reason for racial bias. Were they all equally qualified? Too many of these studies only focus on percentages and the end result is unqualified candidates getting hired from minority groups at the expense of qualified ones.
- gacgacgac 3mo ago
  Please read the study or at least the comments here before jumping to the conclusion. Yes, they used constructed resumes, so the qualifications were exactly the same. And no, literally no one is suggesting this proves racial discrimination. It's applying the four fifths rule, a fast, coarse evaluation that is used to identify if maybe theres worth investigating more for a conclusive evidence of racial discrimination. The authors are saying it's worth doing more research, because in a controlled data set the results appear unbalanced.
  - Oras 3mo ago
    > Please read the study or at least the comments here before jumping to the conclusion. Yes, they used constructed resumes Looks like you didn't read the paper. There are no resumes involved. It is about assessment games.
  - etchalon 3mo ago
    I think you're confusing this specific study with a different study, which did use duplicative resumes, and has been repeated: https://www.aeaweb.org/articles?id=10.1257/0002828042002561 https://www.aeaweb.org/articles?id=10.1257/0002828042002561
GrinningFool 3mo ago
[dead]
ortusdux 3mo ago
Ayres, I., Banaji, M. and Jolls, C. (2015), Race effects on eBay. The RAND Journal of Economics, 46: 891-917. https://doi.org/10.1111/1756-2171.12115 https://doi.org/10.1111/1756-2171.12115 "Cards held by African-American sellers sold for approximately 20% ($0.90) less than cards held by Caucasian sellers, and the race effect was more pronounced in sales of minority player cards."
JuniperMesos 3mo ago
[flagged]
- deleted 3mo ago
  [deleted]
- dzonga 3mo ago
  does your anecdote comprise of the various instances when CVs were discriminated against cz people's names sounded black ? but you want to spew nonsense. every racial group includes its own under-qualified people ! there's no social pressure i.e DEI excuse you wanna give - but just economic agents acting for their own interests
- techblueberry 3mo ago
  > These results are consistent with AI hiring tools being completely racially unbiased, and real-world hiring managers feeling social pressure to hire underqualified black people And so managers are feeling social pressure to hire under qualified Asians as well? I must not be up to date on the latest culture war talking points, because I thought Asians were underrepresented.
  - JuniperMesos 3mo ago
    Yeah, if they themselves are asian. One of the most prevalent complaints about Indian hiring managers in the silicon valley tech industry is that they preferentially hire Indians and push out non-Indians to a tremendous degree, and are often helping Indian hires commit pretty blatant credential fraud.
Oras 3mo ago
Misleading title the paper [0] does not mention any CV screening that might suggest racial or gender bias. It is purely about assessment tool. No AI or LLMs. I'm not saying AI is not biased, but this study does not prove that. [0] https://arxiv.org/pdf/2605.27371 https://arxiv.org/pdf/2605.27371 From the paper: > Fig. 1. The pymetrics process. > Stage 1: Applicants apply to positions. > Stage 2: Applicants are directed to the pymetrics platform to play assessment games. > Stage 3: pymetrics algorithms use applicant gameplay features to recommend 58.2% of applicants per position on average. > Stage 4: Employers decide which applicants to interview or hire, typically rejecting applicants that were not recommended by pymetrics.
jmyeet 3mo ago
Many people seem to think racism begins and ends with using a slur. You can usually get a measure of this by seeing someone's reaction to the statement: > There is no such thing as anti-white racism. If you find yourself wanting to disagree with that then, I'm sorry but you simply don't know what racism is. Racism is pervasive, insidious and systemic. A good example in the hiring space is what's called the "second syllable name problem". Traditionally Afrcian names often stress the second syllable (eg Jamal, Lakisha, Malik, Lashonda). Studies have shown that such names have higher rejection rates in job applications [1]. So if you're wondering about the four-fifths rule, it's because it exposes this kind of bias. It's not proof of bias. It simply means further investigation is required. The problem with AI hiring tools is the logic is opaque. You have no idea why an AI system is rejecting or selecting candidates and you may find it's doing something illegal. Some companies want to hide behind this opaqueness, arguing that if no explicit decision was made then there is no bias. But that's not how system racism works. There are many such signals that correlate with race that if they affect selection rate, it could be a problem. Did you go to an HBCU? Was your high school in a minority-majority area? What about your previous employers? This kind of bias doesn't have to be intentional. [1]: https://www.npr.org/2024/04/11/1243713272/resume-bias-study-white-names-black-names https://www.npr.org/2024/04/11/1243713272/resume-bias-study-...
- junofan 3mo ago
  [flagged]
- kbelder 3mo ago
  > > There is no such thing as anti-white racism. > If you find yourself wanting to disagree with that then, I'm sorry but you simply don't know what racism is. You are saying that if you think anti-white racism can exist, you don't know what racism is. That's obviously ludicrous.
  - etchalon 3mo ago
    There are essentially two definitions of racism at this point. The colloquially version, which means "prejudice based on race" and a second version, which specific groups and people have advocated for, which means something like "structural oppression through cultural and governmental means". It's more complicated than just that, but it's a fairly narrow term for them. So when one person says "there's no such thing as anti-white racism", you hear, "No one's prejudiced against white people for being white!" Obviously, that's ludicrous. But that person is likely using the, I have no idea what to call it, "advocate definition" maybe, definition would which preclude anti-white racism from existing within that narrow definition of racism. So it's a debate where people aren't speaking the same version of a language, convinced each other are uninformed, reactionary or stupid.
tlogan 3mo ago
I am not surprised. AI works by learning patterns. So it will become bias by just learning from factors like education history, schools attended, employment history, ZIP codes, or geographic location. Those 3 factors alone are an easy proxy for race. And if you add names into the equation (if the AI was trained without removing applicant names), the model can become even more bias.
jsemrau 3mo ago
Interesting timing as Workday is facing Discrimination Claims in California doing the same thing. https://www.yahoo.com/news/us/articles/california-judge-upholds-discrimination-claims-202610640.html https://www.yahoo.com/news/us/articles/california-judge-upho...
stevenicr 3mo ago
I expected more information from the article and 'the paper' - I see nothing that shows any system was making a decision on race. How is the race being presented to the AI? All this is showing from what I can see, is that certain groups of people were more often denied a next step in the process - but why? Was the AI going by spelling and grammar? Were there names that were different but the rest of the resume was exactly the same? Were there pictures? There were mentions that the rate of each group may be more prominent in the data when you split apart different types of jobs instead of all jobs in aggregate.. One could read that like it's inferred; that more warehouse jobs are offered to a race and less admin jobs.. but that same would happen if AI was more focused on perfect grammar for one job and it was not as much of a factor for a warehouse job. Also if the people applying for the various jobs were self selecting, acceptance percentages this would skew things based upon which ones were applied / not applied to right? There are so many ways you could draw conclusions like this from data, however correlation is not causation, yet this seems to say it is. I feel this is an important thing to watch, but Stanford may not be the place to trust with 'Policy Recommendations' as it's very unclear there is any proof that 'AI Hiring Tools Yield Racial Bias and Systemic Rejection' from this study and paper. PS - now that I see the HN title did not have the word "can" in it, and the title of the article is actually "Tools Can Yield" - maybe that is less accusing and more noting.
OrvalWintermute 3mo ago
The Pymetrics game is rigged by design: Only 40% self report gender/race no resume data, no education information, degrees, schools, GPA, major, work experience, skills/certifications Zero job qualifications
- zerocrates 3mo ago
  Well, they're only looking at whether the pymetrics gameplay algorithm ML thing recommends the candidate, not any of that other stuff. The outcome they're looking at here isn't whether the people actually got hired, or got passed by other screening layers or anything.
ETH_start 3mo ago
A racially disparate outcome is not evidence of racial bias.
- TheMagicHorsey 3mo ago
  Imagine if they applied this same logic to the NBA draft.
- mstewartgallus 3mo ago
  Correlation does not imply causation but it is really curious how the USA has a history of settler-colonialism, slavery, segregation, eugenics, mass incarceration and imperialism. The USA and the tech industry also literally have edgelord internet Nazis in positions of power right now.
jimmy76615 3mo ago
[flagged]
rnxrx 3mo ago
Genuine curiosity: Is there any speculation as to what these tools are keying on to reject those particular applicants? It seems like it just being the applicant's name is too easy an answer, but I could be overthinking it.
marsven_422 3mo ago
[dead]
jongjong 3mo ago
I think the discrimination aspect is downstream from this fact: > We follow 3.4 million people who submit 4 million job applications to 1,700 job postings across 150 employers and 11 industry sectors. Each job application was assessed by an AI hiring tool built by a single third-party vendor. 3.4 million people applying to just 150 employers... Who are all using just 1 platform. WTF. This is where the discrimination is happening. Why the f do 3.4 million people feel forced to apply to just 150 employers and why the f do all these 150 employers feel forced to use just one platform. WTF.
- zeroonetwothree 3mo ago
  That’s the platform that gave them the data. I don’t think they claim it’s all the applications of this set of people.
  - jongjong 3mo ago
    I realise this but it's still incredible to think because that's about 22k applicants per company. Even if that's just part of each company's total hiring pipeline, it's clear; something's wrong. I don't know how long this study has been running but 22k is a lot of people, even over a year. These companies are too big. That's the problem.
    - mstewartgallus 3mo ago
      Capitalism develops into monopolies pretty inevitably. Economies of scale just make monopolies more profitable. Monopolies also get in bed with the state and do dirty tricks to stamp out competition. But monopoly capitalism has been the case for over a 100 years now.
daft_pink 3mo ago
Anyone who’s done hiring wouldn’t be shocked by this: We find applicants are more likely to be rejected from every position they apply to than would be predicted by the baseline of each position making statistically independent decisions. Obviously a rejected resume is more likely to be rejected by every other employer and an accepted resume is more likely to be accepted by every other employer. Like online dating, most employers are looking for some baseline indicators that you are going to be successful and stable.
- zeroonetwothree 3mo ago
  Yes I don’t understand why this is surprising or problematic at all? Actually the fact that they found this result didn’t hold in a different dataset is especially weird.
- pc86 3mo ago
  > a rejected resume is more likely to be rejected by every other employer This makes sense to me, albeit intuitively and in a way I can't articulate. > an accepted resume is more likely to be accepted by every other employer but this doesn't necessarily follow from the prior for me. Plenty of people get really good jobs and are really successful in them only after dozens or hundreds of rejections with a nearly-identical resume.
  - heylook 3mo ago
    The intuition is that they are not truly independent statistical events. Each trial reveals more information about the underlying "quality" of the resume (for passing this trial, not necessarily real world "quality" of the candidate). We are not rolling dice where each toss is fundamentally unrelated to prior tosses.
  - daft_pink 3mo ago
    If you look at the chart, the systemic rejection rate is only like 5-10%. It’s not a huge impact and it’s just about getting an interview not getting the job, so they could still get rejected. I just think certain resumes will get an interview almost every time in some industries and certain resumes will likely never get an interview almost every time, but the majority of resumes are like you say have different aspects that appeal to one empoyer over another.
kenjackson 3mo ago
I think this partially buries the lede: "As a single hiring vendor comes to dominate screening for an industry, it may be more likely that candidates are shut out." If we move to using just a small number of AI models to help do things like hiring, we will amplify biases and possibly completely lock out portions of the population. We need to be very careful when using AI systems to evaluate people in general -- not because they might be biased (which they might be), but because even a small bias, if used by virtually everyone, can be damning.
- pc86 3mo ago
  If you want to make meaningful change in this avenue you really can't use words like "bias" or "systemic" because anywhere from 49-51% of the population will immediately shut down upon hearing that. Someone can argue (and many do to varying levels of success) that systemic bias doesn't exist, which means this doesn't exist, which means there no problem. However, "this AI model can decide that some subset of people, perhaps random, perhaps not, are simply not hirable for any job" makes sense to most people regardless of political bent.
  - rayiner 3mo ago
    The problem with the term “systemic bias” is that it takes a word that’s about differential treatment and changes the subject to disparate outcomes. For example, the article here shows disparate impact: that different percentages of applications are passed through the AI filter. But it doesn’t show differential treatment of otherwise identical applications based on race.
  - nyrikki 3mo ago
    Capitulation is a bad counterpropaganda tactic, especially with terms that have well defined domain specific meanings. Note that the OP uses "systemic rejection", while the paper does reference bias, it is in the precise meaning of the word.[0] And this is not targeted at the general public. You may want to look into the 1990 GOPAC handout "Language: A Key Mechanism of Control" to understand why some groups would simply just weaponize any term that was substituted. Academic papers need to error on being precise, to be effective, not focused on handling the general public with kids gloves IMHO. Edited to add, listen to Lee Atwater's 1981 Interview on the Southern Strategy for even more context. [0] https://arxiv.org/pdf/2605.27371 https://arxiv.org/pdf/2605.27371
tbrownaw 3mo ago
> We find that people who submit multiple applications to positions screened by the same algorithmic hiring vendor are more likely to be rejected from every position to which they apply than would be true if the companies made decisions statistically independently from one another. Ten percent of applicants who submit four applications are rejected from all the places to which they apply. > Our research also found that this pattern does not appear to be the case in other circumstances. We analyzed data from the largest prior study of hiring decisions, which sent 83,000 applications to 108 Fortune 500 firms during the same time period as our study and did not focus on whether AI was used to make decisions. We found that the rate at which applicants were rejected from every firm they applied to in this data was no higher than what you’d expect if each company decided independently of the others. It sounds like this study was using real-world applicants, and the other study they're comparing against was using synthetic applicants. Consider the chance of being accepted as being composed of signal+bias+noise. Noise is random. Signal is a per-applicant value, and what's meant to be measured. Bias is a per-group value, and an artifact of the measuring process. If acceptance/rejection is independent between positions applied for (as in the synthetic applicant study), that suggests that it's random or composed entirely of noise; ie there is no signal; ie the applicants are all equally qualified. If acceptance/rejection is correlated, that means there is some nonzero amount of (signal+bias). But real-world applicants are not all identical, so there should be some amount of signal. So you can't just assume zero signal in order to infer that there must be bias.
- slashdave 3mo ago
  I think I am confused. A inferior candidate (by skill) is going to be consistently rejected, no?
alexpotato 3mo ago
I went to a state school. I then went on to work for multiple firms that placed a premium on candidates from Ivy League/Top Tier (Stanford/Duke etc) candidates. This taught me that: - Their are pros and cons to any selection criteria. - There are smart people everywhere. One of the smartest people I ever worked for spent several years in prison for drug dealing. He was on par with many of the Managing Directors I've worked for - There was a study where they asked big bank recruiters which school consistently produced people who were excellent employees 2-3 years out from hiring and the answer was Penn State (not my alma mater) - There used to be "manager's choice" hires where managers had 1 slot in a training program where they could select whoever they wanted. Sometimes that was terrible. Sometimes that person was top of their training program. - Smart people are just as capable as creating problems as less intelligent people. Smart people, in some ways, are better at creating problems. Especially if the incentives reward them for creating those problems.
- llmslave 3mo ago
  ivy league advantage even after working on the job is unreal and underestimated
- pc86 3mo ago
  > There used to be "manager's choice" hires where managers had 1 slot in a training program where they could select whoever they wanted. Sometimes that was terrible. Sometimes that person was top of their training program. This seems like a great idea to me if you institute a feedback loop so managers who pick trash eventually lost the option to pick, and managers who pick rock stars eventually get more picks / more responsibility.
Terr_ 3mo ago
> If we pool all of its recommendations together — treating the vendor as one giant hiring process — we don’t find adverse impact. If we look at each position separately, as would be typical in an evaluation of adverse impact, then we expose the adverse impact in many positions. Sounds a bit like Simpson's Paradox [0] [0] https://en.wikipedia.org/wiki/Simpson%27s_paradox https://en.wikipedia.org/wiki/Simpson%27s_paradox
deleted 3mo ago
[deleted]
zombot 3mo ago
And then they complain about algorithmic monoculture in applications. https://news.ycombinator.com/item?id=48620142 https://news.ycombinator.com/item?id=48620142
casey2 3mo ago
So if the AI is selecting the economically optimal person for the job does that mean previous hiring systems were biased in favor of Asian and Black hires? Why would that have been the case?

154 points by sizzle 3mo ago

https://algorithmichiring.github.io/ https://algorithmichiring.github.io/ https://arxiv.org/abs/2605.27371 https://arxiv.org/abs/2605.27371

xdennis 3mo ago

It's surprising to me to hear that these systems are considered racist when they're the same ones that are so color blind that they generate pictures of SS soldiers as African American women.

roysting 3mo ago

There is no isolation of variables. This is not science. This is propaganda.

anonfunction 3mo ago

This is something I've been working on exposing to AI labs through my startup LatentEvals[1], and found similar results in other industries from lending to insurance claims. Happy to share some sample reports if anyone is interested! 1. https://www.latentevals.com/ https://www.latentevals.com/

etchalon 3mo ago
Don't have much to add beyond being grateful for everyone working to call this out, with a hope some lawsuits drop and our SCOTUS doesn't decide racial bias in AI is fine because we can't prove the AI is racist in its heart.

alain94040 3mo ago

The European Union passed The Artificial Intelligence Act, which classifies: High-risk – AI applications that are expected to pose significant threats to health, safety, or the fundamental rights of persons. Notably, AI systems used in health, education, recruitment, critical infrastructure management, law enforcement or justice. They are subject to quality, transparency, human oversight and safety obligations That's a pretty common sense legislation to me.

anon373839 3mo ago
The AI “safety” industry is lobbying for federal preemption so that states won’t have the power to enact these types of sensible regulations.
- pc86 3mo ago
  > > European
  - anon373839 3mo ago
    Right, I'm saying that these are sensible regulations and Anthropic et al want to prevent US states from being able to pursue them.
72027372920 3mo ago
[dead]
pc86 3mo ago
This is one of those things where the first sentence sounds completely fine and reasonable, maybe even objectively good. Of all the things listed "recruitment" doesn't belong to me. Is the argument that it is someone's fundamental human right to get someone else to pay them to do a job? Or is it strictly about human oversight?
- tadfisher 3mo ago
  Friend, we are discussing a demonstration of systemic bias in hiring decisions made by AI models. The argument is that it is someone's fundamental right to be treated the same as someone else with the same qualifications but different skin color.
tbrownaw 3mo ago
> That's a pretty common sense legislation to me. There's no reason to single out AI vs any other approach to the same topics.

asdff 3mo ago

Some job application websites I've seen actually have a yes or no option to consent to AI review that they claim is to simply assist HR and not actually screen you. I always select no. There is no way that selecting yes would ever be in my interest. I'm sorry, I'm going to force a real human to look at my stuff if I still can.

bluefirebrand 3mo ago
My fear is that pressing "no" on stuff like that is going to become an auto-rejection in the vast majority of cases
- simpaticoder 3mo ago
  It won't be rejected. Your resume will be meticulously placed into a human review queue pending the allocation of someone to look at the contents. Meanwhile the position will be filled, and so serving no purpose the review queue will be emptied.
  - bluefirebrand 3mo ago
    Oddly enough, being rejected by process versus being rejected by a person doesn't actually make me feel any better about the coming future :)
- jcims 3mo ago
  It's probably not going to be an auto-rejection, it's just going to sit in a queue that looks like this Screened Applications [13] Unscreened Applications [39148]
  - bluefirebrand 3mo ago
    Yeah, I know My point is that this is effectively an auto rejection
booleandilemma 3mo ago
[dead]

everyone 3mo ago

Its fucking crazy that people are using these systems for important tasks like hiring. They have zero understanding about how these systems work. And LLMs are absolutely not designed to do those sorts of jobs, they're designed to be chatbots and to fool a human conversing them that they are responding intelligently. Of course they're gonna be useless at other tasks. (I assume they're just using a big LLM for this, it doesnt say, it just says "AI" when they say "AI like that they usually mean LLM".. A custom trained hiring ML system would be better)

engineer_22 3mo ago
Isn't HR basically just an LLM with ears and teeth?

anonreeeeplor 3mo ago

[dead]

bakugo 3mo ago

> To put this in perspective: If the AI had recommended Black and Asian candidates at the same rate as it recommended the most-favored group (typically white applicants) Some people just can't help but put their biases on display at every opportunity, even when it comes to the most minute details.

moate 3mo ago
Where do you think this sentence shows bias? The phrase "most-favored" means, "most recommended by the AI relative to the field". What did you think this sentence meant?
- redsocksfan45 3mo ago
  [dead]
gacgacgac 3mo ago
Nothing in this has any bias in it? Which words are you suggesting are biased? This study measured constructed resumes where only names were changed, and observed the rate each group was favored (the percentage of resumes that passed). One group must be "most favored" because thats how math works. It's the group whose percentage was the highest. The resumes were fictional and equivalent across race, only the names were changed.
- bakugo 3mo ago
  Look closer at the capitalization of the words in the quoted sentence.

x313 3mo ago

This study only looks at one specific vendor algorithmn (a job assesment given by a company called pymetrics)

all2 3mo ago
LLMs are trained on the Internet, which isn't exactly known for it's race agnostic opinions.

logicchains 3mo ago

Could the AI actually see the race of the applicants? Or was it just discriminating on the basis of some factor it found that was correlated with race, like SAT scores?

gacgacgac 3mo ago
Name. Other factors were controlled.
- moate 3mo ago
  Where was this listed in the study? I can't find this anywhere in either the linked page or the Github https://algorithmichiring.github.io/ https://algorithmichiring.github.io/
- deleted 3mo ago
  [deleted]
foolserrandboy 3mo ago
It rejected Asians more because of their higher SAT scores? If it’s not directly based on applicants disclosing their ethnicity then probably something more obvious like names.
moate 3mo ago
I'm going to assume that people aren't allowed to put "don't send me black applicants" into their process even if they do see race in the application as that's entirely illegal. The paper's conclusion, that we need to study this more, is showing the authors likely believe this to be a byproduct of inherent/invisible bias.
runako 3mo ago
> discriminating on the basis of some factor it found that was correlated with race, like SAT scores Hypothetical SAT score: 1060 How does that help you predict the race of an individual applicant? It's been a while since I took the SAT, but I didn't realize one's score provided so much information.

dash2 3mo ago

> To measure adverse impact, we apply the EEOC’s “four-fifths rule,” which flags a position when one group is recommended at less than 80% of the rate of the most-recommended group That seems like a nonsensical way to measure racial discrimination. What could justify it?

nemomarx 3mo ago
I guess it measures if there's more than one std deviation gap between highest and lowest? Assuming that's twenty percent here it sounds like how you'd get that kind of metric at least
moate 3mo ago
It's a starting point to flag. Here's some analysis of what it is and why it's useful as a canary in the coal mine: https://www.prevuehr.com/resources/insights/adverse-impact-analysis-four-fifths-rule/ https://www.prevuehr.com/resources/insights/adverse-impact-a...
- dash2 3mo ago
  Thanks. I read the article: > Since the 80% test does not involve probability distributions to determine whether the disparity is a “beyond chance” occurrence, it is usually not regarded as a definitive test for adverse impact. Instead, other statistically significance tests, such as the standard deviation analysis, may be used for this purpose. But then my question recurs: isn’t this a ridiculous way to measure discrimination? It’s assuming that the only thing that differs between the different ethnic applicant pools is their ethnicity, which is essentially never going to be true.
  - gacgacgac 3mo ago
    It's not used to measure discrimination. It's used to identify outcomes that appear to be potentially discriminatory. You have to do the legwork afterwards. Like. If I am evaluating a developer on lines of code written, I am a bad manager. But if an engineer has 40% fewer lines of code than the team median, it's absolutely ok for me to go, "Interesting. What's the story there? Are they slower or is there some other factor?" Same idea -- this is purely a fast, first pass metric that can quickly assess if something warrants a deeper evaluation.

groundzeros2015 3mo ago

I don’t think AI screening is effective. But this study is just disparate impact.

engineer_22 3mo ago

> Using our large dataset of real hiring AI recommendations, we test our hypothesis. We find that people who submit multiple applications to positions screened by the same algorithmic hiring vendor are more likely to be rejected from every position to which they apply than would be true if the companies made decisions statistically independently from one another. I would be surprised if the results were different.

verteu 3mo ago

The paper is here: https://arxiv.org/pdf/2605.27371 https://arxiv.org/pdf/2605.27371 They find "disparate impact" of pymetrics across racial groups, but it doesn't seem like they controlled for anything.

efavdb 3mo ago
They also say that if they do the analysis globally the effect goes away. Curious, does that not imply that if one domain is biased against some group there would be another where the bias was in its favor?
- zeroonetwothree 3mo ago
  Also has issues of random chance causing these differences. How many different positions are there that have the chance of a 80% effect?
  - verteu 3mo ago
    They're using a Benjamini–Hochberg correction (alpha=0.05) to account for multiple comparisons (see Table 2).

wand3r 3mo ago

Did I miss the part of the article where they break down how they determined race? Is the algorithm blind to race? It looks like they specifically looked at 83k people applying to ~100 companies which notably were Fortune 500 companies. Could there simply be candidate discrepancies here? Hard for me to follow the full methodology but it doesn't necessarily seem either malicious or that well structured. Don't you need to have a control group of applicants who are similar on paper? To allege DISCRIMINATION is quite bold. Definitely open to opposing or critical views

gacgacgac 3mo ago
Yes. You missed it. They are using a test dataset of 83k resumes generated in 2022 for this paper and comparing it as a baseline against their observational data: https://www.nber.org/papers/w29053 https://www.nber.org/papers/w29053 The dataset is constructed, deliberately, to hold candidate performance constant and vary the names of candidates to appear to be associated with a specific race.
- AStrangeMorrow 3mo ago
  From looking at how that was done, it seems they (the paper you linked) used an older paper which looked at which names are frequent enough and more biased toward a certain demographic (90% of that name occurrence falls within that demographic). But they picked 9 family names per group. Which sounds quite low. And combined that with first names to reach 500 first+last names per group. I wonder how much of the bias we see has to do with the names actually picked versus it being racially motivated (absolutely not denying that this probably is a factor, but might not be the only one). For example, in France there is the national BAC end of high school exam. If you you at the names X grade distribution, and look at the higher “very good” bracket: some names are heavily under-represented (less than 5% of say “Jordan” get that grade) while some are over-represented (35% of “Josephine” get such a grade). The exam is for the most part anonymous, but some names are definitely heavily correlated with lower/higher income groups. So nothing surprising: Josephines tend to come from richer families, thus in average get better education/support, thus better grades. Same thing is true with family names to a smaller extent. So I wonder how much of the bias we see, be it from real persons or the AI has more to do with a class thing than a racial thing. Again those are not neatly separate things, but still

xrd 3mo ago

Would be very interested to see how this affects post-50 workers. That's a protected class and I would imagine an ambulance chasing lawyer would be excited for a class action lawsuit.

black6 3mo ago

I'm struggling to figure out what they're trying to say here in the linked (and very anemic) paper: > 30% of Black applicants apply to at least one position that demonstrates adverse impact against Black applicants. The whole thing reads like a tautology.

gacgacgac 3mo ago
You are reading a paper without understanding the language of the paper. Adverse Impact has a specific meaning, and in this case it's specifically meaning that Black candidates were selected only four fifths as often as white candidates when their qualifications were identical. The study is only suggesting that further investigation is warranted.
- black6 3mo ago
  Thanks. It was unclear reading the article and linked paper. I didn't follow the citation/link trail far enough.
  - rayiner 3mo ago
    Your initial assessment was correct and this part of the post above is incorrect: “when their qualifications were identical.” The paper doesn’t control for identical qualifications.

huflungdung 3mo ago

[dead]

tamimio 3mo ago

You don’t need a complicated study to find out, do it yourself for science. Get a resume, make few different versions but keep the context the same, change the layout (one time education on top other on bottom etc etc), and use different names to signal different backgrounds, and you can extend it to schools too and gender, and send it to the same employers, you will see wonders!! I tried it before, and discrimination is there, I would get one resume rejected quickly and few days later the same company would invite another resume for a screening call. I tried this before and after AI hype, results weren’t that different btw, and that was tested in US and Canada employers only.

ApolloFortyNine 3mo ago

I truly don't doubt it's possible for the AI to be 'racist'. >If the AI had recommended Black and Asian candidates at the same rate as it recommended the most-favored group (typically white applicants), 40,000 more of their applications would have advanced to the next stage of hiring. I don't think this is the right benchmark here, or at least, it would be very interesting if the actual outcome, offer or rejected, was considered at the end.

gacgacgac 3mo ago
You are misreading this sentence. This sentence is saying: "Using a constructed dataset of resumes, whose only difference was a name change, we would anticipate a system evaluating on qualifications to produce an equal distribution of candidates across names. Our observed result was highly unequal, and that warrants further investigation."
- _0ffh 3mo ago
  To me it appears as if the study using the constructed dataset was a completely different one than the one that was concerned with AI. For the AI study real data from "3.4 million people who submit 4 million job applications to 1,700 job postings across 150 employers and 11 industry sectors" was used.

petesergeant 3mo ago

I’m sure (really sure) there are real problems with AI and bias, but this is a weird study that isn’t looking at resumes or anything, it’s looking at how candidates did in some weird psychometric tests.

gacgacgac 3mo ago
Double check the link. The study clearly looked at resumes.
- petesergeant 3mo ago
  I’ve rechecked it, and I still think I’m right. What am I missing? This is the paper under discussion: https://arxiv.org/pdf/2605.27371 https://arxiv.org/pdf/2605.27371

ericol 3mo ago

2 days ago there was another interesting article on the effects of AI in hiring[1] I guess this one just compounds. [1] https://news.ycombinator.com/item?id=48620142 https://news.ycombinator.com/item?id=48620142

jazz9k 3mo ago

We can't take blanket percentages as a reason for racial bias. Were they all equally qualified? Too many of these studies only focus on percentages and the end result is unqualified candidates getting hired from minority groups at the expense of qualified ones.

gacgacgac 3mo ago
Please read the study or at least the comments here before jumping to the conclusion. Yes, they used constructed resumes, so the qualifications were exactly the same. And no, literally no one is suggesting this proves racial discrimination. It's applying the four fifths rule, a fast, coarse evaluation that is used to identify if maybe theres worth investigating more for a conclusive evidence of racial discrimination. The authors are saying it's worth doing more research, because in a controlled data set the results appear unbalanced.
- Oras 3mo ago
  > Please read the study or at least the comments here before jumping to the conclusion. Yes, they used constructed resumes Looks like you didn't read the paper. There are no resumes involved. It is about assessment games.
- etchalon 3mo ago
  I think you're confusing this specific study with a different study, which did use duplicative resumes, and has been repeated: https://www.aeaweb.org/articles?id=10.1257/0002828042002561 https://www.aeaweb.org/articles?id=10.1257/0002828042002561

GrinningFool 3mo ago

[dead]

ortusdux 3mo ago

Ayres, I., Banaji, M. and Jolls, C. (2015), Race effects on eBay. The RAND Journal of Economics, 46: 891-917. https://doi.org/10.1111/1756-2171.12115 https://doi.org/10.1111/1756-2171.12115 "Cards held by African-American sellers sold for approximately 20% ($0.90) less than cards held by Caucasian sellers, and the race effect was more pronounced in sales of minority player cards."

JuniperMesos 3mo ago

[flagged]

deleted 3mo ago
[deleted]
dzonga 3mo ago
does your anecdote comprise of the various instances when CVs were discriminated against cz people's names sounded black ? but you want to spew nonsense. every racial group includes its own under-qualified people ! there's no social pressure i.e DEI excuse you wanna give - but just economic agents acting for their own interests
techblueberry 3mo ago
> These results are consistent with AI hiring tools being completely racially unbiased, and real-world hiring managers feeling social pressure to hire underqualified black people And so managers are feeling social pressure to hire under qualified Asians as well? I must not be up to date on the latest culture war talking points, because I thought Asians were underrepresented.
- JuniperMesos 3mo ago
  Yeah, if they themselves are asian. One of the most prevalent complaints about Indian hiring managers in the silicon valley tech industry is that they preferentially hire Indians and push out non-Indians to a tremendous degree, and are often helping Indian hires commit pretty blatant credential fraud.

Oras 3mo ago

Misleading title the paper [0] does not mention any CV screening that might suggest racial or gender bias. It is purely about assessment tool. No AI or LLMs. I'm not saying AI is not biased, but this study does not prove that. [0] https://arxiv.org/pdf/2605.27371 https://arxiv.org/pdf/2605.27371 From the paper: > Fig. 1. The pymetrics process. > Stage 1: Applicants apply to positions. > Stage 2: Applicants are directed to the pymetrics platform to play assessment games. > Stage 3: pymetrics algorithms use applicant gameplay features to recommend 58.2% of applicants per position on average. > Stage 4: Employers decide which applicants to interview or hire, typically rejecting applicants that were not recommended by pymetrics.

jmyeet 3mo ago

Many people seem to think racism begins and ends with using a slur. You can usually get a measure of this by seeing someone's reaction to the statement: > There is no such thing as anti-white racism. If you find yourself wanting to disagree with that then, I'm sorry but you simply don't know what racism is. Racism is pervasive, insidious and systemic. A good example in the hiring space is what's called the "second syllable name problem". Traditionally Afrcian names often stress the second syllable (eg Jamal, Lakisha, Malik, Lashonda). Studies have shown that such names have higher rejection rates in job applications [1]. So if you're wondering about the four-fifths rule, it's because it exposes this kind of bias. It's not proof of bias. It simply means further investigation is required. The problem with AI hiring tools is the logic is opaque. You have no idea why an AI system is rejecting or selecting candidates and you may find it's doing something illegal. Some companies want to hide behind this opaqueness, arguing that if no explicit decision was made then there is no bias. But that's not how system racism works. There are many such signals that correlate with race that if they affect selection rate, it could be a problem. Did you go to an HBCU? Was your high school in a minority-majority area? What about your previous employers? This kind of bias doesn't have to be intentional. [1]: https://www.npr.org/2024/04/11/1243713272/resume-bias-study-white-names-black-names https://www.npr.org/2024/04/11/1243713272/resume-bias-study-...

junofan 3mo ago
[flagged]
kbelder 3mo ago
> > There is no such thing as anti-white racism. > If you find yourself wanting to disagree with that then, I'm sorry but you simply don't know what racism is. You are saying that if you think anti-white racism can exist, you don't know what racism is. That's obviously ludicrous.
- etchalon 3mo ago
  There are essentially two definitions of racism at this point. The colloquially version, which means "prejudice based on race" and a second version, which specific groups and people have advocated for, which means something like "structural oppression through cultural and governmental means". It's more complicated than just that, but it's a fairly narrow term for them. So when one person says "there's no such thing as anti-white racism", you hear, "No one's prejudiced against white people for being white!" Obviously, that's ludicrous. But that person is likely using the, I have no idea what to call it, "advocate definition" maybe, definition would which preclude anti-white racism from existing within that narrow definition of racism. So it's a debate where people aren't speaking the same version of a language, convinced each other are uninformed, reactionary or stupid.

tlogan 3mo ago

I am not surprised. AI works by learning patterns. So it will become bias by just learning from factors like education history, schools attended, employment history, ZIP codes, or geographic location. Those 3 factors alone are an easy proxy for race. And if you add names into the equation (if the AI was trained without removing applicant names), the model can become even more bias.

jsemrau 3mo ago

Interesting timing as Workday is facing Discrimination Claims in California doing the same thing. https://www.yahoo.com/news/us/articles/california-judge-upholds-discrimination-claims-202610640.html https://www.yahoo.com/news/us/articles/california-judge-upho...

stevenicr 3mo ago

I expected more information from the article and 'the paper' - I see nothing that shows any system was making a decision on race. How is the race being presented to the AI? All this is showing from what I can see, is that certain groups of people were more often denied a next step in the process - but why? Was the AI going by spelling and grammar? Were there names that were different but the rest of the resume was exactly the same? Were there pictures? There were mentions that the rate of each group may be more prominent in the data when you split apart different types of jobs instead of all jobs in aggregate.. One could read that like it's inferred; that more warehouse jobs are offered to a race and less admin jobs.. but that same would happen if AI was more focused on perfect grammar for one job and it was not as much of a factor for a warehouse job. Also if the people applying for the various jobs were self selecting, acceptance percentages this would skew things based upon which ones were applied / not applied to right? There are so many ways you could draw conclusions like this from data, however correlation is not causation, yet this seems to say it is. I feel this is an important thing to watch, but Stanford may not be the place to trust with 'Policy Recommendations' as it's very unclear there is any proof that 'AI Hiring Tools Yield Racial Bias and Systemic Rejection' from this study and paper. PS - now that I see the HN title did not have the word "can" in it, and the title of the article is actually "Tools Can Yield" - maybe that is less accusing and more noting.

OrvalWintermute 3mo ago

The Pymetrics game is rigged by design: Only 40% self report gender/race no resume data, no education information, degrees, schools, GPA, major, work experience, skills/certifications Zero job qualifications

zerocrates 3mo ago
Well, they're only looking at whether the pymetrics gameplay algorithm ML thing recommends the candidate, not any of that other stuff. The outcome they're looking at here isn't whether the people actually got hired, or got passed by other screening layers or anything.

ETH_start 3mo ago

A racially disparate outcome is not evidence of racial bias.

TheMagicHorsey 3mo ago
Imagine if they applied this same logic to the NBA draft.
mstewartgallus 3mo ago
Correlation does not imply causation but it is really curious how the USA has a history of settler-colonialism, slavery, segregation, eugenics, mass incarceration and imperialism. The USA and the tech industry also literally have edgelord internet Nazis in positions of power right now.

jimmy76615 3mo ago

[flagged]

rnxrx 3mo ago

Genuine curiosity: Is there any speculation as to what these tools are keying on to reject those particular applicants? It seems like it just being the applicant's name is too easy an answer, but I could be overthinking it.

marsven_422 3mo ago

[dead]

jongjong 3mo ago

I think the discrimination aspect is downstream from this fact: > We follow 3.4 million people who submit 4 million job applications to 1,700 job postings across 150 employers and 11 industry sectors. Each job application was assessed by an AI hiring tool built by a single third-party vendor. 3.4 million people applying to just 150 employers... Who are all using just 1 platform. WTF. This is where the discrimination is happening. Why the f do 3.4 million people feel forced to apply to just 150 employers and why the f do all these 150 employers feel forced to use just one platform. WTF.

zeroonetwothree 3mo ago
That’s the platform that gave them the data. I don’t think they claim it’s all the applications of this set of people.
- jongjong 3mo ago
  I realise this but it's still incredible to think because that's about 22k applicants per company. Even if that's just part of each company's total hiring pipeline, it's clear; something's wrong. I don't know how long this study has been running but 22k is a lot of people, even over a year. These companies are too big. That's the problem.
  - mstewartgallus 3mo ago
    Capitalism develops into monopolies pretty inevitably. Economies of scale just make monopolies more profitable. Monopolies also get in bed with the state and do dirty tricks to stamp out competition. But monopoly capitalism has been the case for over a 100 years now.

daft_pink 3mo ago

Anyone who’s done hiring wouldn’t be shocked by this: We find applicants are more likely to be rejected from every position they apply to than would be predicted by the baseline of each position making statistically independent decisions. Obviously a rejected resume is more likely to be rejected by every other employer and an accepted resume is more likely to be accepted by every other employer. Like online dating, most employers are looking for some baseline indicators that you are going to be successful and stable.

zeroonetwothree 3mo ago
Yes I don’t understand why this is surprising or problematic at all? Actually the fact that they found this result didn’t hold in a different dataset is especially weird.
pc86 3mo ago
> a rejected resume is more likely to be rejected by every other employer This makes sense to me, albeit intuitively and in a way I can't articulate. > an accepted resume is more likely to be accepted by every other employer but this doesn't necessarily follow from the prior for me. Plenty of people get really good jobs and are really successful in them only after dozens or hundreds of rejections with a nearly-identical resume.
- heylook 3mo ago
  The intuition is that they are not truly independent statistical events. Each trial reveals more information about the underlying "quality" of the resume (for passing this trial, not necessarily real world "quality" of the candidate). We are not rolling dice where each toss is fundamentally unrelated to prior tosses.
- daft_pink 3mo ago
  If you look at the chart, the systemic rejection rate is only like 5-10%. It’s not a huge impact and it’s just about getting an interview not getting the job, so they could still get rejected. I just think certain resumes will get an interview almost every time in some industries and certain resumes will likely never get an interview almost every time, but the majority of resumes are like you say have different aspects that appeal to one empoyer over another.

kenjackson 3mo ago

I think this partially buries the lede: "As a single hiring vendor comes to dominate screening for an industry, it may be more likely that candidates are shut out." If we move to using just a small number of AI models to help do things like hiring, we will amplify biases and possibly completely lock out portions of the population. We need to be very careful when using AI systems to evaluate people in general -- not because they might be biased (which they might be), but because even a small bias, if used by virtually everyone, can be damning.

pc86 3mo ago
If you want to make meaningful change in this avenue you really can't use words like "bias" or "systemic" because anywhere from 49-51% of the population will immediately shut down upon hearing that. Someone can argue (and many do to varying levels of success) that systemic bias doesn't exist, which means this doesn't exist, which means there no problem. However, "this AI model can decide that some subset of people, perhaps random, perhaps not, are simply not hirable for any job" makes sense to most people regardless of political bent.
- rayiner 3mo ago
  The problem with the term “systemic bias” is that it takes a word that’s about differential treatment and changes the subject to disparate outcomes. For example, the article here shows disparate impact: that different percentages of applications are passed through the AI filter. But it doesn’t show differential treatment of otherwise identical applications based on race.
- nyrikki 3mo ago
  Capitulation is a bad counterpropaganda tactic, especially with terms that have well defined domain specific meanings. Note that the OP uses "systemic rejection", while the paper does reference bias, it is in the precise meaning of the word.[0] And this is not targeted at the general public. You may want to look into the 1990 GOPAC handout "Language: A Key Mechanism of Control" to understand why some groups would simply just weaponize any term that was substituted. Academic papers need to error on being precise, to be effective, not focused on handling the general public with kids gloves IMHO. Edited to add, listen to Lee Atwater's 1981 Interview on the Southern Strategy for even more context. [0] https://arxiv.org/pdf/2605.27371 https://arxiv.org/pdf/2605.27371

tbrownaw 3mo ago

> We find that people who submit multiple applications to positions screened by the same algorithmic hiring vendor are more likely to be rejected from every position to which they apply than would be true if the companies made decisions statistically independently from one another. Ten percent of applicants who submit four applications are rejected from all the places to which they apply. > Our research also found that this pattern does not appear to be the case in other circumstances. We analyzed data from the largest prior study of hiring decisions, which sent 83,000 applications to 108 Fortune 500 firms during the same time period as our study and did not focus on whether AI was used to make decisions. We found that the rate at which applicants were rejected from every firm they applied to in this data was no higher than what you’d expect if each company decided independently of the others. It sounds like this study was using real-world applicants, and the other study they're comparing against was using synthetic applicants. Consider the chance of being accepted as being composed of signal+bias+noise. Noise is random. Signal is a per-applicant value, and what's meant to be measured. Bias is a per-group value, and an artifact of the measuring process. If acceptance/rejection is independent between positions applied for (as in the synthetic applicant study), that suggests that it's random or composed entirely of noise; ie there is no signal; ie the applicants are all equally qualified. If acceptance/rejection is correlated, that means there is some nonzero amount of (signal+bias). But real-world applicants are not all identical, so there should be some amount of signal. So you can't just assume zero signal in order to infer that there must be bias.

slashdave 3mo ago
I think I am confused. A inferior candidate (by skill) is going to be consistently rejected, no?

alexpotato 3mo ago

I went to a state school. I then went on to work for multiple firms that placed a premium on candidates from Ivy League/Top Tier (Stanford/Duke etc) candidates. This taught me that: - Their are pros and cons to any selection criteria. - There are smart people everywhere. One of the smartest people I ever worked for spent several years in prison for drug dealing. He was on par with many of the Managing Directors I've worked for - There was a study where they asked big bank recruiters which school consistently produced people who were excellent employees 2-3 years out from hiring and the answer was Penn State (not my alma mater) - There used to be "manager's choice" hires where managers had 1 slot in a training program where they could select whoever they wanted. Sometimes that was terrible. Sometimes that person was top of their training program. - Smart people are just as capable as creating problems as less intelligent people. Smart people, in some ways, are better at creating problems. Especially if the incentives reward them for creating those problems.

llmslave 3mo ago
ivy league advantage even after working on the job is unreal and underestimated
pc86 3mo ago
> There used to be "manager's choice" hires where managers had 1 slot in a training program where they could select whoever they wanted. Sometimes that was terrible. Sometimes that person was top of their training program. This seems like a great idea to me if you institute a feedback loop so managers who pick trash eventually lost the option to pick, and managers who pick rock stars eventually get more picks / more responsibility.

Terr_ 3mo ago

> If we pool all of its recommendations together — treating the vendor as one giant hiring process — we don’t find adverse impact. If we look at each position separately, as would be typical in an evaluation of adverse impact, then we expose the adverse impact in many positions. Sounds a bit like Simpson's Paradox [0] [0] https://en.wikipedia.org/wiki/Simpson%27s_paradox https://en.wikipedia.org/wiki/Simpson%27s_paradox

deleted 3mo ago

[deleted]

zombot 3mo ago

And then they complain about algorithmic monoculture in applications. https://news.ycombinator.com/item?id=48620142 https://news.ycombinator.com/item?id=48620142

casey2 3mo ago

So if the AI is selecting the economically optimal person for the job does that mean previous hiring systems were biased in favor of Asian and Black hires? Why would that have been the case?