6 ms·

Is the UK's liver transplant matching algorithm biased against younger patients?

93 points by randomwalker 2y ago

icegreentea2 2y ago
I think the generalized take away from this article, and the position held by the authors is: "Overall, we are not necessarily against this shift to utilitarian logic, but we think it should only be adopted if it is the result of a democratic process, not just because it’s more convenient." and "Public input about specific systems, such as the one we’ve discussed, is not a replacement for broad societal consensus on the underlying moral frameworks.". I wonder how exactly this would work. As the article identifies, health care in particular is continuously barraged with questions of how to allocate limited resources. I think the article is right to say that the public was probably in the dark to the specifics of this algorithm, and that the transition to utilitarian based decision making frameworks (ie algorithms) was probably -not- arrived by at by a democratic process. But I think had you run a democratic process on the principle of using utilitarian logic in health care decision making, you would end up with consensus to go ahead. And then this returns us to this specific algorithmic failure. What is the scaleable process to retaining democratic oversight to these algorithms? How far down do we push? ER rooms have triage procedures. Are these in scope? If so, what do the authors imagine the oversight and control process to look like.
- loeg 2y ago
  Hm, I think the bigger issue presented is that the algorithm in question is heavily biased against younger patients -- it deviates significantly from an ideal utilitarian model.
  - icegreentea2 2y ago
    Right, so there was a flawed implementation. Even if you had democratic consent to "implement a utilitarian organ matching mode", that would not prevent this failure mode. So what is the governance and oversight framework for ensuring democratic consent from ideation to implementation to monitoring, and how does it differ from what the UK did? The article points out that there were multiple reviews of the algorithm that identified this bias all the way back in 2019. What is the process that connects that feedback with the democratic process to ensure that flawed implementations never deploy, or are adjusted quickly.
steveBK123 2y ago
I think I've worked in software/data long enough to be very very suspicious of a one-size-fits-all algorithm like this. I would be very hesitant to entrust something like organ matching to a singular matching system. There are so many ways to get it wrong - bad data, bad algo design/requirements, mistakes in implementation, people understanding the system too well being able to game it, etc. Human systems have biases, but at least there are diverse biases when there are many decision makers. If you put something important behind a single algorithm, you are locking in a fixed bias inadvertently.
- icegreentea2 2y ago
  What does a non "one size fits all" approach for organ matching look like? What does a non-singular matching system work? Do you arbitrarily (randomly?) split up organs into different pools and let each pool match by a different algorithm?
  - steveBK123 2y ago
    Yes, in the US it might look like state level / hospital system level vs 1 singular national level matching system. US has its problems, but sometimes the "laboratory of ideas" that is federated system of 50 states prevents bad outcomes like this.
    - icegreentea2 2y ago
      The challenge is maintaining the multiple independent systems when faced with pressures like "hey, if we consolidated systems, the the % of waiting list patients who die within 6 months of enrolling goes from 8% to 4%, and the % who receive a transplant go from 60% to 65%". The UK system undoubtedly had a bad outcome, but the reasoning behind consolidation was sound, and the benefits real and ACTUALLY achieved (just not dispersed justly). Maintaining independent systems would mitigate against some of these failures, but would long-term be out performed by a responsive consolidated system (which I think is ultimately what the article is arguing for - not against algorithms, but against black-box algorithms that are not responsive or amendable to public scrutiny and feedback). There are definitely times and places with independent implementations provide a strong benefits, but I think this is a much more borderline scenario. And btw, the US has a unified organ matching system.
ipnon 2y ago
Similarly the main calculator used in the US to calculate 10-year risk of cardiovascular incident literally cannot compute scores for people under 40.[0] There are two consequences to this. The first is that if you are under 40 you will never encounter a physician who believes you are at risk of heart attack or stroke, even though over 100,000 Americans under 40 will experience such an incident each year. The second is that even if you get a heart attack or stroke due to their negligence they will never be liable because that calculator is considered the standard of care in malpractice law! Governing bodies write these guidelines that act like programs, and your local doctor is the interpreter.[1] When was the last time you found a bug that could be attributed to the interpreter rather than the programmer? [0] https://tools.acc.org/ascvd-risk-estimator-plus/#!/calculate/estimate/ https://tools.acc.org/ascvd-risk-estimator-plus/#!/calculate... [1] It’s worth considering what medical schools, emergency rooms, and malpractice lawyers are analogous to in this metaphor.
- thrw42A8N 2y ago
  > When was the last time you found a bug that could be attributed to the interpreter rather than the programmer? On the other hand, when was the last time you used a custom one-off interpreter?
- hombre_fatal 2y ago
  Out of curiosity, how is a physician negligent if decades of exposure to hypertension/LDL/smoking/diabetes (the variables on that calculator) give you a heart attack or stroke? By the time you're put on a statin, for example, you've already had decades of exposure due to your lifestyle. Also, I don't believe the claim that physicians don't care about CVD risk in patients <40yo including high blood pressure and high cholesterol.
  - zamadatix 2y ago
    Flip the issue to something less polarizing and it should appear this is a very separate scenario from what GP is talking about (even if perhaps you still don't agree it should be malpractice for some reason): 1) You go in after feeling confused and have a headache after falling from a skateboard with no helmet. The ER sends you home not having checked anything or any notes to watch out for because they think you're too young to have problems from a fall (despite many young people having problems after a fall each year). At home you die because of a brain bleed. vs. 2) You go in after feeling confused and have a headache after falling from a skateboard with no helmet. The ER runs some tests, sees the problem, and prescribes the best course of treatment given this information. Despite this you still die or have lasting effects on your brain. Despite the doctors not fully remedying your problem in both situations only situation 1 involves negligence for a malpractice claim because the problem isn't the outcome, it's the quality of treatment not meeting the minimum levels. Flip the scenario specifics back and what GP is saying is that it isn't considered negligence to say "you're under 40, you're fine, go home" instead of "you could seriously be having a problem. We should put you on a statin and talk over the risks/symptoms of a heart attack" because the standard of care (sort of one measurement for what's a negligent treatment action) says the calculator defines the appropriate treatment and the calculator doesn't even work for those <40. What GP is not implying is doctors are negligent just because you still had a heart attack anyways.
Havoc 2y ago
I'd very much hope it is biased towards them if anything.
- deleted 2y ago
  [deleted]
gyudin 2y ago
What can go wrong when you let government agencies with no expertise to develop and maintain AI models and algorithms, right? And then we get articles saying that AIs are biased, racist and don’t work as expected and that AI in general as a technology has no future. I can even predict what will be their solution lmao, to pay atrocious lump of money to big consulting agencies with no expertise to develop it for them and fail again.
- cedws 2y ago
  The fairest way to do it I feel is a FIFO. Yeah you might give an organ to a 70 year old on their last legs, but they don’t have any less of a right to live than anyone else. After the Horizon scandal, public trust in complicated computer systems are at an all time low. It shouldn’t be an opaque system making such important decisions. Everything should be in the open and explainable.
  - danpalmer 2y ago
    With FIFO the difficulty simply moves to whether you go on the list or not. What should the threshold be for being on the list. If you make it too high then people die without ever being given the opportunity for a liver, if you make it too low then too many people die waiting. It also creates a weird scenario where all of the worst cases past some level will have no hope of getting a transplant. I.e. if the wait time is uniformly 3 years, then anyone with <3 years of expectancy has little hope despite being the ones who need it most, meanwhile everyone with >3 years expectancy can happily hang out on the list waiting for their transplant. Simplified a little, but you get the idea. It’s arguably fairly obvious that livers should be assigned based on urgency in some form. I absolutely agree that this should be open and explainable though.
binary132 2y ago
So, if I as a 38-year-old had a mild liver impairment which could reduce my life expectancy to 60 (22 years from now) I should get priority over a 60-year-old with a debilitating, excruciating condition which will end his life in six months, merely because his life expectancy with the transplant may only be 70? That’s an outrageous and obscene utility calculation to propose and it should be obviously so to just about anyone.
- JumpCrisscross 2y ago
  > if I as a 38-year-old had a mild liver impairment which could reduce my life expectancy to 60 (22 years from now) I should get priority over a 60-year-old with a debilitating, excruciating condition which will end his life in six months, merely because his life expectancy with the transplant may only be 70 No. Because it's mild and could reduce your life expectancy. Once it becomes worse and a will, yes--you should.
- deleted 2y ago
  [deleted]
- rainforest 2y ago
  The NHS does this calculus routinely using Quality Adjusted Life Years. Treatments that get more are favoured which is also how NICE decides what drugs the NHS should offer. There's obviously some utilitarianism in the decision to use QALYs but to some (including me) it seems a reasonable proxy metric to maximise. Ultimately a sacrifice must be chosen, but I am not sure a discussion about how that should be made is necessarily fit for HN (though I'd be interested in how you'd resolve your proposed scenario).
- defrost 2y ago
  > That’s an outrageous and obscene utility calculation to propose Welcome to the reality of triage .. all decisions are bad from some PoV or another, some are arguably less bad. Oh, to live in a world of infinite matching organs and unlimited theatre slots on demand.
  - binary132 2y ago
jwilk 2y ago
The Financial Times article discussed on HN: https://news.ycombinator.com/item?id=38202885 https://news.ycombinator.com/item?id=38202885 (22 comments)
kreyenborgi 2y ago
> The choice of a 5-year period seems to be because of data availability Also known as "looking for the keys under the lamp-post" https://en.wikipedia.org/wiki/Streetlight_effect https://en.wikipedia.org/wiki/Streetlight_effect (which links to https://en.wikipedia.org/wiki/McNamara_fallacy https://en.wikipedia.org/wiki/McNamara_fallacy which I hadn't heard of before, but which seems to fit very well here too). > An algorithmic absurdity: cancer improves survival > [...] > algorithmic absurdity, something that would > seem obviously wrong to a person based on common sense. A useful term! > optimize “quality-adjusted” life years https://repaer.earth/ https://repaer.earth/ was posted on HN recently as an extreme example of this hehe
jl6 2y ago
It’s worth noting that the algorithm in question is not any kind of AI or ML as we might know it from the tech industry. Underneath, it is plain old statistical modelling. The article doesn’t make this clear, and the name of the blog doesn’t help.
- ClassyJacket 2y ago
  I know we're on Hacker News, but still, that's how I interpreted it
- pkaye 2y ago
  I got my kidney transplant last years. Here are some of the allocation calculators used in the US. https://optn.transplant.hrsa.gov/data/allocation-calculators https://optn.transplant.hrsa.gov/data/allocation-calculators For kidney transplants, for example the EPTS score, compatibility, time on the list, geographic distance and antibody levels are used to generate the wait list ranking. For scale, you accrue 1/365 points for each day waiting on the list. Kids under 10 get 2 extra points. Kids under 20 get 1 extra point. Those with high antibody levels can get up to 20 extra points to increase their chance of getting a match. The KDPI score is an estimate of how risk of graft failure of the donor organ. The lower the number the better the odds. Those with low EPTS (<20%) will get those with KDPI <20%. Age and diabetes heavily factor in an EPTS score. The donor KDPI is something they will tell you when you get a call. You can always pass on any donor organ.

93 points by randomwalker 2y ago

I think the generalized take away from this article, and the position held by the authors is: "Overall, we are not necessarily against this shift to utilitarian logic, but we think it should only be adopted if it is the result of a democratic process, not just because it’s more convenient." and "Public input about specific systems, such as the one we’ve discussed, is not a replacement for broad societal consensus on the underlying moral frameworks.". I wonder how exactly this would work. As the article identifies, health care in particular is continuously barraged with questions of how to allocate limited resources. I think the article is right to say that the public was probably in the dark to the specifics of this algorithm, and that the transition to utilitarian based decision making frameworks (ie algorithms) was probably -not- arrived by at by a democratic process. But I think had you run a democratic process on the principle of using utilitarian logic in health care decision making, you would end up with consensus to go ahead. And then this returns us to this specific algorithmic failure. What is the scaleable process to retaining democratic oversight to these algorithms? How far down do we push? ER rooms have triage procedures. Are these in scope? If so, what do the authors imagine the oversight and control process to look like.

loeg 2y ago
Hm, I think the bigger issue presented is that the algorithm in question is heavily biased against younger patients -- it deviates significantly from an ideal utilitarian model.
- icegreentea2 2y ago
  Right, so there was a flawed implementation. Even if you had democratic consent to "implement a utilitarian organ matching mode", that would not prevent this failure mode. So what is the governance and oversight framework for ensuring democratic consent from ideation to implementation to monitoring, and how does it differ from what the UK did? The article points out that there were multiple reviews of the algorithm that identified this bias all the way back in 2019. What is the process that connects that feedback with the democratic process to ensure that flawed implementations never deploy, or are adjusted quickly.

steveBK123 2y ago

I think I've worked in software/data long enough to be very very suspicious of a one-size-fits-all algorithm like this. I would be very hesitant to entrust something like organ matching to a singular matching system. There are so many ways to get it wrong - bad data, bad algo design/requirements, mistakes in implementation, people understanding the system too well being able to game it, etc. Human systems have biases, but at least there are diverse biases when there are many decision makers. If you put something important behind a single algorithm, you are locking in a fixed bias inadvertently.

icegreentea2 2y ago
What does a non "one size fits all" approach for organ matching look like? What does a non-singular matching system work? Do you arbitrarily (randomly?) split up organs into different pools and let each pool match by a different algorithm?
- steveBK123 2y ago
  Yes, in the US it might look like state level / hospital system level vs 1 singular national level matching system. US has its problems, but sometimes the "laboratory of ideas" that is federated system of 50 states prevents bad outcomes like this.
  - icegreentea2 2y ago
    The challenge is maintaining the multiple independent systems when faced with pressures like "hey, if we consolidated systems, the the % of waiting list patients who die within 6 months of enrolling goes from 8% to 4%, and the % who receive a transplant go from 60% to 65%". The UK system undoubtedly had a bad outcome, but the reasoning behind consolidation was sound, and the benefits real and ACTUALLY achieved (just not dispersed justly). Maintaining independent systems would mitigate against some of these failures, but would long-term be out performed by a responsive consolidated system (which I think is ultimately what the article is arguing for - not against algorithms, but against black-box algorithms that are not responsive or amendable to public scrutiny and feedback). There are definitely times and places with independent implementations provide a strong benefits, but I think this is a much more borderline scenario. And btw, the US has a unified organ matching system.

ipnon 2y ago

Similarly the main calculator used in the US to calculate 10-year risk of cardiovascular incident literally cannot compute scores for people under 40.[0] There are two consequences to this. The first is that if you are under 40 you will never encounter a physician who believes you are at risk of heart attack or stroke, even though over 100,000 Americans under 40 will experience such an incident each year. The second is that even if you get a heart attack or stroke due to their negligence they will never be liable because that calculator is considered the standard of care in malpractice law! Governing bodies write these guidelines that act like programs, and your local doctor is the interpreter.[1] When was the last time you found a bug that could be attributed to the interpreter rather than the programmer? [0] https://tools.acc.org/ascvd-risk-estimator-plus/#!/calculate/estimate/ https://tools.acc.org/ascvd-risk-estimator-plus/#!/calculate... [1] It’s worth considering what medical schools, emergency rooms, and malpractice lawyers are analogous to in this metaphor.

thrw42A8N 2y ago
> When was the last time you found a bug that could be attributed to the interpreter rather than the programmer? On the other hand, when was the last time you used a custom one-off interpreter?
hombre_fatal 2y ago
Out of curiosity, how is a physician negligent if decades of exposure to hypertension/LDL/smoking/diabetes (the variables on that calculator) give you a heart attack or stroke? By the time you're put on a statin, for example, you've already had decades of exposure due to your lifestyle. Also, I don't believe the claim that physicians don't care about CVD risk in patients <40yo including high blood pressure and high cholesterol.
- zamadatix 2y ago
  Flip the issue to something less polarizing and it should appear this is a very separate scenario from what GP is talking about (even if perhaps you still don't agree it should be malpractice for some reason): 1) You go in after feeling confused and have a headache after falling from a skateboard with no helmet. The ER sends you home not having checked anything or any notes to watch out for because they think you're too young to have problems from a fall (despite many young people having problems after a fall each year). At home you die because of a brain bleed. vs. 2) You go in after feeling confused and have a headache after falling from a skateboard with no helmet. The ER runs some tests, sees the problem, and prescribes the best course of treatment given this information. Despite this you still die or have lasting effects on your brain. Despite the doctors not fully remedying your problem in both situations only situation 1 involves negligence for a malpractice claim because the problem isn't the outcome, it's the quality of treatment not meeting the minimum levels. Flip the scenario specifics back and what GP is saying is that it isn't considered negligence to say "you're under 40, you're fine, go home" instead of "you could seriously be having a problem. We should put you on a statin and talk over the risks/symptoms of a heart attack" because the standard of care (sort of one measurement for what's a negligent treatment action) says the calculator defines the appropriate treatment and the calculator doesn't even work for those <40. What GP is not implying is doctors are negligent just because you still had a heart attack anyways.

Havoc 2y ago

I'd very much hope it is biased towards them if anything.

deleted 2y ago
[deleted]

gyudin 2y ago

What can go wrong when you let government agencies with no expertise to develop and maintain AI models and algorithms, right? And then we get articles saying that AIs are biased, racist and don’t work as expected and that AI in general as a technology has no future. I can even predict what will be their solution lmao, to pay atrocious lump of money to big consulting agencies with no expertise to develop it for them and fail again.

cedws 2y ago
The fairest way to do it I feel is a FIFO. Yeah you might give an organ to a 70 year old on their last legs, but they don’t have any less of a right to live than anyone else. After the Horizon scandal, public trust in complicated computer systems are at an all time low. It shouldn’t be an opaque system making such important decisions. Everything should be in the open and explainable.
- danpalmer 2y ago
  With FIFO the difficulty simply moves to whether you go on the list or not. What should the threshold be for being on the list. If you make it too high then people die without ever being given the opportunity for a liver, if you make it too low then too many people die waiting. It also creates a weird scenario where all of the worst cases past some level will have no hope of getting a transplant. I.e. if the wait time is uniformly 3 years, then anyone with <3 years of expectancy has little hope despite being the ones who need it most, meanwhile everyone with >3 years expectancy can happily hang out on the list waiting for their transplant. Simplified a little, but you get the idea. It’s arguably fairly obvious that livers should be assigned based on urgency in some form. I absolutely agree that this should be open and explainable though.

binary132 2y ago

So, if I as a 38-year-old had a mild liver impairment which could reduce my life expectancy to 60 (22 years from now) I should get priority over a 60-year-old with a debilitating, excruciating condition which will end his life in six months, merely because his life expectancy with the transplant may only be 70? That’s an outrageous and obscene utility calculation to propose and it should be obviously so to just about anyone.

JumpCrisscross 2y ago
> if I as a 38-year-old had a mild liver impairment which could reduce my life expectancy to 60 (22 years from now) I should get priority over a 60-year-old with a debilitating, excruciating condition which will end his life in six months, merely because his life expectancy with the transplant may only be 70 No. Because it's mild and could reduce your life expectancy. Once it becomes worse and a will, yes--you should.
deleted 2y ago
[deleted]
rainforest 2y ago
The NHS does this calculus routinely using Quality Adjusted Life Years. Treatments that get more are favoured which is also how NICE decides what drugs the NHS should offer. There's obviously some utilitarianism in the decision to use QALYs but to some (including me) it seems a reasonable proxy metric to maximise. Ultimately a sacrifice must be chosen, but I am not sure a discussion about how that should be made is necessarily fit for HN (though I'd be interested in how you'd resolve your proposed scenario).
defrost 2y ago
> That’s an outrageous and obscene utility calculation to propose Welcome to the reality of triage .. all decisions are bad from some PoV or another, some are arguably less bad. Oh, to live in a world of infinite matching organs and unlimited theatre slots on demand.
- binary132 2y ago

jwilk 2y ago

The Financial Times article discussed on HN: https://news.ycombinator.com/item?id=38202885 https://news.ycombinator.com/item?id=38202885 (22 comments)

kreyenborgi 2y ago

> The choice of a 5-year period seems to be because of data availability Also known as "looking for the keys under the lamp-post" https://en.wikipedia.org/wiki/Streetlight_effect https://en.wikipedia.org/wiki/Streetlight_effect (which links to https://en.wikipedia.org/wiki/McNamara_fallacy https://en.wikipedia.org/wiki/McNamara_fallacy which I hadn't heard of before, but which seems to fit very well here too). > An algorithmic absurdity: cancer improves survival > [...] > algorithmic absurdity, something that would > seem obviously wrong to a person based on common sense. A useful term! > optimize “quality-adjusted” life years https://repaer.earth/ https://repaer.earth/ was posted on HN recently as an extreme example of this hehe

jl6 2y ago

It’s worth noting that the algorithm in question is not any kind of AI or ML as we might know it from the tech industry. Underneath, it is plain old statistical modelling. The article doesn’t make this clear, and the name of the blog doesn’t help.

ClassyJacket 2y ago
I know we're on Hacker News, but still, that's how I interpreted it
pkaye 2y ago
I got my kidney transplant last years. Here are some of the allocation calculators used in the US. https://optn.transplant.hrsa.gov/data/allocation-calculators https://optn.transplant.hrsa.gov/data/allocation-calculators For kidney transplants, for example the EPTS score, compatibility, time on the list, geographic distance and antibody levels are used to generate the wait list ranking. For scale, you accrue 1/365 points for each day waiting on the list. Kids under 10 get 2 extra points. Kids under 20 get 1 extra point. Those with high antibody levels can get up to 20 extra points to increase their chance of getting a match. The KDPI score is an estimate of how risk of graft failure of the donor organ. The lower the number the better the odds. Those with low EPTS (<20%) will get those with KDPI <20%. Age and diabetes heavily factor in an EPTS score. The donor KDPI is something they will tell you when you get a call. You can always pass on any donor organ.