5 ms·

In Praise of Observational Evidence

78 points by fi-le 3mo ago

khalic 2mo ago
Damn fine article, lovely conclusion, a real pleasure to read
samuell 2mo ago
Quite thought-provoking, and connecting it to a related field it seems the (relative) success of LLMs and the likes are indications that enough data can at least learn you something without always needing to interfere with the world first(?)
- AnimalMuppet 2mo ago
  Let me restate that: Enough accurate data about the real world can let you learn something about the real world without having to yourself interface with the real world. So, for example, Kepler's Laws were based on Tycho Brahe's observations, not on Kepler's.
SomeHacker44 2mo ago
They reference RCT without first defining it.
- Vohlenzer 2mo ago
  https://en.wikipedia.org/wiki/Randomized_controlled_trial https://en.wikipedia.org/wiki/Randomized_controlled_trial I noticed that as well. My eyes feverishly scanning the previous paragraph for the definition.
qsera 2mo ago
> The increasing availability of large datasets should make this an especially good time to reconsider observational evidence in many fields. This is not happening. > I was surprised to find that they usually discard papers based on observational evidence wholesale. He he...welcome to the real world!
A_D_E_P_T 2mo ago
In medicine, observational evidence is actually better and far more ethical than the RCT. (Which simply dooms the terminally ill to fake treatment.) You just need large datasets and an agile culture that's responsive to new input. Don't forget that RCTs are very far from perfect and issues -- sometimes literally fatal issues -- have later turned up via observational evidence in large cohorts. Vioxx, for instance. Many others. I believe, without the tiniest shred of doubt, that the only trials drugs need to go through are initial safety/toxicity trials (phases 0/1) and that everything else would be much better left to access+observation.
- taeric 2mo ago
  This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case. Should we demand an RCT before we accept evidence? Of course not. At some point you do have to make a choice on things. And it should be noted that most drugs do have early cutoff criteria if the evidence is strong enough that it is working. It isn't like people are wanting to withhold good treatments from the world. Adding controls and randomizing them, though, has proven to be highly effective at helping progress.
  - A_D_E_P_T 2mo ago
    > "This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case." If you have enough data, you can smooth out individual fluctuations due to things like drug interactions, non-compliance, etc. (And indeed you might discover drug interactions!) Observational trials ultimately mirror how drugs are used in the real world. > "Adding controls and randomizing them, though, has proven to be highly effective at helping progress." I would argue just the opposite. Demands for increasingly byzantine trials have ballooned the costs associated with drug development, and have slowed things to a crawl. There's a reason the field's golden age was in the 1940s and 1950s, and it's not just "low hanging fruit." Today nobody in their right mind wants to work in drug development when they could work in tech or even finance.
doginasuit 2mo ago
The reality that individuals and cultures create can be a dark and confining place. Empiricism is our only window to a universe full of possibility and light. Humanity is like a child, standing on our toes to peer through and wonder.
pinko 2mo ago
Seth Roberts was arguing this ~20 years ago and would have loved the advent of LLMs...
derbOac 2mo ago
Experimental designs are critical for obvious reasons but they have a few critical flaws, that mostly all reduce to the fact you can't randomize manipulations with everything. Whether it be due to ethics or practical constraints, you can't conduct a RCT all the time. This can be more subtly critical than it might seem, in that even if you can manipulate some proxy, often that proxy is insufficient in actually representing the phenomenon of interest, or the conditions under which they actually occur. I often use the example of videogames and aggression. There were plenty of experimental studies of this but it was always questionable whether lab-induced anger is the same thing as, say, the sort of violence we generally are concerned about societally. I generally have tried to teach students that experimental designs when done right provide powerful causal evidence of something, but often with limited generalizability; observational designs in contrast provide powerful generalizable evidence of some kind of association, but often with limited certainty about the causal pathways involved. I've been in a department that was rabidly experimental in its focus and it always seemed sort of short-sighted, because people were idolizing RCTs with proxy manipulations that had questionable generalizability to the real-world phenomena they were trying to model. Ideally you'd bring both experimental and observational evidence to bear on a question. Your conclusions should be robust to different types of designs.
- CGMthrowaway 2mo ago
  Yes. Even conceding every point made in this article, RCTs are still the gold standard. The article points out that RCTs are often impractical, expensive or ethically challenging. Well, so is the gold standard of anything. OP suggests that alternative methods like target trial emulation, propensity scoring and double machine learning can be used to approximate the conditions of an RCT using existing data. In saying so he gives away the tell, which is that RCTs are the standard being aspired to. Observational data may well be undervalued. But RCTs are still the gold standard.
1970-01-01 2mo ago
RCT is Random Controlled Trial. Definitions are the very basics in writing, check them before you publish.
- deleted 2mo ago
  [deleted]
- KempyKolibri 2mo ago
  Where is it defined as "Random Controlled Trial"?
  - esafak 2mo ago
    Ignoring the typo above, it's mentioned in the text: https://en.wikipedia.org/wiki/Randomized_controlled_trial https://en.wikipedia.org/wiki/Randomized_controlled_trial
    - KempyKolibri 2mo ago
      That’s not “Random Controlled Trial”. I think it’s pointless to quibble about very specific things like this, but that gets thrown out the window when someone claims “Definitions are the very basics in writing, check them before you publish” and then use their own nonstandard definition! Seems like they wanted to nitpick about term usage when: a) the usage they’re taking issue with is actually fairly commonplace (e.g. https://www.who.int/news/item/27-10-2017-who-to-review-its-interim-guidance-on-the-use-of-delamanid-in-the-treatment-of-mdr-tb-following-the-release-of-phase-iii-clinical-trial-results https://www.who.int/news/item/27-10-2017-who-to-review-its-i...) b) their own suggested replacement is very heterodox (I’ve certainly never heard anyone call it that).
    - stronglikedan 2mo ago
      Mentioned, yes, but not correctly. They should have put a "(RCT)" immediately after the first instance of "randomized control trial" at the very least. However, I'm of the opinion that the acronym should have been expanded it the first paragraph, or even first sentence, since it's used in the subheader. Either way, this was definitely a failure on the author's part.
compiler-guy 2mo ago
One classic paper for on this topic: _Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials_ https://pmc.ncbi.nlm.nih.gov/articles/PMC300808/ https://pmc.ncbi.nlm.nih.gov/articles/PMC300808/ "Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute."
- timr 2mo ago
  That's not a "classic". It's just a silly, overwrought straw-man [1] that gets passed around by people who think they're being clever, and/or don't understand the topic with any degree of nuance. Nobody reasonable suggests that RCTs must be used for every problem. When the effect size is enormous, or there's obviously no alternative (as in the parachute example), it's silly to suggest the use of an experiment. On the other hand, if you want to compare some subtle difference between two competing parachute designs, for example, then perhaps an RCT would actually be appropriate. This is much closer to what is actually being measured in real clinical trials. [1] To be completely fair to the authors, it was intended as a joke, not an argument. The real joke ended up being that people took it seriously.
mchusma 2mo ago
Good article. I will only add that I think the problems outlined in the article are exacerbated by the regulatory apparatus. If this were a debate related to truth seeking alone, it would be good. But trials drive banning/unbanning of treatments and medicines, as well as their mandated coverage by insurance. We could be much more flexible in our approach to things if we would un-ban things. For example after phase 1 trial or based on sufficient observational evidence, things can no longer be banned but have a higher standard for "insurance is now required to cover it".

78 points by fi-le 3mo ago

Damn fine article, lovely conclusion, a real pleasure to read

Quite thought-provoking, and connecting it to a related field it seems the (relative) success of LLMs and the likes are indications that enough data can at least learn you something without always needing to interfere with the world first(?)

AnimalMuppet 2mo ago
Let me restate that: Enough accurate data about the real world can let you learn something about the real world without having to yourself interface with the real world. So, for example, Kepler's Laws were based on Tycho Brahe's observations, not on Kepler's.

SomeHacker44 2mo ago

They reference RCT without first defining it.

Vohlenzer 2mo ago
https://en.wikipedia.org/wiki/Randomized_controlled_trial https://en.wikipedia.org/wiki/Randomized_controlled_trial I noticed that as well. My eyes feverishly scanning the previous paragraph for the definition.

qsera 2mo ago

> The increasing availability of large datasets should make this an especially good time to reconsider observational evidence in many fields. This is not happening. > I was surprised to find that they usually discard papers based on observational evidence wholesale. He he...welcome to the real world!

A_D_E_P_T 2mo ago

In medicine, observational evidence is actually better and far more ethical than the RCT. (Which simply dooms the terminally ill to fake treatment.) You just need large datasets and an agile culture that's responsive to new input. Don't forget that RCTs are very far from perfect and issues -- sometimes literally fatal issues -- have later turned up via observational evidence in large cohorts. Vioxx, for instance. Many others. I believe, without the tiniest shred of doubt, that the only trials drugs need to go through are initial safety/toxicity trials (phases 0/1) and that everything else would be much better left to access+observation.

taeric 2mo ago
This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case. Should we demand an RCT before we accept evidence? Of course not. At some point you do have to make a choice on things. And it should be noted that most drugs do have early cutoff criteria if the evidence is strong enough that it is working. It isn't like people are wanting to withhold good treatments from the world. Adding controls and randomizing them, though, has proven to be highly effective at helping progress.
- A_D_E_P_T 2mo ago
  > "This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case." If you have enough data, you can smooth out individual fluctuations due to things like drug interactions, non-compliance, etc. (And indeed you might discover drug interactions!) Observational trials ultimately mirror how drugs are used in the real world. > "Adding controls and randomizing them, though, has proven to be highly effective at helping progress." I would argue just the opposite. Demands for increasingly byzantine trials have ballooned the costs associated with drug development, and have slowed things to a crawl. There's a reason the field's golden age was in the 1940s and 1950s, and it's not just "low hanging fruit." Today nobody in their right mind wants to work in drug development when they could work in tech or even finance.

doginasuit 2mo ago

The reality that individuals and cultures create can be a dark and confining place. Empiricism is our only window to a universe full of possibility and light. Humanity is like a child, standing on our toes to peer through and wonder.

pinko 2mo ago

Seth Roberts was arguing this ~20 years ago and would have loved the advent of LLMs...

derbOac 2mo ago

Experimental designs are critical for obvious reasons but they have a few critical flaws, that mostly all reduce to the fact you can't randomize manipulations with everything. Whether it be due to ethics or practical constraints, you can't conduct a RCT all the time. This can be more subtly critical than it might seem, in that even if you can manipulate some proxy, often that proxy is insufficient in actually representing the phenomenon of interest, or the conditions under which they actually occur. I often use the example of videogames and aggression. There were plenty of experimental studies of this but it was always questionable whether lab-induced anger is the same thing as, say, the sort of violence we generally are concerned about societally. I generally have tried to teach students that experimental designs when done right provide powerful causal evidence of something, but often with limited generalizability; observational designs in contrast provide powerful generalizable evidence of some kind of association, but often with limited certainty about the causal pathways involved. I've been in a department that was rabidly experimental in its focus and it always seemed sort of short-sighted, because people were idolizing RCTs with proxy manipulations that had questionable generalizability to the real-world phenomena they were trying to model. Ideally you'd bring both experimental and observational evidence to bear on a question. Your conclusions should be robust to different types of designs.

CGMthrowaway 2mo ago
Yes. Even conceding every point made in this article, RCTs are still the gold standard. The article points out that RCTs are often impractical, expensive or ethically challenging. Well, so is the gold standard of anything. OP suggests that alternative methods like target trial emulation, propensity scoring and double machine learning can be used to approximate the conditions of an RCT using existing data. In saying so he gives away the tell, which is that RCTs are the standard being aspired to. Observational data may well be undervalued. But RCTs are still the gold standard.

1970-01-01 2mo ago

RCT is Random Controlled Trial. Definitions are the very basics in writing, check them before you publish.

deleted 2mo ago
[deleted]
KempyKolibri 2mo ago
Where is it defined as "Random Controlled Trial"?
- esafak 2mo ago
  Ignoring the typo above, it's mentioned in the text: https://en.wikipedia.org/wiki/Randomized_controlled_trial https://en.wikipedia.org/wiki/Randomized_controlled_trial
  - KempyKolibri 2mo ago
    That’s not “Random Controlled Trial”. I think it’s pointless to quibble about very specific things like this, but that gets thrown out the window when someone claims “Definitions are the very basics in writing, check them before you publish” and then use their own nonstandard definition! Seems like they wanted to nitpick about term usage when: a) the usage they’re taking issue with is actually fairly commonplace (e.g. https://www.who.int/news/item/27-10-2017-who-to-review-its-interim-guidance-on-the-use-of-delamanid-in-the-treatment-of-mdr-tb-following-the-release-of-phase-iii-clinical-trial-results https://www.who.int/news/item/27-10-2017-who-to-review-its-i...) b) their own suggested replacement is very heterodox (I’ve certainly never heard anyone call it that).
  - stronglikedan 2mo ago
    Mentioned, yes, but not correctly. They should have put a "(RCT)" immediately after the first instance of "randomized control trial" at the very least. However, I'm of the opinion that the acronym should have been expanded it the first paragraph, or even first sentence, since it's used in the subheader. Either way, this was definitely a failure on the author's part.

compiler-guy 2mo ago

One classic paper for on this topic: _Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials_ https://pmc.ncbi.nlm.nih.gov/articles/PMC300808/ https://pmc.ncbi.nlm.nih.gov/articles/PMC300808/ "Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute."

timr 2mo ago
That's not a "classic". It's just a silly, overwrought straw-man [1] that gets passed around by people who think they're being clever, and/or don't understand the topic with any degree of nuance. Nobody reasonable suggests that RCTs must be used for every problem. When the effect size is enormous, or there's obviously no alternative (as in the parachute example), it's silly to suggest the use of an experiment. On the other hand, if you want to compare some subtle difference between two competing parachute designs, for example, then perhaps an RCT would actually be appropriate. This is much closer to what is actually being measured in real clinical trials. [1] To be completely fair to the authors, it was intended as a joke, not an argument. The real joke ended up being that people took it seriously.

mchusma 2mo ago

Good article. I will only add that I think the problems outlined in the article are exacerbated by the regulatory apparatus. If this were a debate related to truth seeking alone, it would be good. But trials drive banning/unbanning of treatments and medicines, as well as their mandated coverage by insurance. We could be much more flexible in our approach to things if we would un-ban things. For example after phase 1 trial or based on sufficient observational evidence, things can no longer be banned but have a higher standard for "insurance is now required to cover it".