7 ms·

Hmm. Are you familiar enough with research in economics, or the hard sciences, to say that all the work in those disciplines lack an emphasis on “generalizable

by zamfi 2y ago

Hmm. Are you familiar enough with research in economics, or the hard sciences, to say that all the work in those disciplines lack an emphasis on “generalizable findings”? It feels like epistemic weaksauce to claim that entire fields explicitly reject the goal of generalizable knowledge because they question or reject “the scientific method” on the basis of “I’ve read some case study / qualitative papers”.

mistermann 2y ago
The notion that rejecting the scientific method is necessarily bad thing is epistemically unsound, and a rather provocative topic.
- goatlover 2y ago
  Do you have another method in mind that is epistemically sound other than the scientific method?
  - mistermann 2y ago
    I'm more interested in broad capabilities, not only just epistemic soundness.
- peatmoss 2y ago
  And yet I did not state this, and it seems you're rounding my position to an easier one to dismiss. I don't reject non-scientific scholarship. But I see the focus in some disciplines a little like twinkies and soda: a bit of them can be fine and maybe even be good in some circumstances (brings joy; maybe as a recovery item for some diabetic conditions). But my feeling is that some of these disciplines have indexed a bit too much on twinkie and soda.
  - mistermann 2y ago
    I think we are very much on the same general page!
    - peatmoss 2y ago
      Ah, I think I misread the intent of what you wrote—apologies!
      - mistermann 2y ago
        Not entirely I bet! I think of science in much the same way that you think of other disciplines.
UncleMeat 2y ago
It is always amazing to me how many people are willing to say "wow this entire academic field is just garbage and the researchers are making some incredibly childish error" based on what seems just entirely like vibes. Psych is getting a brunt of this because they have actually done a good thing and funded replication studies, which naturally will produce a bunch of "this fails to replicate" findings. So then you get headlines talking about the replication crisis in psych and then people who maybe took a single class in college a decade ago dismiss the entire field as bogus. I invite such people to go speak with some CS academics for a better understanding of the mess in our own field.
peatmoss 2y ago
Sorry, I missed your comment / didn't realize it was directed specifically at me. So, if you go back and read what I wrote: 1. I definitely did not say that economics does not strive for generalizability. Quite the contrary. I think the vast majority of economists strive for generalizable conclusions (i.e. go to great care to find study samples that are representative of the population of interest and use statistical methods that might allow them to plausibly conclude things about that group) 2. I never said "all" work in other social science disciplines rejects generalizability as an aim. However, I do believe, that more "empirical" scientific method based studies in a number of disciplines would be good. That is predicated on 1. a belief about the prevalence of qualitative/theory based scholarship. 2. a normative preference that the ratio is undesirable. I could be mistaken on my perception of the first point, but I don't think I am. On the second point, you're welcome to disagree. These are, like, opinions, man! 3. "reject the goal of generalizable knowledge because they question or reject 'the scientific method'" makes me think you've either missed my point, or you don't understand the nature of other forms of academic scholarship. I'm not leveling a diss that e.g. a participant observation case study isn't a method intended to generalize from. That's just an intrinsic feature of that kind of study method. Of the qualitative researchers I've known, I can't imagine any thinking there is anything controversial about what I've said on that point. Though, some would definitely disagree with me on my opinions about what ratio of scholarship should be of this kind. And lastly you've selectively quoted me there at the end, and attached a conclusion to it that is not mine. Consider this a bit of original qualitative research on my part: a sampling of non-empirical research that I've read in recent years has suggested to me a lapse in rigor in certain disciplines. But this conclusion is grounded in my "situated knowledge" of the space, and thus shouldn't be used to generalize without a suitably operationalized quantitative study ;-)
- zamfi 2y ago
  I'm sorry, but the thrust of your earlier post was absolutely that the social sciences "have taken a pretty hard turn" and "might benefit from a renewed emphasis on methods that can result in generalizable findings", and that "a correction is warranted". Pragmatically speaking, I'm unsure why you would choose this language if you wished to convey the nuance that qualitative methods are actually great, that you're merely wishing that more "empirical" studies would also be undertaken, where "empirical" I guess means "quantitative" and "rigorous" though you don't make that explicit. For what it's worth, you absolutely can generalize from an observational case study. RCTs are not the only way of drawing generalizable conclusions—it depends a lot on what your epistemic goals are. It kind of sounds like you don't like that some social sciences rely more on non-quantitative methods because you don't think those are definitive. That's fine, you're welcome to hold that belief, but let's not pretend like you're a fan of all methods and just wish there were a few more quantitative studies in sociology (or whichever discipline).

by zamfi 2y ago

mistermann 2y ago

The notion that rejecting the scientific method is necessarily bad thing is epistemically unsound, and a rather provocative topic.

goatlover 2y ago
Do you have another method in mind that is epistemically sound other than the scientific method?
- mistermann 2y ago
  I'm more interested in broad capabilities, not only just epistemic soundness.
peatmoss 2y ago
And yet I did not state this, and it seems you're rounding my position to an easier one to dismiss. I don't reject non-scientific scholarship. But I see the focus in some disciplines a little like twinkies and soda: a bit of them can be fine and maybe even be good in some circumstances (brings joy; maybe as a recovery item for some diabetic conditions). But my feeling is that some of these disciplines have indexed a bit too much on twinkie and soda.
- mistermann 2y ago
  I think we are very much on the same general page!
  - peatmoss 2y ago
    Ah, I think I misread the intent of what you wrote—apologies!
    - mistermann 2y ago
      Not entirely I bet! I think of science in much the same way that you think of other disciplines.

UncleMeat 2y ago

It is always amazing to me how many people are willing to say "wow this entire academic field is just garbage and the researchers are making some incredibly childish error" based on what seems just entirely like vibes. Psych is getting a brunt of this because they have actually done a good thing and funded replication studies, which naturally will produce a bunch of "this fails to replicate" findings. So then you get headlines talking about the replication crisis in psych and then people who maybe took a single class in college a decade ago dismiss the entire field as bogus. I invite such people to go speak with some CS academics for a better understanding of the mess in our own field.

peatmoss 2y ago

Sorry, I missed your comment / didn't realize it was directed specifically at me. So, if you go back and read what I wrote: 1. I definitely did not say that economics does not strive for generalizability. Quite the contrary. I think the vast majority of economists strive for generalizable conclusions (i.e. go to great care to find study samples that are representative of the population of interest and use statistical methods that might allow them to plausibly conclude things about that group) 2. I never said "all" work in other social science disciplines rejects generalizability as an aim. However, I do believe, that more "empirical" scientific method based studies in a number of disciplines would be good. That is predicated on 1. a belief about the prevalence of qualitative/theory based scholarship. 2. a normative preference that the ratio is undesirable. I could be mistaken on my perception of the first point, but I don't think I am. On the second point, you're welcome to disagree. These are, like, opinions, man! 3. "reject the goal of generalizable knowledge because they question or reject 'the scientific method'" makes me think you've either missed my point, or you don't understand the nature of other forms of academic scholarship. I'm not leveling a diss that e.g. a participant observation case study isn't a method intended to generalize from. That's just an intrinsic feature of that kind of study method. Of the qualitative researchers I've known, I can't imagine any thinking there is anything controversial about what I've said on that point. Though, some would definitely disagree with me on my opinions about what ratio of scholarship should be of this kind. And lastly you've selectively quoted me there at the end, and attached a conclusion to it that is not mine. Consider this a bit of original qualitative research on my part: a sampling of non-empirical research that I've read in recent years has suggested to me a lapse in rigor in certain disciplines. But this conclusion is grounded in my "situated knowledge" of the space, and thus shouldn't be used to generalize without a suitably operationalized quantitative study ;-)

zamfi 2y ago
I'm sorry, but the thrust of your earlier post was absolutely that the social sciences "have taken a pretty hard turn" and "might benefit from a renewed emphasis on methods that can result in generalizable findings", and that "a correction is warranted". Pragmatically speaking, I'm unsure why you would choose this language if you wished to convey the nuance that qualitative methods are actually great, that you're merely wishing that more "empirical" studies would also be undertaken, where "empirical" I guess means "quantitative" and "rigorous" though you don't make that explicit. For what it's worth, you absolutely can generalize from an observational case study. RCTs are not the only way of drawing generalizable conclusions—it depends a lot on what your epistemic goals are. It kind of sounds like you don't like that some social sciences rely more on non-quantitative methods because you don't think those are definitive. That's fine, you're welcome to hold that belief, but let's not pretend like you're a fan of all methods and just wish there were a few more quantitative studies in sociology (or whichever discipline).