6 ms·

The real issue is careers valuing where papers are published over what they contribute, and incentives won’t change until hiring and funding reward depth over v

The real issue is careers valuing where papers are published over what they contribute, and incentives won’t change until hiring and funding reward depth over volume.

maweki 1y ago
> valuing where papers are published over what they contribute And who is the arbiter of that? This is an imperfect but easy shorthand. Like valuing grades and degrees instead of what people actually took away from school. In an ideal world we would see all this intangible worth in people's contributions. But we don't have time for that. So the PhD committee decides on exactly that measure whether there are enough published articles for a cumulative dissertation and if that's enough. What's exactly the alternative? Calling in fresh reviews to weigh the contributions?
- zipy124 1y ago
  Avoiding the problem altogether is just throwing up your hands and saying "this is too hard so I'm not going to even try". We already know there is some way to do it because researchers do salami slicing where they take one paper and split it up into multiple papers to get more numbers, out of the same work. Therefore one might for example look at a paper and think, how many papers could one get out of this if they were to take part in salami slicing in order to get at-least some measure of this initially.
dimatura 1y ago
Depends on who is doing the "careers valuing" and how closely they're looking. At a coarse level, especially for jobs in industry, venue is a pretty simple (but obviously imperfect) indicator for quality. If you've managed to publish one or more papers at the most selective venues (esp. as main author), then I would assume there's a decent chance you are good at research, even if I don't know anything about the subfield you work on. As a further indicator, the number of citations is also a noisy but easy to check proxy for "impact". But for academic or other high-level research jobs, whoever is doing the valuing is going to look at a lot more than just the venue.
- Al-Khwarizmi 1y ago
  > But for academic or other high-level research jobs, whoever is doing the valuing is going to look at a lot more than just the venue. Depends on where. In some countries (e.g. mine, Spain), the notion that evaluation should be "objetive" leads to it degenerating into a pure bean-counting exercise: a first-quartile JCR indexed journal paper is worth 10 points, a top-tier (according to a specific ranking) conference paper is worth 8 points, etc. In some calls/contexts there is some leeway for evaluators to actually look at the content and e.g. subtract points for salami slicing or for publishing in journals that are known to be crap in spite of good quartile, but in others it's not even allowed to do that (you would face an appeal for not following the official scoring scale).
  - dimatura 1y ago
    Yeah, that's a good point. I was thinking in the US context, but I also have some experience for the academic evaluation process in Chile, and there were similar issues to what you were describing. The "bean counting" part of it was an issue for academics in CS, because the rules were the same across departments, even where it didn't really make sense. So for example CS profs got no credit (towards promotions) for publishing in conferences, even if they were highly selective ones like ICCV or N(eur)IPS.
    - Al-Khwarizmi 1y ago
      Yes, the issue you mention is also typical in many countries with subpar systems. In Spain it used to be exactly the same. Lately, top conferences are getting recognition in some contexts, but there are still some calls where they don't count and it's better to have a crappy journal paper. I mostly publish in conferences but always have to make sure to have enough indexed journal papers per 6-year period to feed the system.
Ar-Curunir 1y ago
It is difficult to judge the impact for most papers in the first few years. Additionally, impact is influenced by visibility, and when there’s a deluge of papers, guess what’s a really good way to make your paper stand out? Yup, publication in a good venue.
bonoboTP 1y ago
This has truth but is not the full story. It gets your foot in the door, but I think junior researcher overinflate the importance of numbers. At the end of the day when you interview, your future boss will ask about your research, and it doesn't matter all that much how many papers they are and how many are top published. Yes you should have some top conf papers, but one great Arxiv paper that is actually used by others can have more value than 4 meh papers that slipped through the reviews. Remember that nobody is a passive actor in the system. Everyone sees the state of conferences and review randomness and the gaming of the system. Senior researchers are area chairs and program chairs. They are well aware and won't take paper counts as the only signal. Again, papers are needed, but it's really not the only thing.

The real issue is careers valuing where papers are published over what they contribute, and incentives won’t change until hiring and funding reward depth over volume.

maweki 1y ago

> valuing where papers are published over what they contribute And who is the arbiter of that? This is an imperfect but easy shorthand. Like valuing grades and degrees instead of what people actually took away from school. In an ideal world we would see all this intangible worth in people's contributions. But we don't have time for that. So the PhD committee decides on exactly that measure whether there are enough published articles for a cumulative dissertation and if that's enough. What's exactly the alternative? Calling in fresh reviews to weigh the contributions?

zipy124 1y ago
Avoiding the problem altogether is just throwing up your hands and saying "this is too hard so I'm not going to even try". We already know there is some way to do it because researchers do salami slicing where they take one paper and split it up into multiple papers to get more numbers, out of the same work. Therefore one might for example look at a paper and think, how many papers could one get out of this if they were to take part in salami slicing in order to get at-least some measure of this initially.

dimatura 1y ago

Depends on who is doing the "careers valuing" and how closely they're looking. At a coarse level, especially for jobs in industry, venue is a pretty simple (but obviously imperfect) indicator for quality. If you've managed to publish one or more papers at the most selective venues (esp. as main author), then I would assume there's a decent chance you are good at research, even if I don't know anything about the subfield you work on. As a further indicator, the number of citations is also a noisy but easy to check proxy for "impact". But for academic or other high-level research jobs, whoever is doing the valuing is going to look at a lot more than just the venue.

Al-Khwarizmi 1y ago
> But for academic or other high-level research jobs, whoever is doing the valuing is going to look at a lot more than just the venue. Depends on where. In some countries (e.g. mine, Spain), the notion that evaluation should be "objetive" leads to it degenerating into a pure bean-counting exercise: a first-quartile JCR indexed journal paper is worth 10 points, a top-tier (according to a specific ranking) conference paper is worth 8 points, etc. In some calls/contexts there is some leeway for evaluators to actually look at the content and e.g. subtract points for salami slicing or for publishing in journals that are known to be crap in spite of good quartile, but in others it's not even allowed to do that (you would face an appeal for not following the official scoring scale).
- dimatura 1y ago
  Yeah, that's a good point. I was thinking in the US context, but I also have some experience for the academic evaluation process in Chile, and there were similar issues to what you were describing. The "bean counting" part of it was an issue for academics in CS, because the rules were the same across departments, even where it didn't really make sense. So for example CS profs got no credit (towards promotions) for publishing in conferences, even if they were highly selective ones like ICCV or N(eur)IPS.
  - Al-Khwarizmi 1y ago
    Yes, the issue you mention is also typical in many countries with subpar systems. In Spain it used to be exactly the same. Lately, top conferences are getting recognition in some contexts, but there are still some calls where they don't count and it's better to have a crappy journal paper. I mostly publish in conferences but always have to make sure to have enough indexed journal papers per 6-year period to feed the system.

Ar-Curunir 1y ago

It is difficult to judge the impact for most papers in the first few years. Additionally, impact is influenced by visibility, and when there’s a deluge of papers, guess what’s a really good way to make your paper stand out? Yup, publication in a good venue.

bonoboTP 1y ago

This has truth but is not the full story. It gets your foot in the door, but I think junior researcher overinflate the importance of numbers. At the end of the day when you interview, your future boss will ask about your research, and it doesn't matter all that much how many papers they are and how many are top published. Yes you should have some top conf papers, but one great Arxiv paper that is actually used by others can have more value than 4 meh papers that slipped through the reviews. Remember that nobody is a passive actor in the system. Everyone sees the state of conferences and review randomness and the gaming of the system. Senior researchers are area chairs and program chairs. They are well aware and won't take paper counts as the only signal. Again, papers are needed, but it's really not the only thing.