6 ms·

I'm long past my academia phase, but recently led the PC for an industry conference (accept rate: ~15%). 1. Curation is important both for the physical limits

by ath0 4y ago

I'm long past my academia phase, but recently led the PC for an industry conference (accept rate: ~15%). 1. Curation is important both for the physical limits (venues only fit a certain number of people), attention limits (attendees will usually retain only a handful of "nuggets" no matter how packed the agenda is) and interaction limits (you can't meet everyone at a large conference). 2. If the goal of a conference is not just to "stamp" research as somehow "approved", but to encourage discovery and knowledge exchange that deepens a specific area, it's important to apply that curation filter with an eye toward best advancing the goals of the conference. That means not just going for things that are okay, but those that best resonate with other presentations / attendees / research topics. 3. While the size of any one conference has to be fixed, tech has made it infinitely easier to create new conferences and journals with other focus areas. They may not start with the prestige of a larger journal, but if the papers published start to have an impact, it can catalyze an entire subfield of work. Some conferences can be tied exclusively to "novelty" - ACM academic conferences - but others to "incremental advancements" - the bigger industry conferences in security, like Usenix Security and some to "best explaining ideas" - like Enigma. There are new ways to find an audience for your work and create impact - that's part of the job now.

blagie 4y ago
I'm well-known in a research community. I'm positioned such that I don't need more academic points. I've mostly stopped publishing in branded prestige academic venues, in part due to rejection rates. My goal in doing work and writing papers is to see them disseminated. The acceptance/rejection process is asinine -- studies show it's basically random. I've had one paper in my whole career where the reviewers did a proper review (e.g. worked through the math). The rest were quick skims. Comments often show the reviewers never read the paper. The stuff that makes it through this process is often nonsense, while very high-quality work is often cut. The very best paper I wrote in my career has never seen the light of day. It was shortened to a 4-page work-in-progress because a reviewer didn't read it (the feedback was literally nonsense: that the sample size was small enough to be anecdotal; I had the largest sample size in the history of the research field). The only impact of this egoistical search for prestige-by-low-accept-rates is that people who have better things to do with their time leave, and that research dissemination is slowed. Those excuses make little sense in the real world: 1) If your conference has a 10% accept rate, it's easy enough to book a bigger venue next year. I've been to conferences with dozens of people, and ones with tens of thousands. It all works well. Bigger ones work better, if anything. 2) PCs aren't thoughtful enough to do that well, and even so, the goal of a conference shouldn't be to select things which resonate with the entrenched PC. That's why many ideas need to wait for a generation of old, conservative professors to die to make it out there. 3) The whole obsession with prestige is stupid and misguided. Journals and conferences ought to have quality bars. Are there typos and grammar errors? Were there clear IRB ethic violations? Did you use error bars on your plots? Was data fabricated? Is the research methodologically sound? Is it coherent and readable? And so on. If it passes those bars, it should be published. If no one reads it / attends a talk, that's okay too -- importance can and should be determined after-the-fact.
- BeetleB 4y ago
  > Comments often show the reviewers never read the paper. This. I was not in computer science, but in a different technical field, and this is sadly common. We would often have to appeal to the editor with "The topic the reviewer said we didn't address? It's in Section X. Get us another reviewer."
  - gautamdivgi 4y ago
    I think this happens in all fields. It’s probably a professor on a PC dumping the review on an unsuspecting and overworked PhD student or MS student who really doesn’t care and just wants to get some sleep. And yes - say what one may - PhD students are overworked and underpaid at least in most of the US.
  - whatshisface 4y ago
    The biggest mystery in the whole thing is why someone who is volunteering to review papers anonymously would bother to do it badly when they could simply not do it at all.
    - BeetleB 4y ago
      Because they want to appear as if they are an active participant in the community.
    - blagie 4y ago
      No mystery. Behavior converges to incentives: * You do get academic points for chairing a conference, and as a chair, you do need to find reviewers. * A colleague is running a conference, and asks you to do a favor. You want to help your colleague. Reviewing papers wins you points with them, and declining to review burns bridges. When you're running a conference, you'd like them to reciprocate. Plus, they might be on a grant / hiring / etc. board / committee / etc. later on. Burning bridges in academia is very bad. On the other hand, there is no incentive to invest more than 30-600 seconds per review. Neither you nor your friend really have any reason to care about the quality of the conference. As this process repeats, people put in less and less time each time around, since it doesn't matter. The process converges to random noise.
theaeolist 4y ago
> 3. While the size of any one conference has to be fixed, tech has made it infinitely easier to create new conferences and journals with other focus areas. They may not start with the prestige of a larger journal, but if the papers published start to have an impact, it can catalyze an entire subfield of work. Does it though? The largest conferences I go to as a CS academic have hundreds of people. There are academic areas where 10x people participate. The size limitation is a self-imposed excuse to keep acceptance low. I have been PC chair of two conferences and my attempts to expand the conference numbers were shot down by the steering committee precisely for this reason, not because we couldn't find a larger room.
kleingeld 4y ago
Try "publishing" in https://researchers.one https://researchers.one

by ath0 4y ago

blagie 4y ago

I'm well-known in a research community. I'm positioned such that I don't need more academic points. I've mostly stopped publishing in branded prestige academic venues, in part due to rejection rates. My goal in doing work and writing papers is to see them disseminated. The acceptance/rejection process is asinine -- studies show it's basically random. I've had one paper in my whole career where the reviewers did a proper review (e.g. worked through the math). The rest were quick skims. Comments often show the reviewers never read the paper. The stuff that makes it through this process is often nonsense, while very high-quality work is often cut. The very best paper I wrote in my career has never seen the light of day. It was shortened to a 4-page work-in-progress because a reviewer didn't read it (the feedback was literally nonsense: that the sample size was small enough to be anecdotal; I had the largest sample size in the history of the research field). The only impact of this egoistical search for prestige-by-low-accept-rates is that people who have better things to do with their time leave, and that research dissemination is slowed. Those excuses make little sense in the real world: 1) If your conference has a 10% accept rate, it's easy enough to book a bigger venue next year. I've been to conferences with dozens of people, and ones with tens of thousands. It all works well. Bigger ones work better, if anything. 2) PCs aren't thoughtful enough to do that well, and even so, the goal of a conference shouldn't be to select things which resonate with the entrenched PC. That's why many ideas need to wait for a generation of old, conservative professors to die to make it out there. 3) The whole obsession with prestige is stupid and misguided. Journals and conferences ought to have quality bars. Are there typos and grammar errors? Were there clear IRB ethic violations? Did you use error bars on your plots? Was data fabricated? Is the research methodologically sound? Is it coherent and readable? And so on. If it passes those bars, it should be published. If no one reads it / attends a talk, that's okay too -- importance can and should be determined after-the-fact.

BeetleB 4y ago
> Comments often show the reviewers never read the paper. This. I was not in computer science, but in a different technical field, and this is sadly common. We would often have to appeal to the editor with "The topic the reviewer said we didn't address? It's in Section X. Get us another reviewer."
- gautamdivgi 4y ago
  I think this happens in all fields. It’s probably a professor on a PC dumping the review on an unsuspecting and overworked PhD student or MS student who really doesn’t care and just wants to get some sleep. And yes - say what one may - PhD students are overworked and underpaid at least in most of the US.
- whatshisface 4y ago
  The biggest mystery in the whole thing is why someone who is volunteering to review papers anonymously would bother to do it badly when they could simply not do it at all.
  - BeetleB 4y ago
    Because they want to appear as if they are an active participant in the community.
  - blagie 4y ago
    No mystery. Behavior converges to incentives: * You do get academic points for chairing a conference, and as a chair, you do need to find reviewers. * A colleague is running a conference, and asks you to do a favor. You want to help your colleague. Reviewing papers wins you points with them, and declining to review burns bridges. When you're running a conference, you'd like them to reciprocate. Plus, they might be on a grant / hiring / etc. board / committee / etc. later on. Burning bridges in academia is very bad. On the other hand, there is no incentive to invest more than 30-600 seconds per review. Neither you nor your friend really have any reason to care about the quality of the conference. As this process repeats, people put in less and less time each time around, since it doesn't matter. The process converges to random noise.

theaeolist 4y ago

> 3. While the size of any one conference has to be fixed, tech has made it infinitely easier to create new conferences and journals with other focus areas. They may not start with the prestige of a larger journal, but if the papers published start to have an impact, it can catalyze an entire subfield of work. Does it though? The largest conferences I go to as a CS academic have hundreds of people. There are academic areas where 10x people participate. The size limitation is a self-imposed excuse to keep acceptance low. I have been PC chair of two conferences and my attempts to expand the conference numbers were shot down by the steering committee precisely for this reason, not because we couldn't find a larger room.

kleingeld 4y ago

Try "publishing" in https://researchers.one https://researchers.one