5 ms·

Scientific Papers: Innovation or Imitation?

66 points by tapanjk 1y ago

birn559 1y ago
The process is far from perfect, but it works well enough mid-term and works pretty well long-term. It's also better than any alternatives, as far as I know. Haven't heard people pushing the idea of restructuring the process, the only exception being that journals shouldn't cost (that much) money and instead institutions should pay for publishing a paper. This wouldn't however change the foundation of the process.
- agumonkey 1y ago
  What about the publish or perish effect ? no ideas on how to rebalance things to avoid it ?
  - Daub 1y ago
    Yes. It's simple. Establish peer review as the metric of tenure and make dam sure those peers know their stuff.
    - friendzis 1y ago
      > Establish peer review as the metric of tenure and make dam sure those peers know their stuff. And you are back at square one: peer reviews become the currency used in academic politics. A relatively small group of tenured academics have all the incentives to independently form a fiefdom. Anonymization does not help as everyone knows work and papers of the rest anyway.
    - ancillary 1y ago
      The supply of knowledgeable and conscientious reviewers in, say, machine learning, is far outmatched by the number of papers less knowledgeable and conscientious people submit.
richarlidad 1y ago
Imitation precedes creation.
flanked-evergl 1y ago
[flagged]
kevinventullo 1y ago
Follow-up papers by other authors which “only extend or expand on the specific finding in very minor ways” have a secondary benefit. In addition to expanding the original findings, they are also implicitly replicating the original result. This is perhaps a crucial contribution in light of the replication crisis!
- Daub 1y ago
  Maybe. But that is a generous reading. I used to attend many computational aesthetic conferences. The sheer volume of non photorealistic rendering cross hatch algorithms was almost laughable.
- tgv 1y ago
  If only. I worked in cog/neuro sci, and the career builders there produce small variations on the original. Variations on the Stroop task, which dates back to 1935(!), are still being published, despite the fact that there is no explanation for the effect. And when you consider that null results are rarely published, and that many aspects of the methodology are flawed, a new paper cannot be considered a replication: it's just wishful thinking upon wishful thinking.
  - kevinventullo 1y ago
    Are you claiming the Stroop effect hasn’t been proven to exist or just that there hasn’t been an explanation? Funnily enough, the first “professional” coding I ever did was writing up a Stroop test in Visual Basic for a neuro professor, and I recall the effect being undeniably clear. At a personal anecdotal level, I would time myself with matching colors versus non-matching, and even with practice I could not bring my non-matching times down to my matching times.
    - tgv 1y ago
      As I wrote: there is no explanation. After 90 years, and thousands of papers (Google scholar counts nearly 90.000), still no idea what causes it. Why go on then? It's obvious that this path isn't going to lead to the solution.
Daub 1y ago
For a few years I worked closely with computer engineers in a S E Asian university. I got to know quite well the sort of stuff they published. Some of the dodgy stuff i saw: Recycling. Some papers seemed to be near duplicates of prior work by the same academic, with minor modification. Faddishnes. Papers featuring the latest buzz technologies regardless of whether they were appropriate. Questionable authorship. Some senior academics would get their name included on publications regardless of whether they had been actively engaged with that project. I saw a few academics get involved in risky and potentially interesting subjects, but they all risked their careers in doing so. But most of all, there was a dearth of true innovation. The university noticed this and established an Innovation Centre. It quickly became full of second hand projects all frustratingly similar to projects in the US from a few years ago. Of course there were exceptions, and learning from them was a genuine growth experience fir which I am grateful.
- bonoboTP 1y ago
  It's not just about the academics but the expectations from higher ups and funding agencies in order to keep your job and have a chance at continuing your career. Over the last few decades the expected amount of papers at good and even mediocre institutions has exploded. Profs who want to be seen as productive and who want good funding publish 30-50 papers per year and sometimes "supervise" dozens of PhD students at the same time (who agree to the deal to get the brand name of the big prof, not for any real supervision). Funding agencies can't evaluate the research itself, so they look at numbers, metrics, impact factors, citations, h-index, publication count etc. They can't simply say "we pay this academic whether he publishes or not because we trust he is still deep in important work when he is not at a work stage to publish" because people will suspect fraud and nepotism and bias, and often the funding is taxpayer money. Not that the metrics prevent that of course. But it seems that way. So metrics it is, so gaming the metrics via Goodhart's law it is. I don't think it's super bad, but it increases administrative work and busywork overhead on top of the actual research. The progress slows somewhat per person, as the same work has to be salami sliced and marketed in chunks, but there's also way more people in it, but of course most of them produce vary low quality stuff but it's not a big loss because these people would not even have published anything some decades ago, they would just have some teaching professorship and publish every few years perhaps just in their national language. It increases the noise but there are ways to find the signal among it, and academics figure out ways to cut through the noise. It's not great, not super easy, and it pushes a lot of people out who dislike the grind but there are plenty who see it as a relatively good deal to move to a richer country and do this.
matusp 1y ago
In my experience, the publication pressure in today's science is to large extent inhibiting innovation. How can you innovate when you need to have X papers every year, otherwise you will not get that position of funding. To fulfill the quota, the only rational strategy is to focus on simple iterative papers that are very similar to what everybody else is doing. There is simply no time to innovate or be brave, you have to comfort. There is also barely time to make sure they what you are doing is actually methodologically correct. If you spend too much time, you will get scooped and forgotten. Case in point, everybody is doing AI research nowadays and NIPS has like 15k submitted papers. But the innovation rate in AI is actually not that much higher than 10 years ago, I would even argue that it is lower. What are all these papers for? They help people build their careers as proofs of work.
- jltsiren 1y ago
  AI is a special case of a special case. First you have the weird CS publication culture with conference papers and a heavy focus on selecting a (small) subset of winners. And then you have a subfield with giant conferences, a lot of money, and a lot of people doing similar things. A typical approach to science is finding your niche and becoming a person known for that thing. You pick something you are interested in, something you are good at, something underexplored, and something close enough to what other people are doing that they can appreciate your work. Then you work on that topic for a number of years and see where you end up in. But you can't do that in AI, because the field is overcrowded.
  - mpascale00 1y ago
    Certainly other field are competitive, but the current AI boom has been ridiculous for a while now. As an outside observer, the competition seems to be for the final money, prestige, or whatever the top papers win, rather than competition at the level of paper acceptance...
    - bonoboTP 1y ago
      The competition racket and inflation keeps turning. It used to be publications. Then it was top conference publications. Now it's going viral on social media, being popularized by big AI aggregators like AK. It's crazy, most Master's students applying for a PhD position already come with multiple top conference papers, which a few years ago would get you like 2/3 of the way to the PhD, and now it just gets you a foot in the door in applying to start a PhD. And then already Bachelor students are expected to publish to get a good spot in a lab to do their Master thesis or internship. And NeurIPS has a track for high school students to write papers, which - I assume - will boost their applications to start university. This type of hustle has been common in many East Asian countries and is getting globalized.
agarttha 1y ago
Random thoughts from physics researcher: - Too much imitation delays innovation. - For all the emphasis on high risk research, the system doesn't reward it. - Creativity isn't valued as much as it should be. - negative results and failed experiments hold back careers but are signs of attempts at innovation - the VC world may understand that only 1/100 projects will be novel and perhaps successful, but funding agencies don't
mpascale00 1y ago
The thesis here is not well elaborated on. Ref [1] for example seems to me to miss more recent progress on our understanding of working memory, and in linguistics, while Chomsky's work is foundational, we have a much better idea now of how the requisite compositionality of behavior necessary for language might arise in the human brain.
- mpascale00 1y ago
  To add to that, I agree with the basic sentiment of the article - but it just doesn't seem to be the reflections of an academic. Ending with optimism about AI, to me, makes me think the author believes AI will solve a problem they are not well acquainted with. Perhaps one expects overgeneralization in consulting blogs though
kj4211cash 1y ago
So much of academic life revolves around bringing in grant money. This is particularly true in STEM fields and at the best research schools. There are ever increasing administrative hoops to jump through to bring in that grant money. And grants nowadays are often given out for research on very specific topics often chosen by bureaucrats. These topics are, almost by definition, not innovative. The NSF is an exception but there are very few NSF grants given out, relative to the number of researchers. My assessment is that the most famous, most published researchers can still afford to explore, if they have the time and inclination, but the rest cannot.
- bonoboTP 1y ago
  Yes, also the ERC grants allow quite broad freedom in the EU. But grants have been annoying for long. Even decades ago the common wisdom, in fields like biology, was to write grant applications promising things you've already done recently, then spend the money on doing something new. But this doesn't work in super fast moving fields like AI/ML where a grant length is an eternity relative to change in the field. Also the bureaucrats want something sexy at the end, so academics overpromise, then there's a bitter taste among the funders that the super fancy thing didn't materialize, high impact factor publications can sweeten this bitterness somewhat, as can awards etc. Also a rising tide lifts all boats so just by keeping up with SOTA, they can woo the bureaucrats.

66 points by tapanjk 1y ago

The process is far from perfect, but it works well enough mid-term and works pretty well long-term. It's also better than any alternatives, as far as I know. Haven't heard people pushing the idea of restructuring the process, the only exception being that journals shouldn't cost (that much) money and instead institutions should pay for publishing a paper. This wouldn't however change the foundation of the process.

agumonkey 1y ago
What about the publish or perish effect ? no ideas on how to rebalance things to avoid it ?
- Daub 1y ago
  Yes. It's simple. Establish peer review as the metric of tenure and make dam sure those peers know their stuff.
  - friendzis 1y ago
    > Establish peer review as the metric of tenure and make dam sure those peers know their stuff. And you are back at square one: peer reviews become the currency used in academic politics. A relatively small group of tenured academics have all the incentives to independently form a fiefdom. Anonymization does not help as everyone knows work and papers of the rest anyway.
  - ancillary 1y ago
    The supply of knowledgeable and conscientious reviewers in, say, machine learning, is far outmatched by the number of papers less knowledgeable and conscientious people submit.

richarlidad 1y ago

Imitation precedes creation.

flanked-evergl 1y ago

[flagged]

kevinventullo 1y ago

Follow-up papers by other authors which “only extend or expand on the specific finding in very minor ways” have a secondary benefit. In addition to expanding the original findings, they are also implicitly replicating the original result. This is perhaps a crucial contribution in light of the replication crisis!

Daub 1y ago
Maybe. But that is a generous reading. I used to attend many computational aesthetic conferences. The sheer volume of non photorealistic rendering cross hatch algorithms was almost laughable.
tgv 1y ago
If only. I worked in cog/neuro sci, and the career builders there produce small variations on the original. Variations on the Stroop task, which dates back to 1935(!), are still being published, despite the fact that there is no explanation for the effect. And when you consider that null results are rarely published, and that many aspects of the methodology are flawed, a new paper cannot be considered a replication: it's just wishful thinking upon wishful thinking.
- kevinventullo 1y ago
  Are you claiming the Stroop effect hasn’t been proven to exist or just that there hasn’t been an explanation? Funnily enough, the first “professional” coding I ever did was writing up a Stroop test in Visual Basic for a neuro professor, and I recall the effect being undeniably clear. At a personal anecdotal level, I would time myself with matching colors versus non-matching, and even with practice I could not bring my non-matching times down to my matching times.
  - tgv 1y ago
    As I wrote: there is no explanation. After 90 years, and thousands of papers (Google scholar counts nearly 90.000), still no idea what causes it. Why go on then? It's obvious that this path isn't going to lead to the solution.

Daub 1y ago

For a few years I worked closely with computer engineers in a S E Asian university. I got to know quite well the sort of stuff they published. Some of the dodgy stuff i saw: Recycling. Some papers seemed to be near duplicates of prior work by the same academic, with minor modification. Faddishnes. Papers featuring the latest buzz technologies regardless of whether they were appropriate. Questionable authorship. Some senior academics would get their name included on publications regardless of whether they had been actively engaged with that project. I saw a few academics get involved in risky and potentially interesting subjects, but they all risked their careers in doing so. But most of all, there was a dearth of true innovation. The university noticed this and established an Innovation Centre. It quickly became full of second hand projects all frustratingly similar to projects in the US from a few years ago. Of course there were exceptions, and learning from them was a genuine growth experience fir which I am grateful.

bonoboTP 1y ago
It's not just about the academics but the expectations from higher ups and funding agencies in order to keep your job and have a chance at continuing your career. Over the last few decades the expected amount of papers at good and even mediocre institutions has exploded. Profs who want to be seen as productive and who want good funding publish 30-50 papers per year and sometimes "supervise" dozens of PhD students at the same time (who agree to the deal to get the brand name of the big prof, not for any real supervision). Funding agencies can't evaluate the research itself, so they look at numbers, metrics, impact factors, citations, h-index, publication count etc. They can't simply say "we pay this academic whether he publishes or not because we trust he is still deep in important work when he is not at a work stage to publish" because people will suspect fraud and nepotism and bias, and often the funding is taxpayer money. Not that the metrics prevent that of course. But it seems that way. So metrics it is, so gaming the metrics via Goodhart's law it is. I don't think it's super bad, but it increases administrative work and busywork overhead on top of the actual research. The progress slows somewhat per person, as the same work has to be salami sliced and marketed in chunks, but there's also way more people in it, but of course most of them produce vary low quality stuff but it's not a big loss because these people would not even have published anything some decades ago, they would just have some teaching professorship and publish every few years perhaps just in their national language. It increases the noise but there are ways to find the signal among it, and academics figure out ways to cut through the noise. It's not great, not super easy, and it pushes a lot of people out who dislike the grind but there are plenty who see it as a relatively good deal to move to a richer country and do this.

matusp 1y ago

In my experience, the publication pressure in today's science is to large extent inhibiting innovation. How can you innovate when you need to have X papers every year, otherwise you will not get that position of funding. To fulfill the quota, the only rational strategy is to focus on simple iterative papers that are very similar to what everybody else is doing. There is simply no time to innovate or be brave, you have to comfort. There is also barely time to make sure they what you are doing is actually methodologically correct. If you spend too much time, you will get scooped and forgotten. Case in point, everybody is doing AI research nowadays and NIPS has like 15k submitted papers. But the innovation rate in AI is actually not that much higher than 10 years ago, I would even argue that it is lower. What are all these papers for? They help people build their careers as proofs of work.

jltsiren 1y ago
AI is a special case of a special case. First you have the weird CS publication culture with conference papers and a heavy focus on selecting a (small) subset of winners. And then you have a subfield with giant conferences, a lot of money, and a lot of people doing similar things. A typical approach to science is finding your niche and becoming a person known for that thing. You pick something you are interested in, something you are good at, something underexplored, and something close enough to what other people are doing that they can appreciate your work. Then you work on that topic for a number of years and see where you end up in. But you can't do that in AI, because the field is overcrowded.
- mpascale00 1y ago
  Certainly other field are competitive, but the current AI boom has been ridiculous for a while now. As an outside observer, the competition seems to be for the final money, prestige, or whatever the top papers win, rather than competition at the level of paper acceptance...
  - bonoboTP 1y ago
    The competition racket and inflation keeps turning. It used to be publications. Then it was top conference publications. Now it's going viral on social media, being popularized by big AI aggregators like AK. It's crazy, most Master's students applying for a PhD position already come with multiple top conference papers, which a few years ago would get you like 2/3 of the way to the PhD, and now it just gets you a foot in the door in applying to start a PhD. And then already Bachelor students are expected to publish to get a good spot in a lab to do their Master thesis or internship. And NeurIPS has a track for high school students to write papers, which - I assume - will boost their applications to start university. This type of hustle has been common in many East Asian countries and is getting globalized.

agarttha 1y ago

Random thoughts from physics researcher: - Too much imitation delays innovation. - For all the emphasis on high risk research, the system doesn't reward it. - Creativity isn't valued as much as it should be. - negative results and failed experiments hold back careers but are signs of attempts at innovation - the VC world may understand that only 1/100 projects will be novel and perhaps successful, but funding agencies don't

mpascale00 1y ago

The thesis here is not well elaborated on. Ref [1] for example seems to me to miss more recent progress on our understanding of working memory, and in linguistics, while Chomsky's work is foundational, we have a much better idea now of how the requisite compositionality of behavior necessary for language might arise in the human brain.

mpascale00 1y ago
To add to that, I agree with the basic sentiment of the article - but it just doesn't seem to be the reflections of an academic. Ending with optimism about AI, to me, makes me think the author believes AI will solve a problem they are not well acquainted with. Perhaps one expects overgeneralization in consulting blogs though

kj4211cash 1y ago

So much of academic life revolves around bringing in grant money. This is particularly true in STEM fields and at the best research schools. There are ever increasing administrative hoops to jump through to bring in that grant money. And grants nowadays are often given out for research on very specific topics often chosen by bureaucrats. These topics are, almost by definition, not innovative. The NSF is an exception but there are very few NSF grants given out, relative to the number of researchers. My assessment is that the most famous, most published researchers can still afford to explore, if they have the time and inclination, but the rest cannot.

bonoboTP 1y ago
Yes, also the ERC grants allow quite broad freedom in the EU. But grants have been annoying for long. Even decades ago the common wisdom, in fields like biology, was to write grant applications promising things you've already done recently, then spend the money on doing something new. But this doesn't work in super fast moving fields like AI/ML where a grant length is an eternity relative to change in the field. Also the bureaucrats want something sexy at the end, so academics overpromise, then there's a bitter taste among the funders that the super fancy thing didn't materialize, high impact factor publications can sweeten this bitterness somewhat, as can awards etc. Also a rising tide lifts all boats so just by keeping up with SOTA, they can woo the bureaucrats.