24 ms·

Thanks everyone for engagement and discussion. Following the range of comments, just a few thoughts: 1. Traceability, transparency and verifiability. I think

Thanks everyone for engagement and discussion. Following the range of comments, just a few thoughts: 1. Traceability, transparency and verifiability. I think the key question for me is not only whether AI can accelerate science, but rather how we can use AI to accelerate science while at the same time enhancing key scientific values, like transparency, traceability and verifiability. More and more these days when I read scientific papers, published either at high impact journals or at more specialized journals, I find it so hard, and sometimes even frustratingly impossible, to understand and check what exactly was done to analyze the raw data and get to the key results, what was the specific chain of analysis steps, what parameters where used, etc, etc. The data is often not there or is poorly annotated, the analysis is explained poorly, the code is missing or is impossible to track, etc etc. At all, it became practically impossible to repeat and check the analysis and the results of many peer reviewed publications. Why are papers so hard to follow and trace? Because writing clear and fully traceable and transparent papers is very hard, and we don’t have powerful tools for doing this, and it requires the scientific process itself (or at least the data analysis part) to be done in an organized and fully traceable way. Our data-to-paper approach is designed to provide ways to use AI powerfully, not only to speed up science (by a lot!), but also at the same time to use AI to enhance transparency, traceability and verifiability. Data-to-paper sets a standard for traceability and verifiability which imo exceeds the current level of human created manuscripts. In particular: 1. “Data-Chaining": by tracing information flow through the research steps, data-to-paper creates what we call “data-chained” manuscripts, where results, methodology and data are programmatically linked. See this video (https://youtu.be/mHd7VOj7Q-g https://youtu.be/mHd7VOj7Q-g). You can also try click-tracing results in this example ms: https://raw.githubusercontent.com/rkishony/data-to-paper-supplementary/main/Supplementary%20Data-chained%20Manuscripts/Supplementary%20Data-chained%20Manuscript%20A.pdf https://raw.githubusercontent.com/rkishony/data-to-paper-sup... See more about this and more examples in our preprint: https://arxiv.org/abs/2404.17605 https://arxiv.org/abs/2404.17605 2. Human in the loop. We are looking at different ways to create a co-piloted environment where human scientists can direct and oversee the process. We currently have a co-pilot app that allows users to follow the process, to set and change prompts and to provide review comments at the end of each steps (https://youtu.be/Nt_460MmM8k https://youtu.be/Nt_460MmM8k). Will be great to get feedback (and help!) on ways in which this could be enhanced. 3. P-value hacking. Data-to-paper is designed to raise an hypothesis (autonomously, or by user input) and then go through the research steps to test the hypothesis. If the hypothesis test is negative, it is perfectly fine and suitable to write a negative-result manuscript. In fact, in one of the tests that we have done we gave it data of a peer reviewed publication that reports a positive and a negative result and data-to-paper created manuscripts that correctly report both of these results. So data-to-paper on its own is not doing multiple hypothesis searches. In fact it can help you realize just how many hypotheses you have actually tested (something very hard for human research even when done honestly). Can people ask data-to-paper to create 1000 papers and then read them all and choose only the single one in which a positive result is found? Yes - people can always cheat and science is built on trust, but it is not going to be particularly easier than any other of the many ways available for people to cheat if they want. 4. Final note: LLMs are here and are here to stay and are already used extensively in science doing (sadly sometimes undisclosed: https://retractionwatch.com/papers-and-peer-reviews-with-evidence-of-chatgpt-writing/ https://retractionwatch.com/papers-and-peer-reviews-with-evi...). The new models of ChatGPT5, ChatGPT6, ... will likely write a whole manuscript for you in just a single prompt. So the question is not whether AI will go into science (it already does), but rather how to do so and use AI in ways that fosters, not jeopardizes, accountability, transparency, verifiability and other important scientific values. This is what we are trying to do with data-to-paper. We hope our project stimulates further discussions on how to harness AI in science while preserving and enhancing key scientific values.

uniqueuid 2y ago
Hi, thanks for the honest and thoughtful discussion you are conducting here. Comments tend to be simplistic and it's great to see that you raise the bar by addressing criticism and questions in earnest! That said, I think the fundamental problem of such tools is unsolvable: Out of all possible analytical designs, they create boring existing results at best, and wrong results (i.e. missing confounders, misunderstanding context ...) as the worst outcome. They also pollute science with harmful findings that lack meaning in the context of a field. These issues have been well-known for about ten years and are explained excellently e.g in papers such as [1]. There is really one way to guard against bad science today, and that is true pre-registration. And that is something which LLMs fundamentally cannot do. So while tools such as data-to-paper may be helpful, they can only be so in the context of pre-registered hypotheses where they follow a path pre-defined by humans before collecting data. [1] http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_...
- alchemist1e9 2y ago
  > That said, I think the fundamental problem of such tools is unsolvable: Out of all possible analytical designs, they create boring existing results at best, and wrong results (i.e. missing confounders, misunderstanding context ...) as the worst outcome. They also pollute science with harmful findings that lack meaning in the context of a field. This doesn't seem correct to me at all. If new data is provided and the LLM is simply an advanced tool that applies known analysis techniques to the data, then why would they create “boring existing results”? I don’t see why systems using an advanced methodology should not produce novel and new results when provided new data. There is a lot of reactionary or even luddite responses to the direction we are headed with LLMs.
  - uniqueuid 2y ago
    Sorry but I think we have very different perspectives here. I assume you mean that LLMs can generate new insights in the sense of producing plausible results from new data or in the sense of producing plausible but previously unknown results from old data. Both these things are definitely possible, but they are not necessarily (and in fact often not) good science. Insights in science are not rare. There are trillions of plausible insights, and all can be backed by data. The real problem is the reverse: Finding a meaningful and useful finding in a sea of billion other ones. LLMs learn from past data, and that means they will have more support for "boring", i.e. conventional hypotheses, which have precedent in training material. So I assume that while they can come up with novel hypotheses and results, these results will probably tend to conform to a (statistically defined) paradigm of past findings. When they produce novel hypotheses or findings, it is unlikely that they will create genuinely meaningful AND true insights. Because if you randomly generate new ideas, almost all of them are wrong (see the papers I linked). So in essence, LLMs should have a hard time doing real science, because real science is the complex task of finding unlikely, true, and interesting things.

uniqueuid 2y ago

Hi, thanks for the honest and thoughtful discussion you are conducting here. Comments tend to be simplistic and it's great to see that you raise the bar by addressing criticism and questions in earnest! That said, I think the fundamental problem of such tools is unsolvable: Out of all possible analytical designs, they create boring existing results at best, and wrong results (i.e. missing confounders, misunderstanding context ...) as the worst outcome. They also pollute science with harmful findings that lack meaning in the context of a field. These issues have been well-known for about ten years and are explained excellently e.g in papers such as [1]. There is really one way to guard against bad science today, and that is true pre-registration. And that is something which LLMs fundamentally cannot do. So while tools such as data-to-paper may be helpful, they can only be so in the context of pre-registered hypotheses where they follow a path pre-defined by humans before collecting data. [1] http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_...

alchemist1e9 2y ago
> That said, I think the fundamental problem of such tools is unsolvable: Out of all possible analytical designs, they create boring existing results at best, and wrong results (i.e. missing confounders, misunderstanding context ...) as the worst outcome. They also pollute science with harmful findings that lack meaning in the context of a field. This doesn't seem correct to me at all. If new data is provided and the LLM is simply an advanced tool that applies known analysis techniques to the data, then why would they create “boring existing results”? I don’t see why systems using an advanced methodology should not produce novel and new results when provided new data. There is a lot of reactionary or even luddite responses to the direction we are headed with LLMs.
- uniqueuid 2y ago
  Sorry but I think we have very different perspectives here. I assume you mean that LLMs can generate new insights in the sense of producing plausible results from new data or in the sense of producing plausible but previously unknown results from old data. Both these things are definitely possible, but they are not necessarily (and in fact often not) good science. Insights in science are not rare. There are trillions of plausible insights, and all can be backed by data. The real problem is the reverse: Finding a meaningful and useful finding in a sea of billion other ones. LLMs learn from past data, and that means they will have more support for "boring", i.e. conventional hypotheses, which have precedent in training material. So I assume that while they can come up with novel hypotheses and results, these results will probably tend to conform to a (statistically defined) paradigm of past findings. When they produce novel hypotheses or findings, it is unlikely that they will create genuinely meaningful AND true insights. Because if you randomly generate new ideas, almost all of them are wrong (see the papers I linked). So in essence, LLMs should have a hard time doing real science, because real science is the complex task of finding unlikely, true, and interesting things.