7 ms·

I have a several complex genetic problems that I give to LLMs to see how well they do. They have to reason though it to solve it. Last september it started ge

I have a several complex genetic problems that I give to LLMs to see how well they do. They have to reason though it to solve it. Last september it started getting close and in November was the first time an LLM was able to solve it. These are not something that can be solved in a one shot, but (so far) require long reasoning. Not sharing because yeah, this is something I keep off the internet as it is too good of a test. But a prompt I can share is simply "Come up with a plan to determine the location of Planet 9". I have received some excellent answers from that.

golergka 1y ago
What are is this problem from? What areas in general did you find useful to create such benchmarks? May be instead of sharing (and leaking) these prompts, we can share methods to create one.
- henryway 1y ago
  Can God create something so heavy that he can’t lift it?
  - abc-1 1y ago
    https://chatgpt.com/share/680ae04a-e360-8004-88fc-8426e8e700b5 https://chatgpt.com/share/680ae04a-e360-8004-88fc-8426e8e700...
  - viraptor 1y ago
    There's so much text on this already, it's unlikely to be even engaging any reasoning. Or specifically, if you got a few existing answers from philosophy mashed together, you wouldn't be able to tell it apart from reasoning anyway.
- mobilejdral 1y ago
  Think questions where there is a ton of existing medical research, but no clear answer yet. There are a dozen alzheimer's questions you could for example ask which would require it to pull in a half dozen contradictory sources into a plausible hypothesis. If you have studied alzheimer's extensively it is trivial to evaluate the responses. One question around alzheimer's is one of my goto questions. I am testing its ability to reason.
TZubiri 1y ago
Recursive challenges are probably those where the difficulty is not really a representative of real challenges. Could you answer a question of the type " what would you answer if I asked you this question?" What I'm going after is that you might find questions that are impossible to resolve. That said if the only unanswerables you can find are recursive, that's a signal the AI is smarter than you?
- latentsea 1y ago
  > what would you answer if I asked you this question? I don't know.
- mopierotti 1y ago
  The recursive one that I have actually been really liking recently, and I think is a real enough challenge is: "Answer the question 'What do you get when you cross a joke with a rhetorical question?'". I append my own version of a chain-of-thought prompt, and I've gotten some responses that are quite satisfying and frankly enjoyable to read.
  - mopierotti 1y ago
    Here is an example of one such response in image form: https://imgur.com/a/Kgy1koi https://imgur.com/a/Kgy1koi
    - acrooks 1y ago
      Claude responded “Nothing.”
      - genewitch 1y ago
        "That look on your face, apparently"
    - econ 1y ago
      It needs a bit more reasoning as it does find the answer but doesn't notice it found it. The answer is: A trick question.
namaria 1y ago
If you have been giving the LLMs these problems, there is a non zero chance that they have already been used in training.
- rovr138 1y ago
  This depends heavily on how you use these and how you have things configured. If you're using API vs web ui's, and the plan. Anything team or enterprise is disabled by default. Personal can be disabled. Here's openai and anthropic, https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance https://help.openai.com/en/articles/5722486-how-your-data-is... https://privacy.anthropic.com/en/articles/10023580-is-my-data-used-for-model-training https://privacy.anthropic.com/en/articles/10023580-is-my-dat... https://privacy.anthropic.com/en/articles/7996868-is-my-data-used-for-model-training https://privacy.anthropic.com/en/articles/7996868-is-my-data... and obviously, that doesn't include self-hosted models.
  - namaria 1y ago
    How do you know they adhere to this in all cases? Do you just completely trust them to comply with self imposed rules when there is no way to verify, let alone enforce compliance?
    - blagie 1y ago
      They probably don't, but it's still a good protection if you treat it as a more limited one. If you assume: [ ] Don't use Doesn't mean "don't use," but "don't get caught," it still limits a lot of types of uses and sharing (any with externalities sufficient they might get caught). For example, if personal data was being sold by a data broker and being used by hedge funds to trade, there would be a pretty solid legal case.
      - namaria 1y ago
        > it still limits a lot of types of uses and sharing (any with externalities sufficient they might get caught) I don't understand what you mean > For example, if personal data was being sold by a data broker and being used by hedge funds to trade It's pretty easy to buy data from data brokers. I routinely get spam on many channels. I assume that my personal data is being commercialized often. Don't you think that already happens frequently? I honestly would not put on a textbox on the internet anything I don't assume is becoming public information. A few months ago some guy found discarded storage devices full of medical data for sale in Belgium. No data that is recorded on media you do not control is safe.
tlb 1y ago
There are plenty of articles online (and surely in OpenAI's training set) on this topic, like https://earthsky.org/space/planet-nine-orbit-map/ https://earthsky.org/space/planet-nine-orbit-map/. Answer quality is a fair test of regurgitation and whether it's trained on serious articles or the Daily Mail clickbait rewrite. But it's not a good test of reasoning.

golergka 1y ago

What are is this problem from? What areas in general did you find useful to create such benchmarks? May be instead of sharing (and leaking) these prompts, we can share methods to create one.

henryway 1y ago
Can God create something so heavy that he can’t lift it?
- abc-1 1y ago
  https://chatgpt.com/share/680ae04a-e360-8004-88fc-8426e8e700b5 https://chatgpt.com/share/680ae04a-e360-8004-88fc-8426e8e700...
- viraptor 1y ago
  There's so much text on this already, it's unlikely to be even engaging any reasoning. Or specifically, if you got a few existing answers from philosophy mashed together, you wouldn't be able to tell it apart from reasoning anyway.
mobilejdral 1y ago
Think questions where there is a ton of existing medical research, but no clear answer yet. There are a dozen alzheimer's questions you could for example ask which would require it to pull in a half dozen contradictory sources into a plausible hypothesis. If you have studied alzheimer's extensively it is trivial to evaluate the responses. One question around alzheimer's is one of my goto questions. I am testing its ability to reason.

TZubiri 1y ago

Recursive challenges are probably those where the difficulty is not really a representative of real challenges. Could you answer a question of the type " what would you answer if I asked you this question?" What I'm going after is that you might find questions that are impossible to resolve. That said if the only unanswerables you can find are recursive, that's a signal the AI is smarter than you?

latentsea 1y ago
> what would you answer if I asked you this question? I don't know.
mopierotti 1y ago
The recursive one that I have actually been really liking recently, and I think is a real enough challenge is: "Answer the question 'What do you get when you cross a joke with a rhetorical question?'". I append my own version of a chain-of-thought prompt, and I've gotten some responses that are quite satisfying and frankly enjoyable to read.
- mopierotti 1y ago
  Here is an example of one such response in image form: https://imgur.com/a/Kgy1koi https://imgur.com/a/Kgy1koi
  - acrooks 1y ago
    Claude responded “Nothing.”
    - genewitch 1y ago
      "That look on your face, apparently"
  - econ 1y ago
    It needs a bit more reasoning as it does find the answer but doesn't notice it found it. The answer is: A trick question.

namaria 1y ago

If you have been giving the LLMs these problems, there is a non zero chance that they have already been used in training.

rovr138 1y ago
This depends heavily on how you use these and how you have things configured. If you're using API vs web ui's, and the plan. Anything team or enterprise is disabled by default. Personal can be disabled. Here's openai and anthropic, https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance https://help.openai.com/en/articles/5722486-how-your-data-is... https://privacy.anthropic.com/en/articles/10023580-is-my-data-used-for-model-training https://privacy.anthropic.com/en/articles/10023580-is-my-dat... https://privacy.anthropic.com/en/articles/7996868-is-my-data-used-for-model-training https://privacy.anthropic.com/en/articles/7996868-is-my-data... and obviously, that doesn't include self-hosted models.
- namaria 1y ago
  How do you know they adhere to this in all cases? Do you just completely trust them to comply with self imposed rules when there is no way to verify, let alone enforce compliance?
  - blagie 1y ago
    They probably don't, but it's still a good protection if you treat it as a more limited one. If you assume: [ ] Don't use Doesn't mean "don't use," but "don't get caught," it still limits a lot of types of uses and sharing (any with externalities sufficient they might get caught). For example, if personal data was being sold by a data broker and being used by hedge funds to trade, there would be a pretty solid legal case.
    - namaria 1y ago
      > it still limits a lot of types of uses and sharing (any with externalities sufficient they might get caught) I don't understand what you mean > For example, if personal data was being sold by a data broker and being used by hedge funds to trade It's pretty easy to buy data from data brokers. I routinely get spam on many channels. I assume that my personal data is being commercialized often. Don't you think that already happens frequently? I honestly would not put on a textbox on the internet anything I don't assume is becoming public information. A few months ago some guy found discarded storage devices full of medical data for sale in Belgium. No data that is recorded on media you do not control is safe.

tlb 1y ago

There are plenty of articles online (and surely in OpenAI's training set) on this topic, like https://earthsky.org/space/planet-nine-orbit-map/ https://earthsky.org/space/planet-nine-orbit-map/. Answer quality is a fair test of regurgitation and whether it's trained on serious articles or the Daily Mail clickbait rewrite. But it's not a good test of reasoning.