6 ms·
This makes a good benchmark LLMs: ``` look at this paper: https://arxiv.org/pdf/2603.21852 https://arxiv.org/pdf/2603.21852 now please produce 2x+y as a compo
by eugene3306 5mo ago
This makes a good benchmark LLMs:
```
look at this paper: https://arxiv.org/pdf/2603.21852 https://arxiv.org/pdf/2603.21852
now please produce 2x+y as a composition on EMLs
```
Opus(paid) - claimed that "2" is circular. Once I told it that ChatGPT have already done this, finished successfully.
ChatGPT(free) - did it from the first try.
Grok - produced estimation of the depth of the formula.
Gemini - success
Deepseek - Assumed some pre-existing knowledge on what EML is. Unable to fetch the pdf from the link, unable to consume pdf from "Attach file"
Kimi - produced long output, stopped and asked to upgrade
GLM - looks ok
- fc417fc802 5mo ago> Once I told it that ChatGPT have already done this, finished successfully. TIL you can taunt LLMs. I guess they exhibit more competitive spirit than I thought.
- varispeed 5mo agoOpus seems to be wired currently to get you to spend more money. Once you tell it "Stop defrauding me, just get to the right solution" it often gets it.
- DoctorOetker 5mo ago[dead]
- spuz 5mo agoSo what is the correct answer?
- eru 5mo agoI copy and pasted the abstract into DeepSeek and asked your question. It's a bit unfair to penalise it for not knowing PDFs. It got a result.
- theanonymousone 5mo agoI changed the prompt to this: """ Consider a mathematical function EML defined as `eml(x,y)=exp(x)−ln(y)` Please produce `sin(x)/x` as a composition on EMLs and constant number 1 (one). """
- XCSme 5mo ago[dead]
- stared 5mo agoIf you like creating such things, consider contributing to Terminal Bench Science, https://www.tbench.ai/news/tb-science-announcement https://www.tbench.ai/news/tb-science-announcement.
- brrrrrm 5mo agometa.ai in instant mode gets it first try too (I think?) ``` 2x + y = \operatorname{eml}\Big(1,\; \operatorname{eml}\big(\operatorname{eml}(1,\; \operatorname{eml}(\operatorname{eml}(1,\; \operatorname{eml}(\operatorname{eml}(L_2 + L_x, 1), 1) \cdot \operatorname{eml}(y,1)),1)\big),1\big)\Big) ``` for me Gemini hallucinated EML to mean something else despite the paper link being provided: "elementary mathematical layers"
- aixpert 5mo agothis should be a tangential proof for the dying bunch of people who still believe that LLMs are just parrots. EML are literally a new invention