27 ms·

I'm a bit surprised it gets this question wrong (ChatGPT gets it right, even on instant). All the pre-reasoning models failed this question, but it's seemed sol

I'm a bit surprised it gets this question wrong (ChatGPT gets it right, even on instant). All the pre-reasoning models failed this question, but it's seemed solved since o1, and Sonnet 4.5 got it right. https://claude.ai/share/876e160a-7483-4788-8112-0bb4490192af https://claude.ai/share/876e160a-7483-4788-8112-0bb4490192af This was sonnet 4.6 with extended thinking.

layer8 7mo ago
Off-by-one errors are one of the hardest problems in computer science.
- anonymous908213 7mo ago
  That is not an off-by-one error in a computer science sense, nor is it "one of the hardest problems in computer science".
  - layer8 7mo ago
    This was in reference to a well-known joke, see here: https://martinfowler.com/bliki/TwoHardThings.html https://martinfowler.com/bliki/TwoHardThings.html
malfist 7mo ago
Chatgpt doesn't get it right: https://chatgpt.com/share/6994c312-d7dc-800f-976a-5e4fbec0ae5d https://chatgpt.com/share/6994c312-d7dc-800f-976a-5e4fbec0ae... ``` Use digit concatenation plus addition: 888 + 88 + 8 + 8 + 8 = 1000 Digit count: 888 → three 8s 88 → two 8s 8 + 8 + 8 → three 8s Total: 3 + 2 + 3 = 9 eights Operation used: addition only ``` Love the 3 + 2 + 3 = 9
- simianwords 7mo ago
  chatgpt gets it right. maybe you are using free or non thinking version? https://chatgpt.com/share/6994d25e-c174-800b-987e-9d32c94d9599 https://chatgpt.com/share/6994d25e-c174-800b-987e-9d32c94d95...
bobbylarrybobby 7mo ago
Interesting, my sonnet 4.6 starts with the following: The classic puzzle actually uses *eight 8s*, not nine. The unique solution is: 888+88+8+8+8=1000. Count: 3+2+1+1+1=8 eights. It then proves that there is no solution for nine 8s. https://claude.ai/share/9a6ee7cb-bcd6-4a09-9dc6-efcf0df6096b https://claude.ai/share/9a6ee7cb-bcd6-4a09-9dc6-efcf0df6096b (for whatever reason the LaTeX rendering is messed up in the shared chat, but it looks fine for me).
- stevepike 7mo ago
  Yeah, earlier in the GPT days I felt like this was a good example of LLMs being "a blurry jpeg of the web", since you could give them something that was very close to an existing puzzle that exists commonly on the web, and they'd regurgitate an answer from that training set. It was neat to me to see the question get solved consistently by the reasoning models (though often by churning a bunch of tokens trying and verifying to count 888 + 88 + 8 + 8 + 8 as nine digits). I wonder if it's a temperature thing or if things are being throttled up/down on time of day. I was signed in to a paid claude account when I ran the test.
leumon 7mo ago
My locally running nemotron-3-nano quantized to Q4_K_M gets this right. (although it used 20k thought tokens before answering the question)
deleted 7mo ago
[deleted]

layer8 7mo ago

Off-by-one errors are one of the hardest problems in computer science.

anonymous908213 7mo ago
That is not an off-by-one error in a computer science sense, nor is it "one of the hardest problems in computer science".
- layer8 7mo ago
  This was in reference to a well-known joke, see here: https://martinfowler.com/bliki/TwoHardThings.html https://martinfowler.com/bliki/TwoHardThings.html

malfist 7mo ago

Chatgpt doesn't get it right: https://chatgpt.com/share/6994c312-d7dc-800f-976a-5e4fbec0ae5d https://chatgpt.com/share/6994c312-d7dc-800f-976a-5e4fbec0ae... ``` Use digit concatenation plus addition: 888 + 88 + 8 + 8 + 8 = 1000 Digit count: 888 → three 8s 88 → two 8s 8 + 8 + 8 → three 8s Total: 3 + 2 + 3 = 9 eights Operation used: addition only ``` Love the 3 + 2 + 3 = 9

simianwords 7mo ago
chatgpt gets it right. maybe you are using free or non thinking version? https://chatgpt.com/share/6994d25e-c174-800b-987e-9d32c94d9599 https://chatgpt.com/share/6994d25e-c174-800b-987e-9d32c94d95...

bobbylarrybobby 7mo ago

Interesting, my sonnet 4.6 starts with the following: The classic puzzle actually uses *eight 8s*, not nine. The unique solution is: 888+88+8+8+8=1000. Count: 3+2+1+1+1=8 eights. It then proves that there is no solution for nine 8s. https://claude.ai/share/9a6ee7cb-bcd6-4a09-9dc6-efcf0df6096b https://claude.ai/share/9a6ee7cb-bcd6-4a09-9dc6-efcf0df6096b (for whatever reason the LaTeX rendering is messed up in the shared chat, but it looks fine for me).

stevepike 7mo ago
Yeah, earlier in the GPT days I felt like this was a good example of LLMs being "a blurry jpeg of the web", since you could give them something that was very close to an existing puzzle that exists commonly on the web, and they'd regurgitate an answer from that training set. It was neat to me to see the question get solved consistently by the reasoning models (though often by churning a bunch of tokens trying and verifying to count 888 + 88 + 8 + 8 + 8 as nine digits). I wonder if it's a temperature thing or if things are being throttled up/down on time of day. I was signed in to a paid claude account when I ran the test.

leumon 7mo ago

My locally running nemotron-3-nano quantized to Q4_K_M gets this right. (although it used 20k thought tokens before answering the question)

deleted 7mo ago

[deleted]