5 ms·

I fed ChatGPT the map with no other context, just “tell me about this function”. It did a bit of work finding the Jacobean etc and eventually worked out the imp

I fed ChatGPT the map with no other context, just “tell me about this function”. It did a bit of work finding the Jacobean etc and eventually worked out the implications of what it was seeing. It then proceeded to check the arithmetic 4 times, and then decided to do a manual verification using an ad hoc symbolic checker in case its SymPy had been tampered with.

Sophira 2mo ago
Can you please share the chat log for this? I would absolutely love to see this.
- dist-epoch 2mo ago
  It's easy to reproduce, I've fed the example in GPT-5.6 Sol Max and it started multi-checking it in all kinds of ways, with multiple symbolic packages then manual computation, then it did extensive literature search on the subject, looked at tens of math websites, extensive arxiv research. this was soon after it was posted, it didn't find the original twits with the finding
  - nujabe 2mo ago
    I’m sure it’s trivial to reproduce but we don’t want to waste those tokens
  - eru 2mo ago
    It was easy to reproduce. It's no longer so, because the web search part of what you described is 'poisoned' now. Partially by this very discussion we are having here.
- thomascountz 2mo ago
  Not OP, but here's Gemini's reaction: https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%5B%221WfqIwRCfmKmHoXueS9WHMbywpUkaw5Ya%22%5D,%22action%22:%22open%22,%22userId%22:%22115652185538866555952%22,%22resourceKeys%22:%7B%7D%7D&usp=sharing https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%...
ethbr1 2mo ago
Skepticism is the flipside of knowledge. If an LLM has knowledge encoded inside it (and it's hard to argue it doesn't), then cognitive dissonance can be experienced. And once experienced, must be dealt with, especially in longer-running agentic loops. A friend was joking the other day about sending some messages under a previously-used Slack identity for an agent (since turned off), then asking the agent about the messages. The agent maintained it hadn't sent those messages (no memory) and then was forced to reconcile the idea that the messages indeed appeared to come from it. Its extremely-agitated conclusion was that there had been a security breach and the entire network should be locked down.
- zamadatix 2mo ago
  Another way to look at it is the LLM is, by definition, what's expected to be probable based on the training data and this, by the same definition, is extremely unlikely data to run across. With high uncertainty comes the need to verify until it can level out as "really surprising" instead of "plausible sounding error".

Sophira 2mo ago

Can you please share the chat log for this? I would absolutely love to see this.

dist-epoch 2mo ago
It's easy to reproduce, I've fed the example in GPT-5.6 Sol Max and it started multi-checking it in all kinds of ways, with multiple symbolic packages then manual computation, then it did extensive literature search on the subject, looked at tens of math websites, extensive arxiv research. this was soon after it was posted, it didn't find the original twits with the finding
- nujabe 2mo ago
  I’m sure it’s trivial to reproduce but we don’t want to waste those tokens
- eru 2mo ago
  It was easy to reproduce. It's no longer so, because the web search part of what you described is 'poisoned' now. Partially by this very discussion we are having here.
thomascountz 2mo ago
Not OP, but here's Gemini's reaction: https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%5B%221WfqIwRCfmKmHoXueS9WHMbywpUkaw5Ya%22%5D,%22action%22:%22open%22,%22userId%22:%22115652185538866555952%22,%22resourceKeys%22:%7B%7D%7D&usp=sharing https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%...

ethbr1 2mo ago

Skepticism is the flipside of knowledge. If an LLM has knowledge encoded inside it (and it's hard to argue it doesn't), then cognitive dissonance can be experienced. And once experienced, must be dealt with, especially in longer-running agentic loops. A friend was joking the other day about sending some messages under a previously-used Slack identity for an agent (since turned off), then asking the agent about the messages. The agent maintained it hadn't sent those messages (no memory) and then was forced to reconcile the idea that the messages indeed appeared to come from it. Its extremely-agitated conclusion was that there had been a security breach and the entire network should be locked down.

zamadatix 2mo ago
Another way to look at it is the LLM is, by definition, what's expected to be probable based on the training data and this, by the same definition, is extremely unlikely data to run across. With high uncertainty comes the need to verify until it can level out as "really surprising" instead of "plausible sounding error".