The difference between traditional software and LLMs is the generality of their intelligence, and there are formal definitions of general intelligence such as h

The difference between traditional software and LLMs is the generality of their intelligence, and there are formal definitions of general intelligence such as https://en.m.wikipedia.org/wiki/AIXI https://en.m.wikipedia.org/wiki/AIXI AIXI is a definition of the optimal agent and is hence uncomputable but LLMs are approximations which are approaching AIXI. I recommend Fridman's interview with Hutter.