I assume that GPT-3 is just exhibiting rote memory. For small numbers, it has accurate answers memorized from the training set, but for larger numbers, it just

I assume that GPT-3 is just exhibiting rote memory. For small numbers, it has accurate answers memorized from the training set, but for larger numbers, it just "remembers" whatever is close... hence the ability to estimate. My take is not that GPT-3 can run code, but rather that GPT-3 has memorized what code looks like and what the output looks like.