It is good that someone is having such a deep look. This is not exclusive to LLMs in the least. Every non-trivial program depends on hundreds of little details

It is good that someone is having such a deep look. This is not exclusive to LLMs in the least. Every non-trivial program depends on hundreds of little details being correct. That is also why I recommend including health checks in such programs. Some function that checks that all assumptions hold. That could be an endpoint, an automated test, a periodic diagnostic job, etc...