The AI being wrong problem is probably not insurmountable. Humans have meta-cognition that helps them judge if they're doing a thing with lots of assumptions v

The AI being wrong problem is probably not insurmountable. Humans have meta-cognition that helps them judge if they're doing a thing with lots of assumptions vs doing something that's blessed. Humans decouple planning from execution right? Not fully but we choose when to separate it and when to not. If we had enough data on here's a good plan given user context and here's a bad plan, it doesn't seem unreasonable to have a pretty reliable meta cognition capability on the goodness of a plan.