That assumes that there's no other way to get your password than by accessing the contents of your password manager. The service itself could have its passwords

That assumes that there's no other way to get your password than by accessing the contents of your password manager. The service itself could have its passwords/hashes leaked, and people unfortunately do reuse passwords across services even with a password manager, so it's very plausible for someone to get your password but not the answers to your questions.