This is a good point. We didn't have any cases of GPT-4 failing to try to pretend to be a human, although GPT-3.5 did this quite frequently. I think RLHF cause

This is a good point. We didn't have any cases of GPT-4 failing to try to pretend to be a human, although GPT-3.5 did this quite frequently. I think RLHF causes other issues though (especially with the tone), and we're interested to try some non-RLHF'd open source models.