I think the story behind Jon Pertwee is that he was afraid of becoming typecast after Dr Who. He had been considering leaving the show for that reason, and then the shock of Roger Delgado's death (his co-star and a close friend) helped him to make his mind up. I can imagine him being eager to dissociate himself with Dr Who outside of the convention circuit. I never met him, but in con-land stories about him are pretty much universally positive. His feud with Tom Baker is legendary, though.
I got the impression that he liked (I mean in a quite innocent way) Katy Manning as well and her decision to move on may have been a factor too.