Seemingly the actor who played Adrec told the Daily Mail, Baker was drunk most of the time and when he said he was leaving the role, seemed to get even more drunk and disinterested. Actually the last Tom Baker series was his worst and I wonder if this has anything to do with it.
Matthew Waterhouse. Well, he had his own... issues... He might be right on this, though.
I recall reading an interview with Baker some time ago where he said that he had wanted to go at the end of season 6, but had been persuaded to stay on longer. The implication was that he regretted staying on, and wished that he had kept to his original plan.
I think the feud between Baker and Pertwee predates their time on Dr Who.