Tom Baker didn't like Graham Williams much either, and I would argue the decline truly started during the Williams panto era. If anything, JNT arrested the decline, for a while anyway. I doubt the show would have lasted another ten years if Graham Williams had been running it.
From watching repeats on UK Gold many years ago and buying the DVD's it seems obvious to me that is when the decline began as well but oddly I don't think this was reflected in ratings was it? I think Leela and K-9 in particular were popular companions.
In my opinion you can place the decline of Doctor Who firmly with the Deadly Assassin story because it was the cliffhanger, with the Doctor drowning, at the end of episode three that prompted Mary Whitehouse to complain, which in turn prompted the BBC to bring in the comedy elements that followed after.