Tom Baker didn't like Graham Williams much either, and I would argue the decline truly started during the Williams panto era. If anything, JNT arrested the decline, for a while anyway. I doubt the show would have lasted another ten years if Graham Williams had been running it.
JNT did make it darker again, particularly the last Davidson series, which had three excellent stories, and also the violence quotient was upped in the mid eighties. However, he had an even bigger enemy than The Daleks to contend with, Michael Grade, who was determined to get his revenge on the show which gave him a hard time at LWT.