There was actually a significant drop in ratings between Hinchcliffe and Williams. Williams' first season had significantly lower ratings than the Hinchcliffe seasons. Of course, ratings never tell the full story, there could have been more competition on the other side. Williams' last season had some of the best ratings in Doctor Who history (but then there was nothing on ITV for many weeks due to a strike!)
I think that under Williams, it became The Tom Baker Show rather than Doctor Who. It became sillier and sillier and sillier, and eventually people were only tuning in because they liked Tom Baker and Lalla, or liked Douglas Adams' jokes. You could tell that, too often, directors and actors were not taking the show as seriously as they once did.
Hmmm, a light hearted series where the characters look like they're enjoying themselves and smile and make jokes as they gallavant around the universe vs ponderings on entropy and other hard maths.
I know which I'd rather watch!
I'm sure the kids watching just want to see a bit of fun, not continue the work they were doing at school that day.
Given that era, that budget and the scale of the show, they were probably right to not take things massively seriously as any time they did could've come off even cheesier.