I started watching Doctor Who when I was 6. I can vaguely recall watching an episode of The Planet of the Spiders, but my first proper memories of the show were of The Ark in Space.
Tom Baker was MY Doctor - they always seem to say your first Doctor is YOUR Doctor, and for me that's definitely the case.
Each Doctor following on from Tom had something I liked about them, and some had things I disliked. I think my least favourites were Colin Baker and Sylvester McCoy, but they each had stories I liked a lot and I wouldn't say either of them were 'the worst', they just weren't given their full time to develop their part of the character properly. (Even Tom's era had pretty dire stories).
Then along came the TV movie with Paul McGann - brilliant, but a rubbish storyline, and again he never had the opportunity to develop beyond the movie on TV.
And then the show came back properly. Christopher Eccleston was excellent, but he was replaced by the brilliant David Tennant.
And I thought he couldn't be bettered - but then we got Matt Smith.
I find it hard to choose favourites, because each Doctor has stories that are superb and stories that are dire, and there are characteristics about each that I both love and hate. To say any Doctor is the worst is to miss the point of the show - it's always been about the same character, there are just many facets to the Doctor and we get to see different facets in different incarnations, not all of them nice. He's a complex character, and to have him always nice and predictable would be boring.
Going back to my original comment about Tom Baker always being MY Doctor, there has never been a point where I've watched a later Doctor and thought 'Please come back Tom' or 'come back whoever', and I've never wanted any of the actors to move on early, because even with stories and actors I've not been overly fond of to begin with, they all developed over time (or at least those who were given the chance to did), and there were always better stories around the corner.
The show moves on, and I'm now over my sadness that Matt has left and I'm eagerly awaiting Peter Capaldi to make his full entrance in the autumn.
I might say 'he's my least favourite Doctor', but I never say 'he's he worst Doctor'.
If someone asked me who was my favourite, I would always say it's whoever is currently playing the role, because that's the one where we're getting new TV adventures - good or bad - and the one who's now continuing to develop the character of the Doctor.
Ask me who the worst is, and there is no answer.