I think retrospectively it's easy to remember what you enjoyed about it, and nostalgically say 'I wish I'd stay longer'. In reality though, they all had reasons to leave at the time.
Eccleston, obviously had various issues with some of the people working on the show at the time.
Tennant never particularly wanted to leave, but said that if he didn't do it at some point he'd end up never being able to, and they'd be 'wheeling him out in his bathchair' and hence decided to go out on a high, with his huge popularity still fully intact.
Smith seemed to want to move on to bigger and better things, and had even done a film whilst before his time as the Doctor officially was over.
With Matt Smith, it does make me think that he's saying this at the moment, because he probably thought being in hollywood films would make him a big star in his own right, known for more than just playing the doctor, but so far he hasn't really had any huge starring roles, and I don't know of any films he's been in that have been huge hits, so thats probably why, for the moment at least, he's remembering how much fame he had as the current doctor, and thinking 'maybe i should have stayed longer'