I'm sure we get these comments every year on BGT and even X factor, when Simon was on it. Simon does sulk on a regular basis, but once the show is over he forgets about it, and knows the whole thing is a pantomime which viewers enjoy. He's a good sport really and knows that taking the mick out of him entertains people.
David is popular with viewers, therefore I think David would choose when to leave.