I do agree - but I think in general that the issue is far more about the fact that anyone is in a position to know so much about their behaviour, rather than the behaviour itself.
It's interesting that you say that many of his fans don't actually want the children to continue in the show - from previous discussions about the kids, I've very much got the impression that some people are actually tuning in specifically to see them.
What would his fans want him to replace the segments with the children with?
I don't know about his fans, but I would like to see them replaced with more Gloria.
The children should be removed from the show entirely IMO. ........... I can not see it happening though (more's the pity)