I don't think getting rid of them is fair, but I would certainly put less of them through to the semi-finals and perhaps limit the type of singer, for example, Luminites, Jordan O'Keefe, Alice Fredenham, Robbie Kennedy, Rosie O'Sullivan, Arisxandra and Asanda (when they're older) are much more suited to X Factor or The Voice, in my opinion.
Other types of singer such as opera singers, musical theatre acts, maybe even gospel are fine on BGT for me, because X Factor and The Voice typically aren't suited for those genres.