I'm guessing some of them have auditioned and don't get through. The first round of auditions for X Factor involves singing to one of the show's producers in a booth, with a huge queue waiting behind you, and where they often stop you after about 10 seconds. Not many (in comparison to the number who audition) get through, or get to show off what they can do. In comparison,everyone at the first round of BGT auditions (assuming they still do it the same way they did for series 2) gets to perform their act, and gets videoed so that the producers can then watch them all back and choose from that who to put through to the judges rounds. Much better chance of them seeing what you can do.
Also, X Factor has the theme weeks where most of the acts are pretty much forced to sing songs that aren't their style, week after week. Some people don't want to have to do that. They'd rather have the one semi-final and maybe the final, selecting their own song (if the producers let them...) and having more opportunity to be themselves.