I'm wearing a man's jumper as I type and don't think it matters as most clothes are unisex. I've yet to see a man who suits women's underwear or a dress though..
I'm ambiguous really about the term Crossdresser, people wear what clothes they want, does it really matter what gender is on the tag? I'm transgender but my views on gender are very fluid.....there's too much regulation on grounds of gender
I don't think men's clothes are suited to women's curves, so no. Though I had to buy a large boy's Birmingham FC top from the club shop just before going to watch the game at the ground, as they had run out of women's. It was end of season and it was only a fiver - bargain