Stupid question

...but here goes anyway! Basically, I'm tall (about 6ft3/4) and quite broad (i.e. shoulders). I usually buy a men's size 32 waist in trousers/jeans and am generally a 38-40 Medium in tops/shirts etc. I am quite skinny but broad with basically no muscle and some surplus belly fat etc. Sometimes my clothes have seemed quite tight for a medium or 32 waist, I think due to my build and figure and the fact I am not particularly toned. My question is, if I am planning to get fitter and more toned, should I start buying bigger sizes of clothes, i.e. 34 waist, and 40-42 Large in tops, or if I tone up will my clothes be less tight? Men's sizes vary quite a lot obviously from shop to shop but lately I feel things in general seem quite tight, however, I have changed my taste in style so that could be it. I'm rambling, but I hope people get what I'm on about. Cheers.