I struggled with this choice myself. In the end I chose the Wildfire for 3 reasons:
1) Smaller - I don't want a big phone
2) Slightly prefer the looks - I'm not so keen on the buttons of the Desire
3) Price
However, the Desire is superior in pretty much all other measures. There's no denying that it's the better phone. The Wildfire is basically a mid-budget version.
Most people notice the better screen of the Desire, so if a clear picture is a big factor for you then I'd recommend the Desire.
It also runs android 2.2, which is a more recent system which is supposed to be faster. Because of the higher res screen and newer version of Android, this also opens up extra apps to you. Another small benefit of the Desire is that you can use 'live wallpapers', which are moving desktop screens for the phone.
Either way, I think HTC phones look great and are excellent quality, much better than the competiton. You've got good taste