Hi Grace
When you say he attacks your visitors, how bad is it? A few years ago one or our neighbours had a cat and he would grab at people's ankles as they walked and nip at you but it wasn't a bad attack, just him trying to play.
Does your cat not like to be stroked at all, or is it just on his terms? Most cats are like that, though some a bit less patient than others!
My cat, Minx, is a ginger and white and quite the most mental cat I've ever had.

She doesn't hesitate to give us a slap if we do something she considers out of order, or don't do something she wants us to. A few examples:
She sits by her food cupboard and if I tell her she's had enough she'll bite me.
Tonight when I got home she jumped up on the bed for a fuss and I accidentally caught her nose with my nail, so I got two bites on the arm for that.
She also bites out of affection though, sometimes they're just soft little nips, but the other day she got a bit carried away while she was sitting on my back and bit my shoulder quite hard.
She's really quite bossy, but absolutely hates visitors and hides under the bed or in the garden when they come.
I assume you've had him neutered, tom cats will be more aggressive if they've not been 'done'.
One other thing, does he get to go outside? Sometimes I think aggression can be because they'e bored.
Chi