I don't think it's insulting to suggest someone has Asperger's - not in the least - but it's not as straight forward as poor social skills/socially awkward = spectrum.
My daughter is not on the spectrum but in certain social situations - namely school - she does a damn good impression - she ticks all the boxes - but its down to her trying to mask her shyness and lack of confidence amongst other things.
My son on the other hand is on the spectrum but you'd never know - he's very sociable, loves company of friends and is very popular.
There are so many things involved when indicating autism that to go to Asperger's when looking at Andy's awkwardness on occasion is a hell of a leap.
My husband will tell you, with the kids and my work, I can be a little preoccupied with traits. (Which is why i'm waffling on here even though it's not really relevant to Andy.

)
He says I see them everywhere - I've even got him on the spectrum

- but I'm honestly not seeing anything with Andy.
If anyone's 'not getting' people it's them not getting Andy. Andy just misjudged how reasonable they would be.