I've been having the same problem, I don't have a cat flap, instead I leave a small window open. Went downstairs this morning to find a neighbours cat struggling to get back out again.
The smell seems to come from outside tho'. Under the window there's a small flower bed, maybe this is being used as the local cat toilet?
It's never happened before, only since this other cat has been hanging around, he's a huuge cat too, I know the owner, she'll be mortified if I tell her,lol.