When you compare it to how those who have been seriously alienated in earlier parts of the series have reacted, then yes, he does seem to be referencing his 'inability to integrate with the group' a great deal.
It's pretty much all we've ever seen him talk about, both in the diary room and with a variety of other housemates. He appears to talk about it every day and to anyone who will hear. Is he distrusted, yes. Is it unfair, almost certainly. But I also feel he's embraced it as his story and he is using it consciously. Or at least he's certainly feeding and nourishing his 'underdog' narrative. Not that I would criticise him for doing so. He's turning it to his advantage, which is what any smart player would do.
People accused Jale of playing the 'victim card' on here, because she sat in the garden once and talked to herself. But her experience of being ostracised was more considerable than Pav's; people were actively hostile to her, and she was being alienated to such a degree BB had to step in. And yet we heard very little about it by comparison.