Agreed, Stifler's Mom was an awful character and it is no wonder they wanted to avoid her, she had no self awareness and crossed social boundaries in my opinion.
Pot Guy was a complete douche, aside from the obvious of turning up high, his behaviour as a result of being high was nothing short of disgraceful, a complete clown.
I hated Doug's boss, in fairness he was probably OK but just really annoying. I always hated the comment he made when Chandler pretends he has split up with Monica, he says something stupid like how there was no love between M&C, but she looked at him with pure lust. Talk about deluded.
In fairness, Janice probably does get treated a bit unfairly. Granted she is annoying, with the OMG and the voice and laugh, but she is quite sweet at times. In the hospital, she is nice to Rachel, giving her advice about raising a child whose father is not in a relationship with the mother and after initial teasing, she is quite nice to Chandler with advice when they bump into her at the fertility clinic.