Some homophobes are in the closet but most male homophobes have adopted a "genital mutilation" phobia from their culture, a subconscious fear that if they were involved in a homosexual encounter, they could likely be in a submissive role, which would result in their male genitals, and their sexual drive and identity, being ineffective or just cut off.
Female homophobes are more complicated. They usually are afraid the subject of their sexual interest may be disoriented or distracted by other males, but female sexuality often has other factors to consider, like potential offspring, etc., so it's generally more difficult to describe.
Regarding the OP's post, the phrase "... I'm tired of [....] people *who accuse* everyone of being gay..." has its own problems and insinuations based on the context they are spoken.
A legit baseline for this issue is just to consider that there were no homophobes in pre Christian societies. Any further commentary on the issue is far beyond the scope of a TV show chat forum.