Homophobia isn't about being afraid of gays. It's about seeing them as somehow "inferior". Any time you refer to something as being Gay and you mean it as an insult, it is homophobic.
I'm not gay, so I don't care personally, but our society is better when we're more tolerant, don't ya think?