Hello,
Well Homophobia, is taught really. No one is really scared or shows a certain hatred of another person, place, or thing unless someone around them or culture has told them to stay away, hate, or accept the thing in question. Not trying to justify Homophobia, just realize that some people just see and comprehend what is taught to them differently. Also, lacking the manners to keep their emotions under control doesn't really help. Personally though, I don't know why they call it a Phobia. Homophobia doesn't really work for me. Most people don't show a hatred towards the thing that triggers their phobia. They just have anxiety and fear, it triggers their fight or flight mechanisms.
Anyway, don't lower yourself to his level. You don't have to tolerate your boyfriend's brother. Just try and show him the common descency that he neglected to show to you and your boyfriend.
Good luck,
Krista