Now that you mention, I've noticed this as well. With my straight friends, physical contact is limited to hand shakes. Some have very physical hand shakes, involving a sort of "half-hug". Not once have I been able to "connect" with them emotional. My gay friends are different. Most of them encourage some kind of deep emotional connection, and insist on holding the other when he's down, including stroking his shoulder or back. It's not sexual at all, yet straight men seem to fear it.
Bingo. Much of our society still views being gay as a bad thing, so most straight guys avoid anything "touchy-feely" because it isn't "manly". I spent so many years not knowing any gay men that when one consoled me in the not-so-distant past, it was strange when he put his arm around me and let the ends of his fingers stroke me softly... then he tickled me, but that's beside the point. The point is it felt strange and it shouldn't have. Again, it wasn't sexual at all.