Here's another perspective based on the following exchange:
“Ever since junior year started,” Maddox continued, deaf to Kane’s reasoning, “Rusty’s just had it bad for me, always rubbing it in my face how much he’s hooking up and shit like that. Like I give a damn. He can fuck the entire campus three-way if he wants, but he doesn’t get to touch my Jonathan’s ass.”
Kane ruffled his hair and snorted. “Maybe he wanted a bit of attention. Thought of that?”
And there lies the crux of the matter. Rusty likely has feelings for Maddox and, not knowing how to deal with these feelings, competes with Maddox as a way to gain Maddox's attention. Kane points this out to Maddox, and is likely being subtle in trying to tell Maddox that "Rusty is attracted to you/has the hots for you". Rusty is jealous that Maddox is paying attention to Jonathan and not to him, given the feelings/attraction he's had for Maddox since junior year.
Maddox was totally oblivious to this attraction and any gay feelings at that age, whereas Rusty likely has been battling with those feelings /attraction since that time.
As @Laura S. Foxsaid, "Rusty just won't leave things alone. I'm not going to tell you why, and I'll let the guy speak for himself when the moment comes."