Maybe I’m seeing what I want to see, but in the several exchanges of words and looks with focus on that between Peter and Ned, before and after Graves confirmed their family relationship, I get the impression that Ned is sincerely surprised that Peter would respond as he did but also, that he’s doing it to defend and protect Ned. Maybe the family pledge extends beyond that of his younger brothers, but to the clan he hardly knows.
Needless to say, Ned’s unexpected sacrificial offer to protect Peter’s relationship and marriage probably bought extra mileage with Peter, West, Will, and the rest of the extended Carter friends and family Clan; he’s not nearly as shallow of previously thought to be…maybe Peter has competition for his acting position.