Ok, I guess I should say a few words here since I have been in that situation.
First, there are enough differences between M/M and F/M sex that comparisons don't constantly jump to your mind. And if it is a raw sexual scene, desire will overwhelm your character and the memory of his wife will be but a mental connection at the back of his mind.
Second, It all depends on how you present things. If your character thinks about feelings, emotions and love, it will be deflating, if it is just about specific sexual acts I doubt it would.
If you haven't already, read St-st-stuffed. If I remember correctly, Paul burst in tears while holding the watch his wife gave him because it represents her love for him. No such drama during sex : she is wholly absent from that picture (as she should be). Anyta was spot-on with Paul's feelings in that story.