I totally do not think that we should label people and villify them for the labels we give. However if people choose to label themselves and to act on the basis of their label then they have to expect to be judged by it. If they stand behind a flag they are judged to be carrying it.
If you are a member of a group that has been labelled and persecuted by others then you have every right to demand that you should not need to justify yourself but I don't beleive that is the case with those who choose a label. If the group you choose to join also contains evil, vicious, ignorant, preudiced bastards then you do one of three things... you push the bastards out of your box (not just speak pretty words that you condemn their behaviour but do everything in your power to root them out and throw them out... take away their power, ostracise them and anyone who supports them) You redifine your box (distance yourself from the bastards... I am not the same as him I don't beleive what he beleives, I don't want to beleive what he beleives and I am going to fit in with you be part of your community and not be like him) or you get the hell out of the box.
If you sit in your box next to the bastards in the box then you expect to go up with the box and are fair game for the dynamite.