Well, some people are just plain stupid.
I smoke alot of weed. That would make me a stoner. If somebody is talking about me, I'm sure the term "stoner" comes up. It signifies that I smoke weed more than the regular person. I don't think it is derogatory actually. I don't think tweaker is either, people that bomb meth, its just a term. Coke-Head, etc. I think alcoholic is different to some people. But I guess to each their own.
However, you are kind of putting an image of your friend out for others if they have not met him. Smoking bud isn't as cool to some as it is to others and they might judge him, so you could be doing a diservice, unless you know you hang out with people chill with smoking mother earth.
I'd say just refer to him as Steve, the good guy you know. The weed thing is just a characteristic of his personality and if anybody asks, yeah, he is a stoner.
But I really relate to what you're talking about. I have friends I'd want to do certain drugs with for instance, as opposed to others. There are some people I'd go to lunch with, but never to dinner or anything past the hour of 6 o'clock. The list goes on. So you're not alone in that.