For brevity, and not to diminish any feelings you may have - I think you may have just answered your own question. For you, anyway.
My first boyfriend, that I lived with (five years), was done without him or me coming out to our families, and began when we were your age. Some never became aware and some made the leap to figuring out our relationship, but not a one said anything. Those that guessed waited for us to bring it up instead of drawing attention. We thrived, until we didn't, but it wasn't do to family interference that we broke up. Five years later when I finally came out to my family, at age thirty-five, because I was I was moving in with another man (who remains my partner) and it was the right time, I told my family and it was to no great fanfare or tears.
So yes, you can be with someone in the closet and still have a fine relationship. But unless your views and feelings have changed, I still think you may have answered your own question.
Edit to add: I don't know how I missed the fact of his not wanting to be seen in public with you. A person would be hard pressed to view this as anything less than irrational behavior even for someone that's not ready to be out. I know fear can make people do crazy things but unless you're wearing a sign announcing what sex you like to sleep with... You could definitely have the conversation, as mentioned, but this seems too far removed from the kind of relationship you're looking for.