I'm glad you're writing about this, and getting it into the open, where you can deal with it. What we hold inside festers.
When someone passes, it's natural for people to relate what they knew of him or her, good or bad, whatever the context. It's how they deal with the grieving they're going through. Some believe that expressing condolences make those closest feel better, but it may only be themselves who feel better. They're not aware of your feelings, but of their own.
Grief is like depression. You can't get over it, you can't get around it, you can't do anything but go through it to the other side.
At some point, I'd encourage you to write about him, to collect the memories, the stories. All of them. Print them out, and bind them together in a special book. Put it with your photos. One day, you or your kids may want to read it, and it may make sense then.
I'll be thinking of you and your family, wishing you peace, and clear memories. As long as we remember somebody, they're not really gone.