I guess I can join the ranks of the wierd ones. When I hear saffron, I assume you mean the spice, and when I think of saffron being used as a spice, I an image of a big bowl of yellow rice comes to mind. Actually, saffron used just about anyway makes me think of yellow.
Except for fabric dye. Fabric dyed with saffron is a shade of orange. I don't know why I know that (okay so maybe I know a little bit more about fashion and textiles than I willingly admit) but I do.
I did however vote yellow, considering, like I said, I just assume people mean the spice.