When boi and I are out and about, I go with. I can't say I notice anyone giving hym looks but then again I walk ahead of hym and engage hym in conversation (hy may look masculine but hy doesn't sound it) while we walk in. When we're coming home from Jersey hy's often still in uniform so no one really takes a second look.
I guess the way I see it is there is nothing keeping a man out of the women's restroom if he was determined to come in, not really. I assume when I see someone resembling a female appearance that they have every right to be in the restroom, and frankly, I don't care. As long as you're not trying to share my stall with me, go for it!