I'm not transgender either, but I wear girls clothes sometimes, and plenty of girly accessories. Been mistaken for a girl plenty of times as well. When I go to a public toilet (they're kinda disgusting, so I don't make a habit of it), I go to the boys. If I identified as a woman, in my head, then I would still go to the men's because I'm a boy and I can use the urinals.
You think practically, not emotionally. I mean, that could just be me. I don't know. But a pervert could drop into any bathroom, for males or females. Just recently I read about it happening rather locally. That's why it's a good idea to avoid public restrooms like the plaque regardless. Not a good place for doing anything other than your business.
Like I said, they should just be unisex. That would save so much trouble. Give everyone their own stalls, a lock perhaps for each one, and completely get rid of urinals. Then everyone has their privacy to do what they've gotta do, and they can be whoever they wanna be.
EDIT: Of course, if you're transsexual, that's completely different. Not sure how anyone could be confused or offended by a physical woman heading into a woman's toilet.