I realise there are not many Torontonians around these parts, but thought I'd share the link in case a few happen to trickle in. For those it might be useful for who weren't aware before (just found this myself as a new student there), the University of Toronto has gender neutral washrooms spread out around its various campuses, particularly St. George campus in the downtown core. Here is a compiled list of all the gender neutral washrooms on campus:
http://www.sgdo.utoronto.ca/washrooms/Locations.htm
I've noticed a few restaurants/coffee shops in the area like Tim Horton's across from the varsity field has single-user washrooms as well that don't require you get a key from the counter to use.