This is a bit of a strange argument to make... Slavery is one of the things that society, fortunately, came to see as wrong. Another thing that we have, in a lot of civilized countries, been able to see as wrong, despite so many societies seeing it as wrong for a llllllllllllllllllllllllllllllooooooooooooooooooooooooooonnnnnnnnnnnnnnnnnngggggggggg time are rigid gender roles - clothing being one of them.
That argument doesn't work so well for opinions stuck in the past, like if somebody were to state that slavery should come back, you wouldn't accept "just because something is acceptable to society, doesn't make it right" from them. Saying you can't crossdress is equally ridiculous with that argument, although obviously much less disgusting.
"Society" has been slowly moving, that is maturing, to the point of allowing men and women to dress the way they want (although it's mostly been women, because women have had a much longer road... You know, actually being able to show any of their skin to the public). To say "just because [it] is acceptable to society, doesn't make it right" is a rather ridiculous thing to say here, because this was never society's default position. It is one that society has had to come to from starting to mature and treat people equally.