We have for a long time discriminated based on sex because we recognise that a minority of people with penises are a bit pervy and/or rapey, and that it'd be safer for people without penises to have spaces where all people with penises were banned, on the basis that you can't tell just by looking that the person with the penis is pervy/rapey. Clothing, makeup, cosmetic surgery and feelings are not a valid reason to change how we discriminate against the bearer of the penis.
So if someone is born with a penis, but has that penis removed as part of a gender reassignment, you still want them banned from spaces where penises are banned, even though they don't have one?