This is an area greyer than John Major's wardrobe, and one there isn't really a rational answer for.
If you are 'that person' (black, Jewish etc) then it's perfectly okay to say the word N*gger or Yid. However, utter outrage when anyone else says it.
If it's an outrageous word which nobody...