If English Romany gypsies can use the phrase to describe travelers from outside their group then I guess its one of those words like the N word that can only be used by the group it refers to.
Its about time calling someone a "cracker" is given the same status as the N word though. Definite...