Absolutely. I wasn't trying to antagonise before. I guess it depends on the definition of 'not all of them being thieves'.
Those who don't pay tax on their work are thieves. Plenty of the general population do that as well. Not all of them break in to your garage.
I'm sure every sane person hates a criminal, but the term is derogatory towards all travellers. Not all of them are thieves.
Having said that, I reckon very few pay any sort of tax.
It doesn't matter where the term derived from. Every word or term derives from somewhere and something. It is used as a derogatory term to cover the travelling community and has been deemed hateful.