Not really. Not all Brexiters, for example, are gammon. There are many with perfectly valid issues with the EU who can express them in perfectly erudite and polite ways. Whereas "gammon" is usually reserved for the "bloody foreigners, why are they all coming here, stealing our jobs, I blame...
Someone in my friendship group is a bit of a gammon and he was apoplectic when Jeremy Clarkson (who he hero worships as you say) came out for Remain. I still like to bring it up every so often because it sets him off like a jack-in-the-box.