I agree. I felt the same way even when the Albion were 4th division but being in the Premier League does sharpen the dislike for opposition players.
I don't feel any embarrassment for loutish behaviour by people I don't know so this reason is irrelevant.