Sorry but if you're that easily put off to the point where you won't sing because of the type of football, or because some "mums with kids are looking at you funny", then you're not trying very hard
The atmosphere, or lack of, is down to the people involved. No-one else. Stop moaning and just...