People go to the football for different reasons, some to irritate those surrounding them as much as they possibly can. It wouldn't be so bad if it's just football talk, even if disagreeable dross it's still tolerable. Already moved once because of it, so no you're not being miserable.