I had a pretty horrendous experience of a couple of gobby Toon fans sat next to me in the WSL for that 2-1 loss a couple of seasons back, when Diame scored the flukiest goal of all time. Still kinda wish to this day I had screamed for a policeman/pointed them out to a steward and gotten them...
From what you've described I don't condone the BHA fans' behaviour - however - openly chatting and making clear reference/inference that you are an away fan in the wrong end for a massively stressful and tense game is always going to be a huge risk, and a slightly daft thing to do.