Depends on what company and context
Usually, I use “we” or “us” if the person listening knows I’m a Brighton fan.
If not, "The Albion" to a bloke. "Brighton" to a girl ( I suppose on some level I think they’re more likely to understand if I explain it that way)