I know what you mean.
The best time I had as a Brighton fan was on the bus to and from Gillingham. The ground was crap, the town was crap and the football was crap. In fact it was about the worst time to be an Albion fan, but it was still so much fun doing it every week.