A dump? Sure, the sugar factory isn't going to win awards and all towns have run-down parts, but the town centre is nice (and i'm not biased, I am one of the London over-spill you mention, that being some 10 yrs ago mind). In fact, the reason why I like bury is that it has both the pretty parts...
I live in Bury St Eds, having 2 small kids I can't advise on having a teeming social life but can give a run down of places to eat/drink and activities (some of which have already been mentioned):
Food and drink:
Pea porridge http://peaporridge.co.uk/ - lovely little relaxed restaurant which...