It's not though, some of it is, I agree, but then so are parts of Brighton, Hastings, Worthing, Eastbourne. In fact I'd go as far as to say most towns have their shitty parts.
Pretty much my point of view as well, just don't see why people resent the town so much, it's far from perfect but I live in a beautiful part of town with Tilgate forest literally outside my door, I'd much rather live here than a lot of other places.