I seem to remember that, in Noddy books, the bit of the town or whatever where the golliwogs lived was a nasty evil place and not somewhere you would dare to venture by yourself. So I think there is some understandable sensitivity through association with that.
I would avoid the word...
That is absolute prejudiced rubbish. Anybody who has any knowledge of local government now and who has any sense will know the enormous pressure that everyone is under and how much good and hard work is done by those people.