If the drummer doesn't try to be that loud, and just beats out a decent rhythm, then I'd be for it with restraint. As an example the drummer at Celtic Park is decent, but very low key, never louder than the chanting or clapping, just helping things along.