You might want to look up what they've done with cricket umpiring and the Elite Panel. They've managed to build a system where they manage to relatively successfully combine backing the umpires alongside holding them to account on both accuracy and consistency. Last time I saw stats on the elite...
Maybe Football needs to bring in some sort of (fully independent) referee review/promotion/demotion system along the lines of what cricket has done with the Elite Panel for international umpires.
Would obviously need to be scaled to accommodate making sure the Prem gets the best of the best...