I think most fans in modern stadiums do actually have a better view of the game than the ref. TV obviously has numerous views that the ref isn't able to see, all of which makes us assume that a ref is useless rather than doing the best that he can.
I realise that this isn't a popular view...