The VAR refs have to watch all the games as its their job to spot a possible mistake or something the on-field ref misses. For example the on field ref might not see a red card offence and so will have their attention drawn to it by the VAR ref.
A thought I had today. Once/if they get it right, how will it work on the last day of the season, for example. It wouldn't be fair if not all PL games had it but how will they do 10 matches at the same time. There are only so many refs available at one time to sit in front of the tv screens.