It shouldn't be annoying unless it's a case of loads of people getting up and blocking your view of the game as they squeeze their way past your seats towards the aisles. That's why whenever I go to games I try and get an end seat in case I want to pop off early.