Never. Because rugby is shithouse and if I did go I'd probably just stay in the bar and hope there was some decent sport on the tv, or a freshly painted wall I could stare at.
Watched loads of cricket matches though and it's never been an issue. Plus watched football abroad, likewise, no...
Works perfectly well with every other sport and with football in most other countries so I really can't see the problem. People are just conditioned to the nonsense situation we've had forced on us for decades.