That’s a reason I have often heard. I personally don’t have an issue with it these days; in some respects it allows me to concentrate/focus on the restaurant and food more.
I’ve had a few snowboarding trips on my own as well as all my US marathon running trips. These were really enjoyable partly due, as you say, to having no one to answer to. Ski trips were always bad for people fannying around getting ready.