Many of that 20k were getting their fix at Withdean, just not 23 times a season. I used to do more aways than home games 2003-07 for example.
No doubt what you say is true to an extent, but in my experience (and based on your user name we share a stand) it is far less common than you are making...