If there is a rivalry between two clubs, then there's a rivalry. and it doesn't matter why. I don't see what difference it makes how far it is between grounds. None of the reasons really matter, it's either there or it's not.
Personally I enjoy my loathing of Palace more than I can be bothered...