I think your right about the rivalry.
I would say Palace fans over 55ish would see Millwall as rivals. 40 >55 see Brighton under that age would see Charlton as rivals.
Me Personally, I dislike Chelsea and Leeds, as I dont really go for the pantomime hatred of any of the above. Cause like...