They're not top-tier enough for him. Potter operates on a different plane these days.
Plus I'm not sure he could handle the weekly status report from Lineker on MOTD.
I have no idea whether Potter would consider going to Palace, or whether Palace would consider Potter.
What I do suspect, is that if it did happen, it would add a whole new dimension to the Brighton-Palace rivalry.