It's a ridiculously complex issue.
From what I can see the immigrant population that most people appear to have a problem with (non-EU), have to come through via a points system. So I'm not too sure how leaving the EU would affect that. There is certainly an argument that it could be more...