For some sadly, the answer IS 'culture' (broadly) - they look different, dress different, act different -and it threatens people.
For others you need to expand 'they took our jobs' to a broader 'they are an additional stress on public resources / job market / housing stock / school places / nhs...