I've got caller display on my phone. If I don't recognise the number, I don't answer. If somebody really wants to speak to me, they'll leave an answerphone message.
I'm also with the telephone preferencing service, so consequently get very few calls now.