I would agree with you had the defender not also added a healthy push as well. Had he gone down like that shoulder to shoulder then I agree but he didnt.
He was pushed off the ball had he not gone down it probably wouldnt have been given but then you have to question our referees for that. It was a foul stopping him playing the ball I dont know why he didnt just go shoulder to shoulder with him instead of the push.