First and foremost, did they ask you to leave before they manhandled you out of the house? This is imperative to know, because if there was a pre-emptive warning, then he has the right to remove you from the house. If he hasn't, unless he admits to no warning. It's his word against yours.