After experiencing that abuse, your son shows resilience in wanting to continue his interest in refereeing. I'm sure he would become a great referee.
As you have said, prioritise your son's happiness and well-being. If he doesn't enjoy it, stop.
Thanks to you and your son for talking about this.