If he took offence, he was in the wrong it wasn't what we meant.
We're not racist, we can prove it as we didn't get enough money for him in the summer. (On a par with "I have plenty of black friends.")
Just my opinion here...
Racism is complex. People who've experienced it all their lives may have been deeply affected and recognise it where 'onlookers' might not. It might seem something we (I'm white) could laugh off or explain away, having never been on the end of it, but still might be...