I need advice. Would you be offended if....?

Personally I would have commented at the comments. Like "Do you think he's going to be a whitey" would have had me say "We don't use that word in this house, it's racist. Especially don't use it around my son. I don't want him growing up thinking there's something wrong with someone's skin tone." And I'd do that in front of my son, since they're saying these things in front of him. He has to see you will not tolerate it, and that references to someone's skin tone is inappropriate.