Do black guys risk becoming ostracized by their white buddies by being overly-sensitive?

For example, say a bunch of my white mates and I are drinking a few and we have a custom of calling each other "boy," like "Boy, what are you talking about?" It's a term of affection, right? We're all acknowledging that we're being lads. But when the black mate enters the picture, the situation becomes weird and we don't know how to welcome him into the fold without dampening the collective fun. How should that be handled?