It's always easy to comment in hindsight, but, I have always told myself in these kinds of situations that I would just agree with them, say, yes, we have no right to be here us whites.
Then I'd be tempted to say "I'm only here cause you locals cannot satisfy your women"