In all honesty, I can't think of any reasonable reason it would detract from the story.
I still feel you ought to have a reason for him being black, even if just for yourself - especially if he's showing up in a location where black people are a minority/rarity. I'm assuming you wouldn't put a black guy where it wouldn't make intuitive sense for him to be without providing an explanation?
Would I need a reason to make him white?