So, I'm rather a fan of human only or human dominated fantasy worlds but fantasy races have their role. What I really, really loathe is "race as culture". Real life humans aren't homogeneous, so why should dwarfs, elves, or orcs be? (they can be, but that should be a major point)
The giantism thing is a good example though. You could have some explanation about a human subspecies that becomes a race of giants... or just call them giants and you've accomplished the same thing much more neatly.
I had something like this in my 'other world' - a city-state of warriors, 'blessed' with immense size (over a few generations) by their deities (whose priests imposed a 'special diet.') These giants, numbering into the thousands, embarked on a campaign of conquest. But leadership squabbles, lack of access to the 'special diet,' and genetic issues pretty much ended that. Now, they exist as isolated villages and enclaves scattered over a wide area.