Myth WeaverWay back when, this is kinda sorta what I did. BUT - I was thinking 'AD&D setting' as much as 'story writing'.
As I created the worlds (yes, there have been quite a few), I created key events and situations to go along with them: a major war here, the rise of an empire there, the development of this kind of magic and the rejection of that sort of magic. Each of these events and situations gave rise to story situations: 'what would it be like to be a person present when 'X' happened?'
Though I've long since moved past the 'game setting' bit, the ideas sparked still serve as inspiration for stories I write now.
That said...take it too far, and world building turns into a sort of dead end: you design A and B and C, but...set out to write a story, and A becomes largely irrelevant, B is revealed to have some serious flaws, plus you WILL discover there is a whole slew of things you missed out on including altogether.
Anymore, what I have is a semi-detailed set of notes for the main parts of my world, and much sketchier notes for other regions. There are entire nations where I could not tell you the name of the ruler or the names of the major cities. What I do have is just enough to provide a bit of a backdrop for characters hailing from there, or making a quick visit. Even with the more detailed area's, I STILL end up creating a lot of stuff 'on the spot' while writing the story...though the notes I have are real useful.
Mostly, I'm writing shorter tales, where a major character in one story might get only a quick cameo or not be mentioned at all in another. The way I look at it is most epic tales seriously shortchange a lot of people in that world; its not so much the continuing adventures of a few, as it is a lot of tales by a lot of people.
