Yes, you can just write the story and build the world details as needed by the story. I tend to work that way. But here's the deal. Every decision you make on the world building side, you're going to have to live with. Elves have gray eyes? Then they have gray eyes in the next story too. Dwarves are monotheistic? Orcs and trolls are mortal enemies?
What I'm saying is, while a world-building decision may be right for the current story, you may find yourself regretting that decision in the next story. So, in that regard, it does actually make sense to step back from time to time and make sure the background is both consistent and is something you feel is robust enough to serve multiple stories.
Happens with me. I'll pick out a place or situation or event on my world, and use it for the core of a story - or rather part of the core. The rest I have to come up with on the spot...which fixes the improvised bit into the worlds overall lore. The stories are part of the world building.