I can really only tell how I began building my setting, maybe it gives you something to work with. Then again, I'm not sure how well I've managed at that, but I went the long way because I think if you're going to use that same setting over and over again there should be a fair amount of details and information available to flesh out that world.
First I drew a map. Well, in fact I drew a whole bunch of maps, adding more details and points of interest with every new version. While I was doing this I pictured locations in my mind, and at the same I was taking some notes. As time went by, the main continent where most things take place got actually quite messy, and I'm planning to remake it at some point. It works for me though because I felt like I really needed all that nitpicking to work. (link to the mentioned map to show what I'm talking about).
As the notes piled up I began to look at the setting more closely. I wrote a brief description of each continent, each part of the continents, and eventually the world begun to take shape. It was slow, but with each new added detail it was coming to life, gaining the proper atmosphere. I wrote a short history from the moment of creation to that world's present day, detailed the magic, and as I'm not a big fan of flashy spells the mages pull out of thin air, I made it difficult for them by bounding it entirely to writing.
To sum it up: I forced myself to sit down and write a chapter about every new idea and a concept I came up with, and piece by piece the setting gained flavor and, at least I like to think so, some unique and interesting elements that give depth and credibility for the stories that happen there. What I also got was a lengthy book of its own that I can use as a reference whenever I'm writing a new story. It helps me to remember everything and keep the story in line with the existing environments. This process may seem a little laborious, but it worked for me. I'm sure there are shorter versions to basically reach the same goal, so I suggest just experiment with things. No matter how crazy idea, it may work incredibly well once you polish it and implement into the general setting.
I'm not saying this is the right way to do it, but it's one way, albeit painstakingly slow process.
I did something a lot like this as well. It depends on the scope of your story, however. Do you need a whole world? Or do you just need a city and a couple of towns?
Another thing I tend to do is to begin with a place I like in my mind. That could be from the real world, a book, a movie or a game. Then, twist it to your needs, play around with it in your mind until it becomes something that works in your story. Give it some defining characteristics if it needs them.
Now for the really important part in my opinion. History. A thing is just that. It's inanimate (for the most part). It is a thing's history that gives it significance. The Berlin Wall was just a load of bricks and mortar, but of course we know it was so much more than that. So I ask myself a bunch of questions. Why is it there? How did it come to exist? Do I plan on it having any special significance, or is it just a really cool backdrop?
Then we have the inane. Most of any world is, in fact, inane. We all know what the inside of an inn/castle/house looks like. I've read plenty of fantasy books that love to describe the inside of a room in all its gory detail. The design of a four-poster bed isn't too important to me, to be quite frank. I like the things that make it different to usual. It could be the people within it, it could be the scorch marks on one side of the keep from such and such a battle, it could be the round windows in the inn, and the story behind that, when square windows are the norm.
Flora and fauna is another area where you can really go to town, without overdoing it. You have a chance to make things unique. I don't want to read about horses, per se, but if a special horse exists (think Shadowfax in LOTR), then horses just got interesting.
So, to summarise... Ignore the inane, or at least have it there only as backdrop, but go to town on the unique. Make it different, attach a story to it, and use that story to make your reader feel something, whether it's awe, fear, a sense of homely comfort, suspense. It's all at your fingertips.