I've taken the opposite approach, and I don't have the same problems, but at the same time I kinda do. For me it becomes how would a planet of people who evolved with this renewable, non-polluting source of magic energy progress technologically? Look at the ways that humans in real life got passed an obstacle, and decide how these people would get it done instead. Like microscopy, I was thinking the other day about how my people might discover that there are organisms so small that they can't be seen with the naked eye with magic? My leading idea so far is someone was just playing around with a spell that zooms in their vision, a spell that could have all sorts of real life applications, and accidentally zoomed in so far that they could see tiny little things floating in the air or in the water around them.So, looked at in one way, I didn't have to do any world building. I just inherited a world. But I quickly found out that I had to do all kinds of inventing, or at least improvising from existing material. It's like Earth was a classic song and I was tasking myself with doing a jazz improv on it.
This is why I want to make sure all of my magic logic and my world logic all fits together before I start writing too much, and especially before I start to publish anything. I wanna make sure that I never break my own rules, in obvious or not so obvious ways.One is, when I invent something--for example, that dwarves don't have a formal religion but instead practice a kind of ancestor veneration--I do this in the context of a specific story. It gets invented on the spot. Even that topic of ancestor veneration is very big, so I invented only pieces of it for that story. Then, later, I chew on that and try to fill it out because, oops, I wrote it into a story so now it's part of the world.