While you are probably right that I spend too much time on perfecting my worldbuilding, I believe that I need to know at the very least the basics of my settings before I start writing. Setting does dictate the range of possibilities for stories or scenes; for instance, if I wanted to write about the Roman Empire, I couldn't exactly write about election fraud without taking a lot of creative liberties with Roman culture. Now to be sure, certain details can be researched on the fly during the writing process, but a little beforehand knowledge of the setting's general mechanics is necessary before plotting.
As a matter of fact, although drawing maps can be fun, the researching part of worldbuilding is not my favorite part of the process. I want to start the actual writing as soon as possible, but there is often so much I need to know about my subjects before putting pen to paper. It can overwhelm me.
Are you writing in the real world? Historical fantasy, as it were?
If not, if you're generating your own world, then I would suggest only worldbuilding enough to get you going, then worldbuild as you write. Figure out the basics (gender roles, class structure, clothing, basic area map), then let yourself just write and think up the rest as the need arises. You will be editing your novel anyways, so you can always make notes of what you worldbuilt as you wrote, then add it back in later.
I would suggest that you do that first. Writing historical fantasy in cultures significantly removed from your own has many pitfalls. Researching that takes longer.