I actually like worldbuilding. A world is useful for so much more than one single story. (For example, two stories.)
What I like most is the diversity and versatility that you seem to fear. One method to put it to an advantage: Ever sat through a class, meeting, lecture, speech or similar that threatened to bore you to death? Ask one simple question: How can I adapt this to my world and suddenly it might become interesting.
This probably does not work for everyone, for me it has worked often enough.
Of course that does not mean you have to include every world detail in your stories or show of with knowledge on topics no one else is interested in. But I know from experience that on strange ond obscure topics worldbuilders tend to be the persons who still know something about that.