That is actually wrong to an extent. Tolkien was absolutely anal about things such as geneaology and dates, and he would always sit down and calculate things while writing stories - that is why he said that worldbuilding always began with a map. You are however correct that he did not limit himself to just that: most of his worldbuilding was indeed in terms of stories, and fact that he had stories of the First Age set down (to an extent) was important in writing Lord of the Rings as it allowed him to give his world a sense of history.An interesting thing about worldbuilding is that people often point to Tolkien as being the gold standard. However, what most people forget is that he did most of his worldbuilding not in the form of encyclopedia like entries. Rather, he just wrote stories where he explored his world. The reason he knew who Hurin was, was not that he'd sat down to write a timeline and figure out the important export of Beleriand. He just wrote the story of Hurin.
That doesn't mean that you shouldn't do any note taking or making lists or drawing up genealogies or whatever. You can and in some cases you should (and Tolkien also did). However, it's just as important I think to put story first.