Writing is communication, and the point you stop communicating to the reader, you are failing to do your job as a writer.
Good way to put it. Having an expansive vocabulary is great, but you have to know how to speak in any given situation. That means you have to know your intended audience. Even adult level fantasy books shouldn't be using crazily complicated words too often, because you want to stick with the vocab your audience will likely know. It's fun to throw in a favorite larger word here and there, but only when it really fits, and you think the pleasure of learning a new word will outweigh the distraction of looking it up for those readers who needed to do so.