Actually, I'm listening to the podcast again and it covers a ton of what has been brought up in this thread. I highly suggest it.
I think we may have a few different ideas of what "foreshadowing" is specifically referring to.
I will take a little different twist than Helio... and assume the foreshadowing is not about tension, which is absolutely necessary.
Here's a little tidbit... A large portion of your foreshadowing (if you do it well) is going to go unnoticed by a large percentage of readers anyhow. Doing none is pretty much a mistake, flat out. It's like any detail, a reader can miss them. I will pick on my editor and Helio here quick like... My editor the other day, after finishing the book, was confused about something and I was like huh? So I sent them a direct quote that explained the entire thing and got the head slap reaction, paraphrased was "holy crap! That one sentence cleared it all up." Ayup. Helio mentioned the other day not knowing the age of a character... I went back and found the age mentioned a few pages previously... we all do this as readers, even the pros. As a writer, we must do some head knocking and "reminding" to the reader in order to keep them in touch. Foreshadowing when done subtly is even easier to miss and forget, so by the time a writer thinks they have too much, it's quite possible they have it just right, which is still too little for some folks.
If a "shock" is obvious in hindsight, it was done well. If a "shock" truly feels out of nowhere... it can lead to issues.
What made Sixth Sense so effective is that the majority of viewers slapped their foreheads on the reveal. "Duh!" and when they watched it a second time and saw all the clues... yeah, that's good stuff.
Another personal example is a short I wrote which was essentially a joke/play on words for the theme of the flash fiction. I thought I went heavy handed with the overall joke, without actually saying it, and far as I know not a soul got it until I told them. I wanted people to "earn" the joke because it wouldn't be as funny if it was in your face... turns out, it needed to be in your face. Although it was fun to see peoples' reactions afterward.
I can't agree more with Helio and Demesnedenoir.
I'd like to share with you one of my first experiences with writing "surprise". I'm not sure if it applies to your situation, but maybe there's something to be gleaned from it.
It was my first writing class in college. I wrote this story that was supposed to be about origins of urban myth. I basically set up this scenario where a a man and woman meet in a supermarket. There's flirtation, and everything seems cute and cuddly. Skip forward to the parking lot. The woman has a flat and the man shows up to help her fix the flat. More flirtation, then the man proceeds to bash her in the face with a tire iron and toss her in the trunk of his car before driving off. The End.
I'd even foreshadowed a few things, but I disguised them as part of the flirtation.
This was the most despicable and dishonest story I've ever written. The story was devoid of any worth. Why? Because there was no purpose to it other than the trick, the "surprise". It was me, the author, showing the reader how 'clever' I was because I could manipulate their expectations and emotions. It was "Ha-Ha, I tricked you. I tricked you. SURPRISE!"
I got a talking-to by my instructor. But me being me, I didn't comprehend things until much later when I saw the same thing done in a story I was critiquing.
Surprise for the sake of surprise is worthless. And writing a story that serves only to surprise is worth even less than that.
I agree with everyone so far, but I don't want to dismiss the thrilling, useful bolts from the blue that can be used in moderation at key points in a story. A lot of my favorite novels will do this.
Often it takes the form of an unexpected, non-foreshadowed encounter between two individuals who have hitherto been on different tracks/arcs—especially, when one is good and one is sinister/evil/vicious.
Sometimes, it's an unexpected magical event or surge of unexpected magical ability.
In these cases, the tension comes from the sudden introduction of the question, "What happens next?" or "What does this mean?"
It's okay to shake things up once in awhile in this way. But it only works because most of the time readers need the foreshadowing and planting of seeds. Readers need to feel themselves to be a part of the discovery, the wondering, etc., that foreshadowing allows, and not like passive bystanders shut out of the process. These bolts-from-the-blue tap into that need by injecting the questions and impetus for discovery via an unexpected event. Tension is introduced by the event rather than used prior to the event as w/ foreshadowing.
Dragon, please listen to that podcast that I posted about irony.
There is NOTHING wrong with surprising or shocking the reader. In fact, things SHOULD be happening on every page that makes the reader sit up and pay attention. Something to make the reader go "Oh! I never saw that coming!"
What we are talking about though, is the concept of dramatic irony. How there should be "meaningful gap between expectations and outcome."
Setting up reader expectations, and then subverting those expectations with something shocking is good, if it is meaningful. But setting it up and giving a shock that the reader has no investment in can be a bit of a letdown.
So in the case of yours, it sounds great. It is meaningful in that it relates to what the reader has already seen. They know the headmistress is hiding something... but what? You have already set up the question in the reader's heads so that when your MC stumbles upon the boy the reader has that "Aha!" moment.
However, if you didn't set it up in that way it might be problematic. If you set didn't set up the expectation in the reader's mind first the girl might stumble upon the boy and the reader would be confused. "Wait? Who is the boy?"
Dramatic voice: "It's the Headmistresses son!"
Reader: The headmistress has a son? Where did he come from?
Dramatic voice: She's been hiding him all along!
Reader: I don't get it, why does she have a secret son?
You get the idea. When you foreshadow first then it builds to the dramatic moment of the reveal, adding suspense for the reader. If you don't then it can come off really anticlimactic, leaving the reader wondering WTF just happened and why they should care.
I'm not interested in surprise for the sake of surprise, If I chose to have a genuine surprise or complete shift in direction
how would I or any writer get people to just go along with it without "foreshadowing" that they were going to happen;because
if people are told that they are coming there is no point in having them.
Secrets are awesome, but you have to think about how to get the impact and response you want with them.
Second: What if a bolt from the blue acts as a inciting incident (to the story or just to a new stage in plot) and is explained later, but is used as a catalyst to the story?
I don't think about foreshadowing in my stories as much as creating the rules for the story world. I try to demonstrate the rules with scenes and dialogue, limiting exposition and info dumps. This allows readers to create their own expectations, which might be right or wrong. The ending of the story must adhere to the rules, but it need not be what the reader expects, creating what may feel like a bolt from the blue for readers, but also leave them satisfied with the read. If the ending of the story is basically a new rule, it may leave the reader feeling cheated. If the ending is an unexpected application of an existing rule, that's bound to be a much more satisfying read.