Well, the modified quote from Alfred Hitchcock is a good place to start, MORE or LESS. (Heh. Basically, he had movie instead of book and audience instead of reader).
MYSTERY is when the reader knows LESS than the characters in the book.
A mystery plays to the intellectual, to the “little gray cells”. In a mystery, facts are unknown and it’s a puzzle to be solved a little bit at a time. You could use the blossoming of a flower as a metaphor for clues revealed or peeling layers of an onion, but I prefer layers of a parfait, like Donkey. A mystery is a story to be SOLVED.
In a mystery, you have a sleuth, and—this is important—the reader discovers the clues as the sleuth discovers the clues. No fair for the sleuth (author) to withhold clues from the reader.
Choosing to write mystery can also have an effect on HOW an author writes the story. For example, creating a single POV (point of view) character to tell the story makes it easier for the reader to know exactly what the sleuth knows, since the reader is in his or her head.
Writing a single POV sleuth is also a great way to learn how to lay out a mystery, manage clue drops and evidence collection, conduct the investigation, interview the suspects and witnesses, and lead the reader to all of the pieces necessary to solve the puzzle. For the reader, seeing the mystery unveiled seems to be as enjoyable and important as knowing the outcome itself. How do I know this? The mystery genre is one of the most popular genres in entertainment.
And FOOLING the reader and sleuth with a FAST FAIL beat (see Demystifying the Beats, our newest book!) is a classic technique to subvert to the wrong conclusion before the true REVELATION beat. Plus, a well-executed FAST FAIL can lead to a very satisfying final twist, if the author has set up the story well.
SUSPENSE is when the reader knows MORE than the characters in the book
A suspense appeals to the emotional. The reader is willing to experience anxiety and stress and fear of imminent danger. It keeps them turning the page. The reader understands that, to reach a satisfying conclusion, the emotions and danger in the story must be faced and RESOLVED.
In the story of David and Goliath, Goliath is much larger, much stronger, much more experienced and dangerous and frightening than young David, and more is as stake than David’s life. Yet David must face this danger—and potential death—alone in the final confrontation and resolve it. But how will David do that? (He “throws” in a really good TWIST). Makes for a compelling story.
Suspense focuses on the fear of not knowing what could happen next. It is an emotional journey that the reader takes with the protagonist (which could also be a sleuth, BTW.)
And while I talk about the protagonist’s point of view, suspense is trickier than mystery when choosing how to write it. Selecting a single point of view where the reader only knows what that character knows won’t work, because the reader needs to know MORE than your protagonist. How does the author get that MORE to the reader? More point of view characters who present that information or an omniscient narrator, and this can be tricky. Who reveals the information? How much to they reveal? And how will the information revealed lead your hero into danger and your reader to ramp up their emotional reaction to the story?
Let’s go back to Alfred Hitchcock and his 1960 movies “Psycho”. At the end of the film, Norman moves his mother to the basement. The viewer sees him do this with his mother complaining and berating him the whole time. We (the viewer) have seen his mother brutally murder Janet Leigh. When her sister (Vera Miles) decides to GO INTO THE BASEMENT, we already know something dangerous and evil is lurking down there, we KNOW something bad is going to happen (but WHAT?), and it builds the emotion and fear and suspense of the scene to a screaming point (I was actually screaming DON’T GO INTO THE BASEMENT) because of what was revealed to us OUTSIDE of a single point of view character.
Mystery versus Suspense. Got it? Because I’m about to add one more definition to the mix. What’s a THRILLER? A thriller is when the sleuth in a mystery is not some detached police officer or PI or even the nosy neighbor in a town filled with cottages and cats.
A Thriller is when an author takes their mystery sleuth and puts that character in danger.
I guess when I think about it, I like to blend all three in my stories: MYSTERY—SUSPENSE—THRILLER.