I believe that one of the best ways to create an unexpected/unpredicted storyline is to play on people's assumptions.
I've personally done this a few times. In my first story, I'm sure most people would have read it assuming that the narrator was going to turn out to be gay -- the fact that he wasn't was the first surprise. It was a surprise because the target audience is made up of mainly gay readers -- they weren't going to be expecting a story that's not centred on a gay character. Another example is my short story Ambush but I won't ruin that one by saying what assumption most readers would have made.
The mark of a very good short story is the twist at the end that makes the reader go back to the start of the story and revisit what they've read because they realise they've made some basic incorrect assumptions. Two examples of that from recent anthologies would be Bardeara's It's Time To Go and C James's Category 5. Both of these show how a reader can be led to make assumptions, and then be shown that those assumptions have been incorrect -- and hence resulted in an unexpected/unpredicted change in the story. Extending that out in a novel is more difficult, but achievable. I believe I did that in part two of my novel Heart of The Tree where I carefully led the readers into making assumptions, only to find out that they'd made a mistake.
I don't believe you can create an unpredictable storyline without a completely outside incident taking over, because that assumes that no reader will pick up on the preceding events. Taking a real life example, 9/11 was unpredicted by the vast majority of the world, but there were a few people for which is wasn't a complete surprise because they had been expecting something. In a story, however, that is close enough to a completely outside incident that for most people it's just setting up an environment for the story to continue in a new direction.