It's usually not a problem for me either. My one notable encounter came as I am well into my third book. The situation may have been magnified by the fact that I seemed to have a lot of extras on my plate at the time, and time for working on the story or even to contemplate it were at minimum.

Usually I'll have a path I want the story to follow, and will be proceeding merrily along that plot route. Then a thought will strike me, and I'll say, "let's turn of here and take the story on a bit of a detour?" I do, and everything is fine. In the one instance where I had trouble and delays, I had three or four different routes that I could take to get to a specific point in the story. Finally chose one and went with it.
Dave