I was curious about this so I did some reading and figuring out a while ago.
Even going back in time on the same day WILL count as time travel IF you've played at that time before.
So if it's 11:00pm in your game, you cannot set your clock to anything earlier than that or the game will count it as time travel.
You can save and quit, and put it aside for days and days, and then set the clock to 11:01pm today and it will be fine. It's because 11:01pm on Oct 14th has not passed in your game.
Does this make sense? I hope so.
Edit:
http://www.thonky.com/animal-crossing-new-leaf/stalk-market This is where I first read about what the game considers time travel.