It is generally believed that time flows in one direction and that a reversal of time’s arrow would render the external world nonsensical. We evaluated our ability to tell the direction of time’s arrow in a wide range of dynamic scenes in our daily life by presenting 360 video clips in the correct or incorrect direction. Participants, who judged the direction in a speeded manner, erred in 39% of trials when a video was played in reverse, but in only 9% when it was played normally. Due to the bias favouring the “forward” judgement, reaction was generally faster for the forward response. However, the reaction became paradoxically faster and more synchronous for the detection of reversal in some critical occasions such as forward motion, free fall, diffusion, division, and addition of materials by hand. Another experiment with a fraction of the video clips revealed that reversal replay of these videos provided instantaneous evidence strong enough to overtake the forward judgment bias. We suggest that our brain is equipped with a system that predicts how the external organisms behave or move in these critical occasions and that the prediction error of the system contributes to the fast “reversal” detection.