Is time moving forward or backward? Computers learn to spot the difference news.sciencemag.org/brain-behavior/2014/06/time-moving-forward-or-backward-computers-learn-spot-difference Play the video above, and it won’t take you long to notice there’s something off about it. Rather than drifting down toward the forest below, the snow is falling …up? Then it hits you: The video is running backward. Now, a computer watching the movie could arrive at that same realization. For the first time, scientists have taught computers to figure out the direction of time in videos, a result that could help researchers better understand our own perception of time. Computer vision—a field focused on how computers could learn to detect objects, motion, or even human emotions and intentions in photographs and video—has long fascinated computer scientists, with applications ranging from more realistic-looking video games to video surveillance and intelligence. At the same time, computer vision has attracted the interest of psychologists, who hope to use it to study how people turn raw visual data into an understanding of the world around them. The latter question led Lyndsey Pickup, now a human and computer vision researcher at Mirada Medical in Oxford, U.K., to wonder how we tell the difference between time running forward and backward in movies. After all, we only ever observe time running forward in the real world—a concept called time’s arrow. How do we pick up on it when a movie has put time arrow’s in reverse? And might she be able to teach computers to make the same observation? To find out, she and her collaborators broke down 180 YouTube videos into square patches of a few hundred pixels, which they further divided into four-by-four grids. Combining standard techniques for discovering objects in still photographs with motion detection algorithms, the researchers identified 4000 typical patterns of motion, or “flow words,” across a grid’s 16 cells. The gentle downward drifting of snowflakes, for example, would be one flow word. From those patterns, the team created flow word descriptions of each video along with three other versions—a time-reversed version, a mirror-image version, and a mirror-image and time-reversed version. Then, they made a computer program watch 120 of these clips, training it to identify which flow words best revealed whether a video ran forward or backward. When they tested their program on the remaining 60 videos, the trained computers could correctly determine whether a video ran forward or backward 80% of the time, the team reported this week at the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pamitc.org/cvpr14/) in Columbus, Ohio. A closer analysis found that flow words associated with divergence (water splashing outward as someone dives into a pool) or dissipation (a steam train’s exhaust spreading out in air) were especially good indicators of the direction in which time was moving. In principle, the work could provide clues about how humans, as well as computers, perceive time passing, Pickup says, and reverse-time videos could play a role akin to the optical illusions and reverse-color photographs psychologists have used to study visual perception in the past. Seeing time running backward is “really captivating,” and continuing the research might help her team figure out why, she says. Shai Avidan, an associate professor of engineering at Tel Aviv University in Israel who wasn’t involved with the work, says that the team is the first to ask, “Can we analyze and understand” what attributes distinguish backward from forward video? It’s an intriguing if largely academic question. Still, practical applications may be possible. Avidan compares the flow words approach to earlier work on identifying different kinds of texture in still images. Though studying image textures was considered “superficial” 15 years ago, he says, the tools developed to do so have since proved essential for engineers working on how to reduce noise in low-light digital photographs and other images. By identifying the subtle features that indicate time’s arrow and separating those from noise, flow words could play a similar role for improving otherwise fuzzy low-light video, Avidan ventures. Regardless of any possible applications, “we just thought it was a great problem,” says Pickup’s co-author and computer scientist William Freeman of the Massachusetts Institute of Technology in Cambridge. Pickup agrees: Teaching computers to see the arrow of time combines computer science, physics, and human perception to get at the heart of the question, “How do we understand the visual world?”
Posted on: Tue, 08 Jul 2014 23:14:58 +0000