Imagine that you are watching TV in your living room. An item (e.g. a dress or perfume) on the screen catches your eyes, and you want to find out where you can buy it. Wouldn’t it be wonderful if you could simply aim the remote at the object, press a button to select it while the TV program is still progressing? Then the TV will automatically send the information to a search engine and tell you where to purchase this item.

In this project, we develop an interactive prototype system that enables users to directly select objects from the video using their hand gestures. In our system, users’ hand movements are mapped to the cursor control and used to select the object. Once an object in a video scene is selected, users can perform basic queries (e.g., product name, price, store information) on the object.