SixthSense (device)
Encyclopedia
Sixthsense is a wearable gestural interface
device by Pranav Mistry
, a PhD student in the Fluid Interfaces Group at the MIT Media Lab
. The concept is similar to Telepointer
, a neckworn projector/camera system developed by Media Lab student Steve Mann
(which Mann originally referred to as "Synthetic Synesthesia of the Sixth Sense").
, a mirror and a camera contained in a pendant like, wearable device. Both the projector and the camera are connected to a mobile computing
device in the user’s pocket. The projector projects visual information enabling surfaces, walls and physical objects around us to be used as interfaces; while the camera recognizes and tracks users' hand gestures and physical objects using computer-vision based techniques. The software program processes the video stream data captured by the camera and tracks the locations of the colored markers (visual tracking fiducials) at the tips of the user’s fingers. The movements and arrangements of these fiducials are interpreted into gestures that act as interaction instructions for the projected application interfaces. SixthSense supports multi-touch and multi-user interaction.
The system recognizes a user's freehand gestures as well as icons/symbols drawn in the air with the index finger, for example:
Gesture recognition
Gesture recognition is a topic in computer science and language technology with the goal of interpreting human gestures via mathematical algorithms. Gestures can originate from any bodily motion or state but commonly originate from the face or hand. Current focuses in the field include emotion...
device by Pranav Mistry
Pranav Mistry
Pranav Mistry is one of the inventors of SixthSense. He is a research assistant and a PhD candidate at MIT Media Lab. Before joining MIT he worked as a UX Researcher with Microsoft. He received Master in Media Arts and Sciences from MIT and Master of Design from IIT Bombay. He has completed...
, a PhD student in the Fluid Interfaces Group at the MIT Media Lab
MIT Media Lab
The MIT Media Lab is a laboratory of MIT School of Architecture and Planning. Devoted to research projects at the convergence of design, multimedia and technology, the Media Lab has been widely popularized since the 1990s by business and technology publications such as Wired and Red Herring for a...
. The concept is similar to Telepointer
Telepointer
Telepointer is a neckworn gestural interface system developed by MIT Media Lab student Steve Mann in 1998. Mann originally referred to the device as "Synthetic Synesthesia of the Sixth Sense"...
, a neckworn projector/camera system developed by Media Lab student Steve Mann
Steve Mann
Steven Mann , is a tenured professor at the Department of Electrical and Computer Engineering at the University of Toronto.-Education:...
(which Mann originally referred to as "Synthetic Synesthesia of the Sixth Sense").
Construction and workings
The SixthSense prototype comprises a pocket projectorVideo projector
A video projector is an image projector that receives a video signal and projects the corresponding image on a projection screen using a lens system. All video projectors use a very bright light to project the image, and most modern ones can correct any curves, blurriness, and other...
, a mirror and a camera contained in a pendant like, wearable device. Both the projector and the camera are connected to a mobile computing
Mobile computing
Mobile computing is a form of human–computer interaction by which a computer is expected to be transported during normal usage. Mobile computing has three aspects: mobile communication, mobile hardware, and mobile software...
device in the user’s pocket. The projector projects visual information enabling surfaces, walls and physical objects around us to be used as interfaces; while the camera recognizes and tracks users' hand gestures and physical objects using computer-vision based techniques. The software program processes the video stream data captured by the camera and tracks the locations of the colored markers (visual tracking fiducials) at the tips of the user’s fingers. The movements and arrangements of these fiducials are interpreted into gestures that act as interaction instructions for the projected application interfaces. SixthSense supports multi-touch and multi-user interaction.
Example applications
The SixthSense prototype contains a number of demonstration applications.- The map application lets the user navigate a map displayed on a nearby surface using hand gestures to zoom and pan
- The drawing application lets the user draw on any surface by tracking the fingertip movements of the user’s index finger.
- SixthSense also implements Augmented realityAugmented realityAugmented reality is a live, direct or indirect, view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data. It is related to a more general concept called mediated reality, in which a view of reality is...
; projecting information onto objects the user interacts with.
The system recognizes a user's freehand gestures as well as icons/symbols drawn in the air with the index finger, for example:
- A 'framing' gesture takes a picture of the scene. The user can stop by any surface or wall and flick through the photos he/she has taken.
- Drawing a magnifying glass symbol takes the user to the map application while an ‘@’ symbol lets the user check his mail.
- The gesture of drawing a circle on the user’s wrist projects an analog watch.