Demonstrating Touch & Magnet Based Interactions with a Photo Browser (Piknik)
This paper presents an alternative method of interacting with mobile devices. Most modern phones contain magnetometers that output the strength of the surrounding magnetic field in the x, y, and z direction. If a magnet is brought closer to the device (and the magnetometer), the values from the magnetometer will be altered. By determining the change in the altered magnetic field, we can determine the position of the magnet. With the position, we can determine the location of the magnet relative to the screen and detect gestures above the screen in 3D space. We will be building a photo browser & drawing application that will contain implementation for magnet based interactions. Users can flip through, select, and annotate/draw on photos using external magnets.
Gesture-based interaction, magnetometer, mobile devices
ACM Classification Keywords
H.5.2 [Information interfaces and presentation]: User Interfaces: Input Devices and Strategies.
A mobile device has a multitude of sensors ranging from GPS to a barometer. One sensor that is used daily is the capacitive sensor. Capacitive sensors detect anything that is conductive and they are used for touch input in mobile phones. We rely tremendously on touch input to interact with our devices ranging from playing games to checking notifications. However, touch input on mobile devices have its own sets of problems. We are unable to easily interact with our phone due to phones being too large, and not large enough thus limiting user interactions.
Is there a way to provide another method of interaction with your mobile device that expands the capabilities of touch? Another sensor that can be used for alternative methods of interaction is a magnetometer. The magnetometer senses changes in the magnetic field in all three axes (x,y,z) (Ketabdar 2011). If the magnetic field can be detected, would there be a way to alter the magnetic field? This can be accomplished by using an external magnet. The external magnet will alter the magnetic field thus producing different (x,y,z) values.
This new user interaction requires the use of a magnet attached or placed on a finger or embedded in a ring. The use of a magnet and a magnet sensor will increase the area of user interaction around and above the phone. Gestures and taps in 2D and 3D space can be recognized and attached to certain actions. This would allow interaction with a mobile device without physical contact with that device. For example, interacting with your phone while its in your pocket or purse. These are just a few use possibilities of interacting with your phone by using a magnet.
Using the magnet based interactions described above, we will be building a photo browser + drawing application that will allow you to flip through, select, and annotation/draw on photos using magnets and touch. Magnets also can be embedded on physical objects, such as styluses (which will represent paintbrushes). This will allow the user to have a more natural interaction with the app (Ali Mazalek).
This project spans across multiple different fields, tangible interactions, magnet based interactions, and drawing with every day objects. We have taken ideas from existing research in these areas and built on top of it.
Piknik overall is focusing on creating alternative methods of interactions that use magnets embedded in real world objects. According to Tangible Bits (Ishii 1997), there are three concepts for tangible interaction: transforming surfaces into an active interface, coupling physical objects with digital information, and the use of ambient media with the digital world. In the context of Piknik, the real world objects can be generic (e.g. a pointing device) or embedded in other objects (e.g. paintbrushes). Piknik will make use of tangible interactions by using magnets embedded in rings or styluses (representing paintbrushes).
The use of magnets to paint on the screen or to flip through images is a rich interaction on its own but is there a way to combine touch and magnet based input (physically and mentally). TUIC (Yu 2011) looks into just this idea. It allows tangible interaction directly on multi touch devices. It embeds objects with circuits that simulate touch input to allow the mobile device to detect the object. There are three methods that "TUIC" achieves this, spatial (static touch patterns), frequency (dynamic modulation of touch), and hybrid (a combination of spatial and frequency). Although this would not directly apply to Piknik since it is possible to detect magnets without touching the phone, we will be looking into how we can detect different sizes of magnets and various locations. This can be done using the magnet's intensity or frequency modulation (as described in TUIC). Frequency modulation could be used in a unique way to build custom hardware to modulate the magnetic field. However due to the timeframe of this project, this will be considered as future work.
"Tangible Meets Gestural" (Ali Mazalek) mentions the value of learning and thinking is greater when using physical objects. Touching physical objects can help children learn how to count and keep track of their activities. We will use the findings of this paper help us create physical objects to allow users to draw with (e.g. styluses or paint brushes).
Magnet Based Interactions
Several works have investigated input methods making use of magnetometers, whether they are looking for more absolute locators mimicking a mouse or more gestural input. uTrack (Chen 2013) implements an absolute locator using a magnet attached to the user's thumb and two magnetometers attached to their ring finger. The combined readings from the two sensors allow a fairly accurate location reading for the magnet. While this does show that it is possible, the implementation also highlights that a single sensor does have its limitations, especially when compromises are made such as in smartphones. Smartphone sensor reliability for augmented reality applications (Blum 2013) touches on the inherit inaccuracy of cheaper sensors used in smartphones. In addition, a lot of attention is paid to how external forces can affect the readings. Improving Heading Accuracy in Smartphone-based PDR Systems using Multi-Pedestrian Sensor Fusion (Abadi 2013) investigates this further and attempts to imrove upon the results by fusing readings from multiple devices. In their studie