Demonstrating Touch & Magnet Based Interactions with a Photo Browser (Piknik)


This paper presents an alternative method of interacting with mobile devices. Most modern phones contain magnetometers that output the strength of the surrounding magnetic field in the x, y, and z direction. If a magnet is brought closer to the device (and the magnetometer), the values from the magnetometer will be altered. By determining the change in the altered magnetic field, we can determine the position of the magnet. With the position, we can determine the location of the magnet relative to the screen and detect gestures above the screen in 3D space. We will be building a photo browser & drawing application that will contain implementation for magnet based interactions. Users can flip through, select, and annotate/draw on photos using external magnets.

Author Keywords

Gesture-based interaction, magnetometer, mobile devices

ACM Classification Keywords

H.5.2 [Information interfaces and presentation]: User
Interfaces: Input Devices and Strategies.


A mobile device has a multitude of sensors ranging from GPS to a barometer. One sensor that is used daily is the capacitive sensor. Capacitive sensors detect anything that is conductive and they are used for touch input in mobile phones. We rely tremendously on touch input to interact with our devices ranging from playing games to checking notifications. However, touch input on mobile devices have its own sets of problems. We are unable to easily interact with our phone due to phones being too large, and not large enough thus limiting user interactions.

Is there a way to provide another method of interaction with your mobile device that expands the capabilities of touch? Another sensor that can be used for alternative methods of interaction is a magnetometer. The magnetometer senses changes in the magnetic field in all three axes (x,y,z) (Ketabdar 2011). If the magnetic field can be detected, would there be a way to alter the magnetic field? This can be accomplished by using an external magnet. The external magnet will alter the magnetic field thus producing different (x,y,z) values.

This new user interaction requires the use of a magnet attached or placed on a finger or embedded in a ring. The use of a magnet and a magnet sensor will increase the area of user interaction around and above the phone. Gestures and taps in 2D and 3D space can be recognized and attached to certain actions. This would allow interaction with a mobile device without physical contact with that device. For example, interacting with your phone while its in your pocket or purse. These are just a few use possibilities of interacting with your phone by using a magnet.

Using the magnet based interactions described above, we will be building a photo browser + drawing application that will allow you to flip through, select, and annotation/draw on photos using magnets and touch. Magnets also can be embedded on physical objects, such as styluses (which will represent paintbrushes). This will allow the user to have a more natural interaction with the app (Ali Mazalek).