Demonstrating Touch & Magnet Based Interactions with a Photo Browser (Piknik)
This paper presents an alternative method of interacting with mobile devices. Most modern phones contain magnetometers that output the strength of the surrounding magnetic field in the x, y, and z direction. If a magnet is brought closer to the device (and the magnetometer), the values from the magnetometer will be altered. By determining the change in the altered magnetic field, we can determine the position of the magnet. With the position, we can determine the location of the magnet relative to the screen and detect gestures above the screen in 3D space. We will be building a photo browser & drawing application that will contain implementation for magnet based interactions. Users can flip through, select, and annotate/draw on photos using external magnets.
Gesture-based interaction, magnetometer, mobile devices
ACM Classification Keywords
H.5.2 [Information interfaces and presentation]: User Interfaces: Input Devices and Strategies.
A mobile device has a multitude of sensors ranging from GPS to a barometer. One sensor that is used daily is the capacitive sensor. Capacitive sensors detect anything that is conductive and they are used for touch input in mobile phones. We rely tremendously on touch input to interact with our devices ranging from playing games to checking notifications. However, touch input on mobile devices have its own sets of problems. We are unable to easily interact with our phone due to phones being too large, and not large enough thus limiting user interactions.
Is there a way to provide another method of interaction with your mobile device that expands the capabilities of touch? Another sensor that can be used for alternative methods of interaction is a magnetometer. The magnetometer senses changes in the magnetic field in all three axes (x,y,z) (Ketabdar 2011). If the magnetic field can be detected, would there be a way to alter the magnetic field? This can be accomplished by using an external magnet. The external magnet will alter the magnetic field thus producing different (x,y,z) values.
This new user interaction requires the use of a magnet attached or placed on a finger or embedded in a ring. The use of a magnet and a magnet sensor will increase the area of user interaction around and above the phone. Gestures and taps in 2D and 3D space can be recognized and attached to certain actions. This would allow interaction with a mobile device without physical contact with that device. For example, interacting with your phone while its in your pocket or purse. These are just a few use possibilities of interacting with your phone by using a magnet.
Using the magnet based interactions described above, we will be building a photo browser + drawing application that will allow you to flip through, select, and annotation/draw on photos using magnets and touch. Magnets also can be embedded on physical objects, such as styluses (which will represent paintbrushes). This will allow the user to have a more natural interaction with the app (Ali Mazalek).
This project spans across multiple different fields, tangible interactions, magnet based interactions, and drawing with every day objects. We have taken ideas from existing research in these areas and built on top of it.
Piknik overall is focusing on creating alternative methods of interactions that use magnets embedded in real world objects. According to Tangible Bits (Ishii 1997), there are three concepts for tangible interaction: transforming surfaces into an active interface, coupling physical objects with digital information, and the use of ambient media with the digital world. In the context of Piknik, the real world objects can be generic (e.g. a pointing device) or embedded in other objects (e.g. paintbrushes). Piknik will make use of tangible interactions by using magnets embedded in rings or styluses (representing paintbrushes).
The use of magnets to paint on the screen or to flip through images is a rich interaction on its own but is there a way to combine touch and magnet based input (physically and mentally). TUIC (Yu 2011) looks into just this idea. It allows tangible interaction directly on multi touch devices. It embeds objects with circuits that simulate touch input to allow the mobile device to detect the object. There are three methods that "TUIC" achieves this, spatial (static touch patterns), frequency (dynamic modulation of touch), and hybrid (a combination of spatial and frequency). Although this would not directly apply to Piknik since it is possible to detect magnets without touching the phone, we will be looking into how we can detect different sizes of magnets and various locations. This can be done using the magnet's intensity or frequency modulation (as described in TUIC). Frequency modulation could be used in a unique way to build custom hardware to modulate the magnetic field. However due to the timeframe of this project, this will be considered as future work.
"Tangible Meets Gestural" (Ali Mazalek) mentions the value of learning and thinking is greater when using physical objects. Touching physical objects can help children learn how to count and keep track of their activities. We will use the findings of this paper help us create physical objects to allow users to draw with (e.g. styluses or paint brushes).
Magnet Based Interactions
Several works have investigated input methods making use of magnetometers, whether they are looking for more absolute locators mimicking a mouse or more gestural input. uTrack (Chen 2013) implements an absolute locator using a magnet attached to the user's thumb and two magnetometers attached to their ring finger. The combined readings from the two sensors allow a fairly accurate location reading for the magnet. While this does show that it is possible, the implementation also highlights that a single sensor does have its limitations, especially when compromises are made such as in smartphones. Smartphone sensor reliability for augmented reality applications (Blum 2013) touches on the inherit inaccuracy of cheaper sensors used in smartphones. In addition, a lot of attention is paid to how external forces can affect the readings. Improving Heading Accuracy in Smartphone-based PDR Systems using Multi-Pedestrian Sensor Fusion (Abadi 2013) investigates this further and attempts to imrove upon the results by fusing readings from multiple devices. In their studies, they were able to reduce the error in the heading readings by some 27% using only naive averaging, leaving room for improvement using more complex algorithms. MagPairing (Jin 2014) attempts to make use of the unique magnetic forces that a device will pick up in a given location. By tapping the phones, you move them close enough together that the magnetic fields picked up by each device are extremely similar. They are able to reliable pair devices together by encoding these readings with authentication keys and comparing similarly time stamped readings to verify the connection.
Although, these do highlight how difficult and complex it can be to make these sensors accurate enough to be used as absolute locators, there is a lot that can be done by using them in a relative sense, especially as a form of gestural input. Not only can gestures be reliably recorded, but gestures made in 3D Space are also unique to every user. If two users made box like gestures over the phone, both of their gestures would be somewhat different. This is due to users not being able to reproduce each other's gestures in 3D space within a certain threshold (Shirazi 2012). As we are developing this application, we need to take into account this threshold to be able to recognize certain swipe gestures across all users. MagiMusic (Ketabdar 2011) and Magnetic Marionette (Hwang 2013) show that these gesture-based movements are a lot more feasible with the magnetometer than absolute inputs. MagiMusic allows digital instruments to be played by making gestures with a magnet, such as strumming a guitar. Magnetic Marionette introduces a tangible avatar attached to the device, which can be moved around to produce different facial expressions on the screen.
While the sensors in these smartphones are generally inaccurate due to compromises made in the devices manufacturing and the effect of external magnetic forces, they can be made to be more accurate in a fixed setting. If you know the ambient forces you can account for them in order to pick out the desired readings. In Pulse (Weiwei Jiang 2014) for example, a system is designed through which the device can recieve and decode a signal sent out as magnetic forces, and can even reach transfer speeds up to 44 bps. Due to the nature of the sensors used, it is an especially short-range communication method. A Sensor Fusion Method for Smart phone Orientation Estimation (Ayub 2012) also provides methods to work around the inherit issues with these sensors by fusing data from multiple sensors for correction. Although the final implementation here is beyond this project, much of specifics are a good base for filtering some of the sensor data. For example the low-pass filter used to smooth the readings will prove important in Piknik, and adjustments to reference frames are something that very well may come into play.
Drawing with Everyday Objects
In addition to allowing the user to flip through photos and rotate them, Piknik will also offer the capability of drawing on existing photos using magnets. However, instead of just using magnets, we will be embedding the magnets in styluses (acting as paintbrushes) to allow the user to have a more natural interaction with the application. I/O Brush (Ryokai 2004) is similar idea, it is a drawing tool aimed at young children to be able to draw with everyday objects. The authors of I/O brush built a device that can select the color from anywhere in the real world and be able to paint with that color on the tablet. It molded its every day object as a brush so users can easily create the mental mappings and seamlessly work with the every day objects.
BlowBrush (Shen 2014) is another idea that looks into tangible painting systems using a windmill. A user can blow on the windmill to paint leaves or other objects on screen. BlowBrush used this criteria to analyze and study the effectiveness of their application. This criteria is consisted of: metaphorical affordance, enjoyable engagement, tangible manipulation, spatial interaction, embodied facilitation, and expressive representation. As we are developing this application, we need to look into the criteria that BlowBrush used to analyze their own system.