MagPen: A Novel Method of Digitizing Notes Using Magnets
This paper presents a novel method of digitizing notes and/or diagrams that are drawn on a sheet of paper. Most modern phones contain magnetometers that output the strength of the surrounding magnetic field in the x, y, and z direction. If a magnet is brought closer to the device (and the magnetometer), the values from the magnetometer will be altered. By determining the change in the altered magnetic field, we can determine the position of the magnet. With the position, we can determine the location of the magnet relative to the phone. We have created a magnet based pen (MagPen) and built an android application that allows users to write notes on a sheet of paper while their mobile phone automatically digitizes. Users will also be able to perform certain actions using the button on the pen and select various pen attributes using the MagPen.
Pen input, magnetometer, mobile devices, notes
ACM Classification Keywords
H.5.2 [Information interfaces and presentation]: User Interfaces: Input Devices and Strategies.
A mobile device has a multitude of sensors ranging from GPS to a barometer. One sensor that is used daily is the capacitive sensor. Capacitive sensors detect anything that is conductive and they are used for touch input in mobile phones. We rely tremendously on touch input to interact with our devices ranging from playing games to checking notifications. We also have the ability to use our finger or styluses to write notes directly on a mobile device. However, touch/stylus input on mobile devices has its own set of problems. We are unable to easily draw detailed diagrams or write notes using the touch screen.
Is there another way to provide a different method of digitizing notes with a mobile device that expands the capabilities of touch? Another sensor that can be used for alternative methods of interaction is a magnetometer. The magnetometer senses changes in the magnetic field in all three axes (x,y,z) (Ketabdar 2011). If the magnetic field can be detected, then it is possible to alter the magnetic field by using an external magnet. When the external magnet is moved around, it will alter the magnetic field thus producing different (x,y,z) values.
MagPen is a system that allows the user to digitize notes while they are written on a sheet of paper. The pen itself contains magnets or it can be an electromagnet. As the pen moves around in the area next to the phone, the changes in the magnetic field are detected and they are mapped out onto an x-y plane. To simulate an actual pen, the pen will only emit a magnetic field when the tip is pressed down. That allows the phone to only recognize actual pen inputs and not the pen movements.
To enable a richer set of interactions, the magnetic field strength can be used to control various pen attributes, such as stroke size or color. This is accomplished by moving the magnet based pen closer and further away to control the size of color. There has been existing work in the industry that attempts to solve this problem. A few examples are Livescribe, Equil JOT, or the Wacom Tablets. This devices however require additional devices and/or special pen and paper to be able to digitize these notes. MagPen attempts to remove the need of additional devices by just requiring your mobile device and a custom pen.
This project spans across multiple different fields, tangible interactions, magnet based interactions, drawing with everyday objects, and pens & touch. We have taken ideas from existing research in these areas and integrated them into MagPen.
MagPen overall is focusing on creating alternative methods of input and interactions that use magnets embedded in real world objects. According to Tangible Bits (Ishii 1999), there are three concepts for tangible interaction: transforming surfaces into an active interface, coupling physical objects with digital information, and the use of ambient media with the digital world. In the context of MagPen, the real world objects can be generic (e.g. a pointing device) or embedded in other objects (e.g. pens). MagPen will make use of tangible interactions by using magnets embedded in pens or markers. Users will be able to take notes or draw on a sheet of paper beside the phone and control pen attributes using the magnetic field strength.
The use of a magnet to draw on the screen is a complex interaction on its own but is there a way to combine touch and magnet based input (physically and mentally). TUIC looks into just this idea (Yu 2011). It allows tangible interaction directly on multi touch devices. It embeds objects with circuits that simulate touch input to allow the mobile device to detect the object. There are three methods that "TUIC" achieves this, spatial (static touch patterns), frequency (dynamic modulation of touch), and hybrid (a combination of spatial and frequency). Although this would not directly apply to MagPen since it is possible to detect magnets without touching the phone, we will be looking into how we can detect different sizes of magnets and various locations. This can be done using the magnet's intensity or frequency modulation (as described in TUIC). Frequency modulation could be used in a unique way to build custom hardware to modulate the magnetic field and detect different kinds of pens. However this will be considered as future work as per the scope of this project. In addition to detecting various pen types, we will be integrating interaction elements using a button on the pen.
"Tangible Meets Gestural" (Ali Mazalek) mentions the value of learning and thinking is greater when using physical objects. Touching physical objects can help children learn how to count and keep track of their activities. We believe with MagPen, the value of using an actual pen will be greatly beneficial since it does not replace the user’s natural way of taking notes.
Magnet Based Interactions
Several works have investigated input methods making use of magnetometers, whether they are looking for more absolute locators mimicking a mouse or more gestural input. uTrack (Chen 2013) implements an absolute locator using a magnet attached to the user's thumb and two magnetometers attached to their ring finger. The combined readings from the two sensors allow a fairly accurate location reading for the magnet. While this does show that it is possible, the implementation also highlights that a single sensor does have its limitations, especially when compromises are made such as in smartphones. Smartphone sensor reliability for augmented reality applications (Blum 2013) touches on the inherent inaccuracy of cheaper sensors used in smartphones. In addition, a lot of attention is paid to how external forces can affect the readings. Improving Heading Accuracy in Smartphone-based PDR Systems using Multi-Pedestrian Sensor Fusion (Abadi 2013) investigates this further and attempts to improve upon the results by fusing readings from multiple sensors. In their studies, they were able to reduce the error in the heading readings by some 27% using only naive averaging, leaving room for improvement using more complex algorithms.
MagPairing (Jin 2014) attempts to make use of the unique magnetic forces that a device will pick up in a given location. By tapping the phones, you move them close enough together that the magnetic fields picked up by each device are extremely similar. They are able to reliable pair devices together by encoding these readings with authentication keys and comparing similarly time stamped readings to verify the connection.
Although, these do highlight how difficult and complex it can be to make these sensors accurate enough to be used as absolute locators, there is a lot that can be done by using them in a relative sense, especially as a form of gestural input. Not only can gestures be reliably recorded, but gestures made in 3D Space are also unique to every user. If two users made box like gestures over the phone, both of their gestures would be somewhat different. This is due to users not being able to reproduce each other's gestures in 3D space within a certain threshold (Shirazi 2012). As we are developing this application, we need to take into account this threshold to be able to recognize certain swipe gestures across all users.
MagiMusic (Ketabdar 2011) and Magnetic Marionette (Hwang 2013) show that these gesture-based movements are a lot more feasible with the magnetometer than absolute inputs. MagiMusic allows digital instruments to be played by making gestures with a magnet, such as strumming a guitar.