MagPen: A Novel Method of Digitizing Notes Using Magnets
This paper presents a novel method of digitizing notes and/or diagrams that are drawn on a sheet of paper. Most modern phones contain magnetometers that output the strength of the surrounding magnetic field in the x, y, and z direction. If a magnet is brought closer to the device (and the magnetometer), the values from the magnetometer will be altered. By determining the change in the altered magnetic field, we can determine the position of the magnet. With the position, we can determine the location of the magnet relative to the phone. We have created a magnet based pen (MagPen) and built an android application that allows users to write notes on a sheet of paper while their mobile phone automatically digitizes. Users will also be able to perform certain actions using the button on the pen and select various pen attributes using the MagPen.
Pen input, magnetometer, mobile devices, notes
ACM Classification Keywords
H.5.2 [Information interfaces and presentation]: User Interfaces: Input Devices and Strategies.
A mobile device has a multitude of sensors ranging from GPS to a barometer. One sensor that is used daily is the capacitive sensor. Capacitive sensors detect anything that is conductive and they are used for touch input in mobile phones. We rely tremendously on touch input to interact with our devices ranging from playing games to checking notifications. We also have the ability to use our finger or styluses to write notes directly on a mobile device. However, touch/stylus input on mobile devices has its own set of problems. We are unable to easily draw detailed diagrams or write notes using the touch screen.
Is there another way to provide a different method of digitizing notes with a mobile device that expands the capabilities of touch? Another sensor that can be used for alternative methods of interaction is a magnetometer. The magnetometer senses changes in the magnetic field in all three axes (x,y,z) (Ketabdar 2011). If the magnetic field can be detected, then it is possible to alter the magnetic field by using an external magnet. When the external magnet is moved around, it will alter the magnetic field thus producing different (x,y,z) values.
MagPen is a system that allows the user to digitize notes while they are written on a sheet of paper. The pen itself contains magnets or it can be an electromagnet. As the pen moves around in the area next to the phone, the changes in the magnetic field are detected and they are mapped out onto an x-y plane. To simulate an actual pen, the pen will only emit a magnetic field when the tip is pressed down. That allows the phone to only recognize actual pen inputs and not the pen movements.
To enable a richer set of interactions, the magnetic field strength can be used to control various pen attributes, such as stroke size or color. This is accomplished by moving the magnet based pen closer and further away to control the size of color. There has been existing work in the industry that attempts to solve this problem. A few examples are Livescribe, Equil JOT, or the Wacom Tablets. This devices however require additional devices and/or special pen and paper to be able to digitize these notes. MagPen attempts to remove the need of additional devices by just requiring your mobile device and a custom pen.
This project spans across multiple different fields, tangible interactions, magnet based interactions, drawing with everyday objects, and pens & touch. We have taken ideas from existing research in these areas and integrated them into MagPen.
MagPen overall is focusing on creating alternative methods of input and interactions that use magnets embedded in real world objects. According to Tangible Bits (Ishii 1999), there are three concepts for tangible interaction: transforming surfaces into an active interface, coupling physical objects with digital information, and the use of ambient media with the digital world. In the context of MagPen, the real world objects can be generic (e.g. a pointing device) or embedded in other objects (e.g. pens). MagPen will make use of tangible interactions by using magnets embedded in pens or markers. Users will be able to take notes or draw on a sheet of paper beside the phone and control pen attributes using the magnetic field strength.
The use of a magnet to draw on the screen is a complex interaction on its own but is there a way to combine touch and magnet based input (physically and mentally). TUIC looks into just this idea (Yu 2011). It allows tangible interaction directly on multi touch devices. It embeds objects with circuits that simulate touch input to allow the mobile device to detect the object. There are three methods that "TUIC" achieves this, spatial (static touch patterns), frequency (dynamic modulation of touch), and hybrid (a combination of spatial and frequency). Although this would not directly apply to MagPen since it is possible to detect magnets without touching the phone, we will be looking into how we can detect different sizes of magnets and various locations. This can be done using the magnet's intensity or frequency modulation (as described in TUIC). Frequency modulation could be used in a unique way to build custom hardware to modulate the magnetic field and detect different kinds of pens. However this will be considered as future work as per the scope of this project. In addition to detecting various pen types, we will be integrating interaction elements using a button on the pen.
"Tangible Meets Gestural" (Ali Mazalek) mentions the value of learning and thinking is greater when using physical objects. Touching physical objects can help children learn how to count and keep track of their activities. We believe with MagPen, the value of using an actual pen will be greatly beneficial since it does not replace the user’s natural way of taking notes.
Magnet Based Interactions
Several works have investigated input methods making use of magnetometers, whether they are looking for more absolute locators mimicking a mouse or more gestural input. uTrack (Chen 2013) implements an absolute locator using a magnet attached to the user's thumb and two magnetometers attached to their ring finger. The combined readings from the two sensors allow a fairly accurate location reading for the magnet. While this does show that it is possible, the implementation also highlights that a single sensor does have its limitations, especially when compromises are made such as in smartphones. Smartphone sensor reliability for augmented reality applications (Blum 2013) touches on the inherent inaccuracy of cheaper sensors used in smartphones. In addition, a lot of attention is paid to how external forces can affect the readings. Improving Heading Accuracy in Smartphone-based PDR Systems using Multi-Pedestrian Sensor Fusion (Abadi 2013) investigates this further and attempts to improve upon the results by fusing readings from multiple sensors. In their studies, they were able to reduce the error in the heading readings by some 27% using only naive averaging, leaving room for improvement using more complex algorithms.
MagPairing (Jin 2014) attempts to make use of the unique magnetic forces that a device will pick up in a given location. By tapping the phones, you move them close enough together that the magnetic fields picked up by each device are extremely similar. They are able to reliable pair devices together by encoding these readings with authentication keys and comparing similarly time stamped readings to verify the connection.
Although, these do highlight how difficult and complex it can be to make these sensors accurate enough to be used as absolute locators, there is a lot that can be done by using them in a relative sense, especially as a form of gestural input. Not only can gestures be reliably recorded, but gestures made in 3D Space are also unique to every user. If two users made box like gestures over the phone, both of their gestures would be somewhat different. This is due to users not being able to reproduce each other's gestures in 3D space within a certain threshold (Shirazi 2012). As we are developing this application, we need to take into account this threshold to be able to recognize certain swipe gestures across all users.
MagiMusic (Ketabdar 2011) and Magnetic Marionette (Hwang 2013) show that these gesture-based movements are a lot more feasible with the magnetometer than absolute inputs. MagiMusic allows digital instruments to be played by making gestures with a magnet, such as strumming a guitar. Magnetic Marionette introduces a tangible avatar attached to the device, which can be moved around to produce different facial expressions on the screen.
While the sensors in these smartphones are generally inaccurate due to compromises made in the devices manufacturing and the effect of external magnetic forces, they can be made to be more accurate in a fixed setting. If you know the ambient forces you can account for them in order to pick out the desired readings. In Pulse (Weiwei Jiang 2014) for example, a system is designed through which the device can receive and decode a signal sent out as magnetic forces, and can even reach transfer speeds up to 44 bps. Due to the nature of the sensors used, it is an especially short-range communication method. A Sensor Fusion Method for Smart phone Orientation Estimation (Ayub 2012) also provides methods to work around the inherent issues with these sensors by fusing data from multiple sensors for correction. Although the final implementation here is beyond this project, much of specifics are a good base for filtering some of the sensor data. For example the low-pass filter used to smooth the readings will prove important in MagPen, and adjustments to reference frames are something that very well may come into play.
Drawing with Everyday Objects
The android application in MagPen relies on detecting changes in the magnetic field with external magnets. However, instead of just using magnets, we will be embedding the magnets in pens to allow the user to have a more natural interaction with the application. I/O Brush (Ryokai 2004) is similar idea, it is a drawing tool aimed at young children to be able to draw with everyday objects. The authors of I/O brush built a device that molded its everyday object as a brush so users can easily create the mental mappings and seamlessly work with the everyday objects.
BlowBrush (Shen 2014) is another idea that looks into tangible painting systems using a windmill. A user can blow on the windmill to paint leaves or other objects on screen. BlowBrush used this criteria to analyze and study the effectiveness of their application. This criteria is consisted of: metaphorical affordance, enjoyable engagement, tangible manipulation, spatial interaction, embodied facilitation, and expressive representation. As we are developing this application, we need to look into the criteria that BlowBrush used to analyze their own system. We especially need to evaluate the tangible and spatial interactions of our system, so that the use of the pen around the device is as intuitive and frustration-free as possible.
** Pens and Touch ** Pens and Touch focuses on pen based interactions with devices that have touch capabilities. This can include augmenting touch with a stylus/pen or replacing touch completely. “Pen + Touch” aims to augment touch with the capabilities and benefits of using a pen (Ken Hinckley). The pen is relatively more accurate, has a small area of occlusion, and provides a natural way of interacting with objects. Although MagPen does not aim to augment touch completely, it contains the ability to combine touch and pen input. For example, as MagPen is recording the pen input, the user can make annotations directly on the screen using touch. We believe there is a larger opportunity here to combine pen and touch input.
Another paper also called “MagPen” (unknown to us at the time of our app’s creation) looked into pen interaction directly on or around smartphones (Hwang 2013). It combined a stylus and a magnet to allow touchscreen input while controlling various attributes of the pen. One interesting concept that was implemented was changing the brush stroke size based on the magnetic strength. As the magnet in the pen moved closer the brush stroke size became larger. Also as the magnetic field strength changed, the stroke color options also changed. We will be implementing a similar functionality however the stroke size and color will need to be explicitly set and cannot be automatically detected. This is due to the fact that the user will be writing with the pen thus moving it around the phone. The magnetic field strength will change as the user moves the pen around the phone. Therefore this value will need to be explicitly set from a menu option. This also allows the user to set these attributes as they wish rather than having set attributes for certain pens.