This project spans across multiple different fields, tangible interactions, magnet based interactions, and drawing with every day objects. We have taken ideas from existing research in these areas and built on top of it.
Tangible Interaction
Piknik overall is focusing on creating alternative methods of interactions that use magnets embedded in real world objects. According to Tangible Bits \cite{Ishii_1997}, there are three concepts for tangible interaction: transforming surfaces into an active interface, coupling physical objects with digital information, and the use of ambient media with the digital world.
In the context of Piknik, the real world objects can be generic (e.g. a pointing device) or embedded in other objects (e.g. paintbrushes). Piknik will make use of tangible interactions by using magnets embedded in rings or styluses (representing paintbrushes).
The use of magnets to paint on the screen or to flip through images is a rich interaction on its own but is there a way to combine touch and magnet based input (physically and mentally). TUIC \cite{Yu_2011} looks into just this idea. It allows tangible interaction directly on multi touch devices. It embeds objects with circuits that simulate touch input to allow the mobile device to detect the object. There are three methods that "TUIC" achieves this, spatial (static touch patterns), frequency (dynamic modulation of touch), and hybrid (a combination of spatial and frequency). Although this would not directly apply to Piknik since it is possible to detect magnets without touching the phone, we will be looking into how we can detect different sizes of magnets and various locations. This can be done using the magnet's intensity or frequency modulation (as described in TUIC). Frequency modulation could be used in a unique way to build custom hardware to modulate the magnetic field. However due to the timeframe of this project, this will be considered as future work.
"Tangible Meets Gestural" \cite{Mazalek} mentions the value of learning and thinking is greater when using physical objects. Touching physical objects can help children learn how to count and keep track of their activities. We will use the findings of this paper help us create physical objects to allow users to draw with (e.g. styluses or paint brushes).
Magnet Based Interactions
Several works have investigated input methods making use of magnetometers, whether they are looking for more absolute locators mimicking a mouse or more gestural input. uTrack \cite{Chen_2013} implements an absolute locator using a magnet attached to the user's thumb and two magnetometers attached to their ring finger. The combined readings from the two sensors allow a fairly accurate location reading for the magnet. While this does show that it is possible, the implementation also highlights that a single sensor does have its limitations, especially when compromises are made such as in smartphones. Smartphone sensor reliability for augmented reality applications \cite{Blum_2013} touches on the inherit inaccuracy of cheaper sensors used in smartphones. In addition, a lot of attention is paid to how external forces can affect the readings. Improving Heading Accuracy in Smartphone-based PDR Systems using Multi-Pedestrian Sensor Fusion \cite{nicta_7445} investigates this further and attempts to imrove upon the results by fusing readings from multiple devices. In their studies, they were able to reduce the error in the heading readings by some 27% using only naive averaging, leaving room for improvement using more complex algorithms. MagPairing \cite{CNS14} attempts to make use of the unique magnetic forces that a device will pick up in a given location. By tapping the phones, you move them close enough together that the magnetic fields picked up by each device are extremely similar. They are able to reliable pair devices together by encoding these readings with authentication keys and comparing similarly time stamped readings to verify the connection.
Although, these do highlight how difficult and complex it can be to make these sensors accurate enough to be used as absolute locators, there is a lot that can be done by using them in a relative sense, especially as a form of gestural input. Not only can gestures be reliably recorded, but gestures made in 3D Space are also unique to every user. If two users made box like gestures over the phone, both of their gestures would be somewhat different. This is due to users not being able to reproduce each other's gestures in 3D space within a certain threshold \cite{Sahami_Shirazi_2012}. As we are developing this application, we need to take into account this threshold to be able to recognize certain swipe gestures across all users. MagiMusic \cite{Ketabdar_2011} and Magnetic Marionette \cite{Hwang_2013} show that these gesture-based movements are a lot more feasible with the magnetometer than absolute inputs. MagiMusic allows digital instruments to be played by making gestures with a magnet, such as strumming a guitar. Magnetic Marionette introduces a tangible avatar attached to the device, which can be moved around to produce different facial expressions on the screen.
While the sensors in these smartphones are generally inaccurate due to compromises made in the devices manufacturing and the effect of external magnetic forces, they can be made to be more accurate in a fixed setting. If you know the ambient forces you can account for them in order to pick out the desired readings. In Pulse \cite{pulse} for example, a system is designed through which the device can recieve and decode a signal sent out as magnetic forces, and can even reach transfer speeds up to 44 bps. Due to the nature of the sensors used, it is an especially short-range communication method. A Sensor Fusion Method for Smart phone Orientation Estimation \cite{afcb6e17c450492f8c17ad6ac212d48c} also provides methods to work around the inherit issues with these sensors by fusing data from multiple sensors for correction. Although the final implementation here is beyond this project, much of specifics are a good base for filtering some of the sensor data. For example the low-pass filter used to smooth the readings will prove important in Piknik, and adjustments to reference frames are something that very well may come into play.
Drawing with Everyday Objects
In addition to allowing the user to flip through photos and rotate them, Piknik will also offer the capability of drawing on existing photos using magnets. However, instead of just using magnets, we will be embedding the magnets in styluses (acting as paintbrushes) to allow the user to have a more natural interaction with the application. I/O Brush \cite{Ryokai_2004} is similar idea, it is a drawing tool aimed at young children to be able to draw with everyday objects. The authors of I/O brush built a device that can select the color from anywhere in the real world and be able to paint with that color on the tablet. It molded its every day object as a brush so users can easily create the mental mappings and seamlessly work with the every day objects.
BlowBrush \cite{Shen_2014} is another idea that looks into tangible painting systems using a windmill. A user can blow on the windmill to paint leaves or other objects on screen. BlowBrush used this criteria to analyze and study the effectiveness of their application. This criteria is consisted of: metaphorical affordance, enjoyable engagement, tangible manipulation, spatial interaction, embodied facilitation, and expressive representation. As we are developing this application, we need to look into the criteria that BlowBrush used to analyze their own system.