In this research, a Leap Motion sensor was implemented to control a smart notification system. The smart notification system consists out of software that recognized an individual, via Bluetooth connection, and displayed notifications accordingly through a beamer. This innovative system allows different individuals living in the same house to leave personalized messages for one another. The system is controlled by using gestures, recorded by the Leap Motion sensor. This research aimed to improve the efficiency of hand gestures and have the least amount of errors for selecting a notification. To minimize subjectivity, all choices were based on quantitative data.
To test the effectiveness of certain gestures, a prototype was built in Python utilizing the Leap motion. This program was further developed throughout several iterations. The working prototype consisted of a screen on the computer with four different post-its. These could be selected through the use of hand gestures. To select the post-it's the participant had to move their hand in front of the post-it in space. Multiple gestures were chosen based on the most logical and intuitiveness of the movement of your hand to make something that has been selected bigger. Afterward, the participants would fill in a questionnaire about their experience with the gestures.
Throughout the project, we did several user-tests. After every test, we looked at the data we had collected. The first iteration taught us what data we needed. The second taught us several improvements, such as implementing hysteresis. Finally the third iteration we could see the big improvements we had made for both hand gestures.
Overall we believe that designers can play a crucial role when designing complex sensing systems, which are inherently more complex than more straightforward "point and click" interactions. The software engineer can develop the code in such a way that the software works smooth and remains free of errors. The data-analyst can provide the right tools for analyzing the raw and often complex data generated during user tests. The designer can improve the system based on this data, and validate these improvements through user tests and identify new causes of problems. As such it is the designer that has the closest connection to the user and can make decisions based on gathered insights.
User interaction design
Implementing Leap motion data
Analyzing user data
Concept smart notification system
Lab study: optimalization for hand gestures
Technology & Realisation
During our project, we applied the interaction framework and resulted in overview of our concept. Especially with a complex sensor, such as the Leap Motion sensor, it is sometimes confusing to work with, because the sensor itself does not give any feedback or feedforward. It made me, as a designer, think about how the user will interact with the system and the other way around. For my future projects, I will use the 5 A's and the interaction loops (MPCA model) as a checklist. By keeping track of this framework, this will prevent a lot of unnecessary miscommunications between user and system.
Math, Data & Computing
It was decided to build a prototype in python, so we were able to learn a new code language. I was able to create a graphical user interface (GUI) on a screen, which was connected to a leap motion sensor. The screen showed four post-it messages. Upon attaching the sensor to the computer, the user was able to select one of these four messages. The prototype registered data of every gesture performed by the user, even ones that were not close to the originally chosen gestures. Data was collected when the hand was registered until completing the action.
After the first lab study, I found it quite remarkable that lots of users had difficulties interacting with our created system. Later on, we processed and plotted the user data and found new opportunities to improve the user interaction. The collected data suddenly made the interactions more understandable for me as a designer. This made me realize how we could improve interactions based on different data features.