While I do have some, my experience in designing natural user interfaces (NUIs) is certainly limited. Most recently, I designed a natural user interface for creating theatre lighting designs with the Leap Motion Controller (LEAP). Using hand motions above the LEAP, users could select and move lights around the theatre, rotate lights around two axes, and adjust the intensity of lights. A user would simply ‘touch’ the light in virtual space to select it, and could then drag the light in order to change its position. By tracking five degrees of freedom of a user’s pointed finger, the interface allowed the user to rotate a given light to any orientation by pointing their finger in the direction they wished to point the light. Finally, pinching gestures translated to the adjustment of other non-spatial parameters of the light, such as intensity and color.
In Brave NUI World, Wigdor and Wixon explain:
A NUI is not a natural user interface, but rather an interface that makes your user act and feel like a natural. An easy way of remembering this is to change the way you say ‘natural user interface’—it’s not a natural user interface, but rather a natural user interface.
It turned out that our NUI promoted anything but the user feeling like a natural. For example, in our user study, we found that users quickly tired of using the interface. This occurred as a result of holding one hand above the desk for an extended period of time. Also, rotating lights to extreme angles at time required users to either twist their hands into awkward positions (which may have made them quite sore), or to awkwardly clutch the virtual light several times in order to correctly orient it. While our subjects were often fascinated with the novelty of the LEAP, it was clear that they often found using it (at least with our UI) to be awkward, tiresome, and frustrating.
While some of these issues may have had to do with basic interaction with the LEAP itself, certainly some of these problems were the result of our own poor design choices. For one, when iterating on our design, we often only tested our interactions briefly, rather than using them for extending periods of time (as we later expected users to do). Secondly, while we attempted to do so, we missed the mark when it came to designing for the LEAP. We designed interactions that we thought would be intuitive for a user in order to manipulate virtual objects in mid-air. These interactions required the user to keep their arms unsupported above the table for longer lengths of time than were comfortable (resting one’s elbows on the table interfered wit the LEAP’s line of sight.)
I’ve primarily taken two lessons away from this experience. First, when designing NUIs, it is extremely important to design to the strengths of the involved technologies (input devices, etc.), and to avoid the weaknesses of the same. Had we been more careful about this, I believe that our NUI would have been much more successful. In line with this, the second thing I have learned is of my desire to continue to explore and design new NUIs. My previous experience has shown me both how easy it can be to poorly design a NUI, but also how exciting it would be to feel like a natural when using a UI.