NUIs in Everyday Computing

In a recent post on the Leap Motion blog, Alex Colgan discusses the influences that fictional user interfaces (read ‘user interfaces depicted in movies’) have on the development of motion controls being developed today. He draws on examples from Minority Report, Ender’s Game, and The Avengers to illustrate his three main points. In short, these are:

  • Successful motion controls ‘make us feel powerful and in control of our environment’.
  • Successful motion controls keep the user in a state of flow.
  • Successful motion controls leverage immersion and ‘anti-immersion’ well.

I’d like to focus on the second of those points. In his post, Colgan references Mihaly Csikszentmihalyi’s description of flow (the psychologist who initially proposed the notion):

Human beings seek optimal experiences, where we feel a sense of exhilaration–a deep sense of enjoyment. In these intense moments of concentration, our ego disappears, time stands still. Art. Sport. Gaming. Hacking. Every flow activity shares one thing in common: a sense of discovery, a creative feeling of transporting us into a new reality to higher levels of performance.

Many people who speak of flow (Colgan included) only discuss flow as occurring in creative activities, sports, gaming, and the like. Need this be the case? Is enabling a flow state really only a goal fit for user interfaces built for entertainment and gaming (as Wigdor and Wixon might have us believe?)

Csikszentmihalyi says no. To support this (drawing from his thousands of interviews with not only creatives and athletes, but also CEOs, shepherds, and the like) he describes seven indicators that one is in a flow state:

  1. Completely involved in what we are doing–focused, concentrated.
  2. A sense of ecstasy–of being outside everyday reality.
  3. Greater inner clarity–knowing what needs to be done, and how well we are doing.
  4. Knowing that the activity is doable–that our skills are adequate to the task.
  5. A sense of serenity–no worries about oneself, and a feeling of growing beyond the boundaries of the ego.
  6. Timelessness–thoroughly focused on the present, hours seem to pass by in minutes.
  7. Intrinsic motivation–whatever produces flow becomes its own reward.

While I don’t disagree that gaming and entertainment interfaces should aim to be conducive to flow, I’m convinced that flow has a place outside of the latest Call of Duty release. In my work, being completely involved in what I am doing, having inner clarity, having confidence in my abilities, finding serenity, being excited and motivated to do my work are all certainly desirable and achievable. Furthermore, I would hope that the tools I choose to do my work are conducive to these, as well. While NUIs, on the outside, may seem most appropriate for gaming and entertainment, no one has yet convinced me that these are the only applications where they are appropriate. And, if they are especially capable in enabling flow, we should be considering ways to incorporate them in all manner of UI.

Second First Thoughts on NUIs

While I do have some, my experience in designing natural user interfaces (NUIs) is certainly limited. Most recently, I designed a natural user interface for creating theatre lighting designs with the Leap Motion Controller (LEAP). Using hand motions above the LEAP, users could select and move lights around the theatre, rotate lights around two axes, and adjust the intensity of lights. A user would simply ‘touch’ the light in virtual space to select it, and could then drag the light in order to change its position. By tracking five degrees of freedom of a user’s pointed finger, the interface allowed the user to rotate a given light to any orientation by pointing their finger in the direction they wished to point the light. Finally, pinching gestures translated to the adjustment of other non-spatial parameters of the light, such as intensity and color.

In Brave NUI World, Wigdor and Wixon explain:

A NUI is not a natural user interface, but rather an interface that makes your user act and feel like a natural. An easy way of remembering this is to change the way you say ‘natural user interface’—it’s not a natural user interface, but rather a natural user interface.

It turned out that our NUI promoted anything but the user feeling like a natural. For example, in our user study, we found that users quickly tired of using the interface. This occurred as a result of holding one hand above the desk for an extended period of time. Also, rotating lights to extreme angles at time required users to either twist their hands into awkward positions (which may have made them quite sore), or to awkwardly clutch the virtual light several times in order to correctly orient it. While our subjects were often fascinated with the novelty of the LEAP, it was clear that they often found using it (at least with our UI) to be awkward, tiresome, and frustrating.

While some of these issues may have had to do with basic interaction with the LEAP itself, certainly some of these problems were the result of our own poor design choices. For one, when iterating on our design, we often only tested our interactions briefly, rather than using them for extending periods of time (as we later expected users to do). Secondly, while we attempted to do so, we missed the mark when it came to designing for the LEAP. We designed interactions that we thought would be intuitive for a user in order to manipulate virtual objects in mid-air. These interactions required the user to keep their arms unsupported above the table for longer lengths of time than were comfortable (resting one’s elbows on the table interfered wit the LEAP’s line of sight.)

I’ve primarily taken two lessons away from this experience. First, when designing NUIs, it is extremely important to design to the strengths of the involved technologies (input devices, etc.), and to avoid the weaknesses of the same. Had we been more careful about this, I believe that our NUI would have been much more successful. In line with this, the second thing I have learned is of my desire to continue to explore and design new NUIs. My previous experience has shown me both how easy it can be to poorly design a NUI, but also how exciting it would be to feel like a natural when using a UI.