NUIs in Everyday Computing

In a recent post on the Leap Motion blog, Alex Colgan discusses the influences that fictional user interfaces (read ‘user interfaces depicted in movies’) have on the development of motion controls being developed today. He draws on examples from Minority Report, Ender’s Game, and The Avengers to illustrate his three main points. In short, these are:

  • Successful motion controls ‘make us feel powerful and in control of our environment’.
  • Successful motion controls keep the user in a state of flow.
  • Successful motion controls leverage immersion and ‘anti-immersion’ well.

I’d like to focus on the second of those points. In his post, Colgan references Mihaly Csikszentmihalyi’s description of flow (the psychologist who initially proposed the notion):

Human beings seek optimal experiences, where we feel a sense of exhilaration–a deep sense of enjoyment. In these intense moments of concentration, our ego disappears, time stands still. Art. Sport. Gaming. Hacking. Every flow activity shares one thing in common: a sense of discovery, a creative feeling of transporting us into a new reality to higher levels of performance.

Many people who speak of flow (Colgan included) only discuss flow as occurring in creative activities, sports, gaming, and the like. Need this be the case? Is enabling a flow state really only a goal fit for user interfaces built for entertainment and gaming (as Wigdor and Wixon might have us believe?)

Csikszentmihalyi says no. To support this (drawing from his thousands of interviews with not only creatives and athletes, but also CEOs, shepherds, and the like) he describes seven indicators that one is in a flow state:

  1. Completely involved in what we are doing–focused, concentrated.
  2. A sense of ecstasy–of being outside everyday reality.
  3. Greater inner clarity–knowing what needs to be done, and how well we are doing.
  4. Knowing that the activity is doable–that our skills are adequate to the task.
  5. A sense of serenity–no worries about oneself, and a feeling of growing beyond the boundaries of the ego.
  6. Timelessness–thoroughly focused on the present, hours seem to pass by in minutes.
  7. Intrinsic motivation–whatever produces flow becomes its own reward.

While I don’t disagree that gaming and entertainment interfaces should aim to be conducive to flow, I’m convinced that flow has a place outside of the latest Call of Duty release. In my work, being completely involved in what I am doing, having inner clarity, having confidence in my abilities, finding serenity, being excited and motivated to do my work are all certainly desirable and achievable. Furthermore, I would hope that the tools I choose to do my work are conducive to these, as well. While NUIs, on the outside, may seem most appropriate for gaming and entertainment, no one has yet convinced me that these are the only applications where they are appropriate. And, if they are especially capable in enabling flow, we should be considering ways to incorporate them in all manner of UI.

Second First Thoughts on NUIs

While I do have some, my experience in designing natural user interfaces (NUIs) is certainly limited. Most recently, I designed a natural user interface for creating theatre lighting designs with the Leap Motion Controller (LEAP). Using hand motions above the LEAP, users could select and move lights around the theatre, rotate lights around two axes, and adjust the intensity of lights. A user would simply ‘touch’ the light in virtual space to select it, and could then drag the light in order to change its position. By tracking five degrees of freedom of a user’s pointed finger, the interface allowed the user to rotate a given light to any orientation by pointing their finger in the direction they wished to point the light. Finally, pinching gestures translated to the adjustment of other non-spatial parameters of the light, such as intensity and color.

In Brave NUI World, Wigdor and Wixon explain:

A NUI is not a natural user interface, but rather an interface that makes your user act and feel like a natural. An easy way of remembering this is to change the way you say ‘natural user interface’—it’s not a natural user interface, but rather a natural user interface.

It turned out that our NUI promoted anything but the user feeling like a natural. For example, in our user study, we found that users quickly tired of using the interface. This occurred as a result of holding one hand above the desk for an extended period of time. Also, rotating lights to extreme angles at time required users to either twist their hands into awkward positions (which may have made them quite sore), or to awkwardly clutch the virtual light several times in order to correctly orient it. While our subjects were often fascinated with the novelty of the LEAP, it was clear that they often found using it (at least with our UI) to be awkward, tiresome, and frustrating.

While some of these issues may have had to do with basic interaction with the LEAP itself, certainly some of these problems were the result of our own poor design choices. For one, when iterating on our design, we often only tested our interactions briefly, rather than using them for extending periods of time (as we later expected users to do). Secondly, while we attempted to do so, we missed the mark when it came to designing for the LEAP. We designed interactions that we thought would be intuitive for a user in order to manipulate virtual objects in mid-air. These interactions required the user to keep their arms unsupported above the table for longer lengths of time than were comfortable (resting one’s elbows on the table interfered wit the LEAP’s line of sight.)

I’ve primarily taken two lessons away from this experience. First, when designing NUIs, it is extremely important to design to the strengths of the involved technologies (input devices, etc.), and to avoid the weaknesses of the same. Had we been more careful about this, I believe that our NUI would have been much more successful. In line with this, the second thing I have learned is of my desire to continue to explore and design new NUIs. My previous experience has shown me both how easy it can be to poorly design a NUI, but also how exciting it would be to feel like a natural when using a UI.

Playability Heuristics

Next up in the reading stack is Playability Heuristics for Mobile Games.

Stemming from the literature on usability heuristics, the authors (Korhonen and Koivisto) develop a set of playability heuristics for mobile games. In the paper, they present their motivations for developing these heuristics, the heuristics themselves, and the ‘results’ of their ‘validation’ of these characteristics.

Their heuristics are grouped into three categories: gameplay, mobility, and game usability. Their initial list of heuristics was made up of the following:

  • Don’t waste the player’s time
  • Prepare for interruptions
  • Take other persons into account
  • Follow standard conventions
  • Provide gameplay help
  • Differentiation between device UI and the game UI should be evident
  • Use terms that are familiar to the player
  • Status of the characters and the game should be clearly visible
  • The player should have clear goals
  • Support a wide range of players and playing styles
  • Don’t encourage repetitive and boring tasks

In order to validate these heuristics, six evaluators applied them to a selected application and noted all playability problems that were both covered and not covered by the list of heuristics. The evaluators found 61 playability problems, but 16 of these were not adequately described by one of their heuristics. Thus, the authors expanded their initial set of heuristics into three expanded sublists (one for each ‘category’):

  • Game Usability
    • Audio-visual representation supports the game
    • Screen layout is efficient and visually pleasing
    • Device UI and game UI are used for their own purposes
    • Indicators are visible
    • The player understands the terminology
    • Navigation is consistent, logical, and minimalist
    • Control keys are consistent and follow standard conventions
    • Game controls are convenient and flexible
    • The game gives feedback on the player’s actions
    • The player cannot make irreversible errors
    • The player does not have to memorize things unnecessarily
    • The game contains help
  • Mobility
    • The game and play sessions can be started quickly
    • The game accommodates with the surroundings
    • Interruptions are handled reasonably
  • Gameplay
    • The game provides clear goals or supports player- created goals
    • The player sees the progress in the game and can compare the results
    • The players are rewarded and rewards are meaningful
    • The player is in control
    • Challenge, strategy, and pace are in balance
    • The first-time experience is encouraging
    • The game story supports the gameplay and is meaningful
    • There are no repetitive or boring tasks
    • The players can express themselves
    • The game supports different playing styles
    • The game does not stagnate
    • The game is consistent
    • The game uses orthogonal unit differentiation4
    • The player does not lose any hard-won possessions

This expanded set of heuristics was validated using the same process, only now with five different games. Based on this process, the authors draw the following conclusions:

  • Usability problems were both the easiest to identify with their heuristics, as well as the easiest violations to make.
  • More mobility problems were found than expected.
  • Gameplay is the most difficult aspect of playability to evaluate.

Yowza–talk about a scattered paper. I mean, this bad boy is all over the place. It seems as though the authors’ thoughts simply haven’t gelled well at all. Nevertheless, they do present what seem to be reasonable heuristics for the evaluation of playability. I have two primary problems with this paper. First, an the world of smartphones and mobile games has changed dramatically in the last decade. I would imagine an more recent look at playability is both available and more useful. Second, while their heuristics seem reasonable, and they claim to have validated these heuristics, I can’t find any evidence of this. Do Korhonen and Koivisto not understand that just using a set of heuristics doesn’t imply that they are valid? This leads to the bigger question of what it means for a set of heuristics to be valid. Do valid heuristics completely describe all possible problems? Is the ‘most’ valid set of heuristics that which completely describes all possible problems with the fewest heuristics? I’m not sure. I am sure, however, that writing a list of heuristics and then applying them absolutely does not make them valid. The analysis necessary to do so just isn’t present in this paper. Even if the authors claim to have begun to validate their framework of heuristics, they certainly haven’t presented any such results in this paper. While the work shows (showed) promise, I find this both misleading and frustrating.

Usability Heuristics Usability

My reading this week included Nielsen’s Enhancing the Explanatory Power of Usability Heuristics. As usual, I’ll get my trivial beef out of the way up front.

First, the paper is downright painful to read. The English-as-a-second-language rule buys back a few points for Nielsen here, but seriously?:

Note that it would be insufficient to hand different groups of usability specialists different lists of heuristics and let them have a go at a sample interface: it would be impossible for the evaluators to wipe their minds of the additional usability knowledge they hopefully had, so each evaluator would in reality apply certain heuristics from the sets he or she was supposed not to use.

Sure, I’m nitpicking, but that sentence makes my inner ear bleed.

Before going any further, some orientation with respect to the aim of the paper is in order. Surrounding the multiple self-citations Nielsen makes right out of the gate (before the third word of the paper), he defines heuristic evaluation as

a ‘discount usability engineering’ method for evaluation user interfaces to find their usability problems. Basically, a set of evaluators inspects the interface with respect to a small set of fairly broad usability principles, which are referred to as ‘heuristics.’

(I’ll forego my opinion that usability should be concerned with issues beyond just those in the interface itself…) A number of batteries of these usability heuristics have been developed by different authors, and in this paper Nielsen’s aim is to synthesize ‘a new set of usability heuristics that is as good as possible at explaining the usability problems that occur in real systems.’ In short, Nielsen compiles a master list of 101 heuristics from seven lists found in the literature. Armed with this master list, he examines 249 usability problems across different stages of development and types of interfaces. Each of the heuristics was given a grade for how well it explained each of the 249 problems. A principal components analysis (PCA) of these grades revealed that no heuristics account for a large portion of variability in the problems he examined.

After his PCA Nielsen groups individual heuristics into larger factors–essentially heuristic categories. In his opinion, seven of these categories warrant presentation given here in decreasing order of PCA loadings as calculated by Nielsen:

  • Visibility of system status
  • Match between system and real world
  • User control and freedom
  • Consistency and standards
  • Error prevention
  • Recognition rather than recall
  • Flexibility and efficiency of use

His presentation of these factors and their component heuristics is troubling and confusing. First, the highest PCA loading of any of these factors is 6.1%. Not only is this an exceedingly small amount of explanatory power, it represents the aggregated contribution of 12 individual heuristics! Furthermore, the individual heuristic loadings themselves seem to be at odds. As an example, the heuristic speak the user’s language taken from one source in the literature and another speak the user’s language taken from another source give respective loadings of 0.78 and 0.67 Why do two identically phrased heuristics have different loadings? Furthermore, why are two identically phrased heuristics even present in the master list at all? This should, at the very least, be addressed by the author. Without some sort of explanation, I am wary of taking Nielsen’s PCA results seriously. Nielsen sweeps this under the rug, stating that ‘it was not possible to account for a reasonably large part of the variability in the usability problems with a small, manageable set of usability factors.’ (That, or some data preprocessing or an upgraded PCA gizmo was in order…)

Nielsen states that 53 factors are needed to account for 90% of the variance in the usability problems in the dataset. I’m lost. The factors for which Nielsen did show the component heuristics had an average of 10 heuristics each. With only 101 total heuristics, how does one arrive at 53 factors (in addition to the others that account for the remaining 10% of variability)? Is Nielsen shuffling heuristics around into different factors to try and force something to work? To make matters worse, Nielsen states that ‘we have seen that perfection is impossible with a reasonably small set of heuristics’. No, you’re missing the point, Nielsen. Perfection is impossible even with a very large set of heuristics. At this point, I’m beginning to lose faith that this paper is going anywhere meaningful…

So, since perfection is impossible, Nielsen pivots to using a new lens for the data. Now, it’s a head-to-head match of the individual lists of heuristics gathered by Nielsen. Here, he ‘consider[s] a usability problem to be “explained” by a set of heuristics if it has achieved an explanation score of at least 3 (“explains a major part of the problem, but there are some aspects of the problem that are not explained”) from at least one of the heuristics in the set.’ Strange, I guess we are now ignoring Nielsen’s previous statement that ‘the relative merits of the various lists can only be determined by a shoot-out type comparative test, which is beyond the scope of the present study’… Nevertheless, based on this approach, Nielsen gives the ten heuristics that explain all usability problems in the dataset and the ten that explain the serious usability problems in the dataset. With this new analysis in hand, and after jumping through several hoops (I’m not entirely clear on how Nielsen’s data were rearranged to make this new analysis work), Nielsen concludes that ‘it would seem that [these lists of ten heuristics indicate] the potential for the seven usability factors to form the backbone of an improved set of heuristics.’ Going on, Nielsen then states that two important factors are missing: error handling and aesthetic integrity…so we’ll add those to the list, too. In other words, even though my data don’t bear this out, I’m adding them because they’re important to me, dammit.

I’m utterly confused. How is it that one can take real data, slice and dice the analysis several ways, never really get the data to shape up and prove your point, and then act like it does? Add to this the necessary hubris to come out and say, ‘Hey, even without the data to prove it, I’m stating that these are equally important factors’, and I’m left wholly unimpressed with this paper.

Pay Attention!

Physically and virtually embodied agents offer great potential due to their capacity to afford interaction using the full range of human communicative behavior. To know when to best utilize these behaviors, these agents must be able to perceive subtle shifts in users’ emotional and mental states.

Contributing to the development of agents with such capabilities, Dan Szafir and Bilge Mutlu presented their work in implementing a robotic agent capable of sensing and responding to decreasing engagement in humans in their recent paper, Pay Attention! Designing Adaptive Agents that Monitor and Improve User Engagement.

In any learning situation, teachers communicate most effectively to learners when they sense that learners are beginning to lose attention and focus, and re-engage their attention through verbal and non-verbal immediacy cues. These cues–for example, changes in speech patterns, gaze, and gestures–create a greater sense of immediacy in the relationship between the teacher and learner, drawing the focus of the learner back to the topic at hand. Szafir and Mutlu posit that equipping robotic agents with the ability to monitor learners’ brain wave patterns through electroencephalography (EEG), recognize declining attention, and respond with such immediacy cues, the efficacy of learning can be improved. They also argue that such agents will promote a stronger sense of rapport between learner and agent, as well as a greater motivation to learn in learners.