NUIs in Everyday Computing

In a recent post on the Leap Motion blog, Alex Colgan discusses the influences that fictional user interfaces (read ‘user interfaces depicted in movies’) have on the development of motion controls being developed today. He draws on examples from Minority Report, Ender’s Game, and The Avengers to illustrate his three main points. In short, these are:

  • Successful motion controls ‘make us feel powerful and in control of our environment’.
  • Successful motion controls keep the user in a state of flow.
  • Successful motion controls leverage immersion and ‘anti-immersion’ well.

I’d like to focus on the second of those points. In his post, Colgan references Mihaly Csikszentmihalyi’s description of flow (the psychologist who initially proposed the notion):

Human beings seek optimal experiences, where we feel a sense of exhilaration–a deep sense of enjoyment. In these intense moments of concentration, our ego disappears, time stands still. Art. Sport. Gaming. Hacking. Every flow activity shares one thing in common: a sense of discovery, a creative feeling of transporting us into a new reality to higher levels of performance.

Many people who speak of flow (Colgan included) only discuss flow as occurring in creative activities, sports, gaming, and the like. Need this be the case? Is enabling a flow state really only a goal fit for user interfaces built for entertainment and gaming (as Wigdor and Wixon might have us believe?)

Csikszentmihalyi says no. To support this (drawing from his thousands of interviews with not only creatives and athletes, but also CEOs, shepherds, and the like) he describes seven indicators that one is in a flow state:

  1. Completely involved in what we are doing–focused, concentrated.
  2. A sense of ecstasy–of being outside everyday reality.
  3. Greater inner clarity–knowing what needs to be done, and how well we are doing.
  4. Knowing that the activity is doable–that our skills are adequate to the task.
  5. A sense of serenity–no worries about oneself, and a feeling of growing beyond the boundaries of the ego.
  6. Timelessness–thoroughly focused on the present, hours seem to pass by in minutes.
  7. Intrinsic motivation–whatever produces flow becomes its own reward.

While I don’t disagree that gaming and entertainment interfaces should aim to be conducive to flow, I’m convinced that flow has a place outside of the latest Call of Duty release. In my work, being completely involved in what I am doing, having inner clarity, having confidence in my abilities, finding serenity, being excited and motivated to do my work are all certainly desirable and achievable. Furthermore, I would hope that the tools I choose to do my work are conducive to these, as well. While NUIs, on the outside, may seem most appropriate for gaming and entertainment, no one has yet convinced me that these are the only applications where they are appropriate. And, if they are especially capable in enabling flow, we should be considering ways to incorporate them in all manner of UI.

Second First Thoughts on NUIs

While I do have some, my experience in designing natural user interfaces (NUIs) is certainly limited. Most recently, I designed a natural user interface for creating theatre lighting designs with the Leap Motion Controller (LEAP). Using hand motions above the LEAP, users could select and move lights around the theatre, rotate lights around two axes, and adjust the intensity of lights. A user would simply ‘touch’ the light in virtual space to select it, and could then drag the light in order to change its position. By tracking five degrees of freedom of a user’s pointed finger, the interface allowed the user to rotate a given light to any orientation by pointing their finger in the direction they wished to point the light. Finally, pinching gestures translated to the adjustment of other non-spatial parameters of the light, such as intensity and color.

In Brave NUI World, Wigdor and Wixon explain:

A NUI is not a natural user interface, but rather an interface that makes your user act and feel like a natural. An easy way of remembering this is to change the way you say ‘natural user interface’—it’s not a natural user interface, but rather a natural user interface.

It turned out that our NUI promoted anything but the user feeling like a natural. For example, in our user study, we found that users quickly tired of using the interface. This occurred as a result of holding one hand above the desk for an extended period of time. Also, rotating lights to extreme angles at time required users to either twist their hands into awkward positions (which may have made them quite sore), or to awkwardly clutch the virtual light several times in order to correctly orient it. While our subjects were often fascinated with the novelty of the LEAP, it was clear that they often found using it (at least with our UI) to be awkward, tiresome, and frustrating.

While some of these issues may have had to do with basic interaction with the LEAP itself, certainly some of these problems were the result of our own poor design choices. For one, when iterating on our design, we often only tested our interactions briefly, rather than using them for extending periods of time (as we later expected users to do). Secondly, while we attempted to do so, we missed the mark when it came to designing for the LEAP. We designed interactions that we thought would be intuitive for a user in order to manipulate virtual objects in mid-air. These interactions required the user to keep their arms unsupported above the table for longer lengths of time than were comfortable (resting one’s elbows on the table interfered wit the LEAP’s line of sight.)

I’ve primarily taken two lessons away from this experience. First, when designing NUIs, it is extremely important to design to the strengths of the involved technologies (input devices, etc.), and to avoid the weaknesses of the same. Had we been more careful about this, I believe that our NUI would have been much more successful. In line with this, the second thing I have learned is of my desire to continue to explore and design new NUIs. My previous experience has shown me both how easy it can be to poorly design a NUI, but also how exciting it would be to feel like a natural when using a UI.

How CS6724 has changed my life…

A hyperbolic statement? Possibly. Really though, I have learned a great deal from my time in this course (Applied Theories in Human-Computer Interaction). When I first chose to register for the course, I was in a situation where I wasn’t entirely sure how to situate my own research under the umbrella of human-computer interaction. My research spans a number of disciplinary boundaries–primarily computer science, music, and psychology. As I’ve said elsewhere, I currently deal with the interactions of music and human emotion, using affective computing as a tool for exploring these interactions. Certainly then, I believe that this research does have a place in human-computer interaction, but I was at a loss when it came to describing those theories, models, or frameworks that have been developed by other HCI researchers that might come to bear on my own work. In all truthfulness, I can’t say I’m in a very different position today than I was at the beginning of this course. What I can say, however, is that I’m in a much better position to begin to look for the answers to these questions now, both due to my own time presenting for this class, and also due to my experiences participating in the presentations that others have given. From each side of the lectern, I’ve read and discussed scholarship that has all but slapped me across the face and screamed, “Don’t let your own work end up looking like this!”, as well as scholarship that has very clearly demonstrated the right way to go about my work.

Bad research. I came into the course incredibly biased against what I thought (and often still think) is the status quo for rigor in HCI research. I’ll admit that there are plenty of exceptions to the rule, but it seemed that after reading any arbitrary piece of HCI literature I was more often than not left wondering what reviewer in their right mind would recommend this paper/article for acceptance. Part of my reason for taking this course–one that I articulated in one of the first course meetings–was that I hoped that this course would provide me with the opportunity to have this assumption to be proven wrong. To be honest, there have been a number of things we’ve read over the course of the semester that aren’t worth the paper on which they’re printed. Reading these, combined with discussions with others in the course and the gripes of other well-respected scholars, I’m now convinced that our field does have a problem with ‘research’ that plays fast-and-loose with academic rigor.

Good research. On the other hand, I have been heartened both by calls for more disciplined approaches to research, as well as exemplars of the same. For example, Ackerman’s argument for creating a new science of the artificial from CSCW (and HCI as a whole), coupled with a call for carefully planned and executed, fundamentally sound inquiry brings me the hope that there are researchers in our field who do give a damn about the respectability of their work. At the same time, several pieces of literature that we’ve read serve as wonderful examples of such clearly thought-through and well-executed examples of such research. What does this mean for me, as a young(ish) scholar, in the end? It means that I believe that I have an issue with the level at which we’ve set the bar for acceptable research in our field. It means that I see the fingerprints of the allure of quick and easy work in my own research. And finally, it means that in recognizing my own shortcomings and these larger problems in our field I am responsible for bringing what I can to the table in my own work to be a part of a change for the better in our collective work as a group of academics. More to the point (and at the risk of sounding esoteric), I see the problem both in our field as a whole and in my own work, and its up to me to make a difference where I can by letting my own work serve as an example of what quality scholarship should look like. This is how CS6724 has changed my life.

Houston, we have a problem.

Here’s the punch line from Mark Ackerman’s The Intellectual Challenge of CSCW: The Gap Between Social Requirements and Technical Feasibility:

If CSCW (or HCI) merely contributes “cool toys” to the world, it will have failed its intellectual mission. Our understanding of the gap is driven by technological exploration through artifact creation and deployment, but HCI and CSCW systems need to have at their core a fundamental understanding of how people really work and live in groups, organizations, communities, and other forms of collective life. Otherwise, we will produce unusable systems, badly mechanizing and distorting collaboration and other social activity.

This “social-technical” gap is the space between how human behavior and activity actually work and our ability to understand, model/represent, and design for human behavior and activity in human-computer interactions. And, coming to grips with this gap presents, for Ackerman, the primary challenge for computer-supported cooperative work as a field.

Ackerman borrows Simon’s idea of sciences of the artificial to build a case for an approach toward better studying, understanding, and addressing the social-technical gap in CSCW. Simon differentiates between the artificial (those things that exist as the products of “human design and agency”), and the natural (those things that exist apart from human intervention). For Simon, the existing sciences focused on understanding the natural, and engineering focused on synthesizing the artificial. Between these two, Simon proposed a space for new sciences–those that seek to understanding the nature of design and engineering. Ackerman places CSCW squarely in the realm of these new sciences:

CSCW is at once an engineering discipline attempting to construct suitable systems for groups, organizations, and other collectivities, and at the same time, CSCW is a social science attempting to understand the basis for that construction in the social world (or everyday experience).
CSCW’s science, however, must centralize the necessary gap between what we would prefer to construct and what we can construct. To do this as a practical program of action requires several steps-—palliatives to ameliorate the current social conditions, first-order approximations to explore the design space, and fundamental lines of inquiry to create the science.

I’m most interested by Ackerman’s call for fundamental lines of inquiry to create this new science of the artificial, primarily because I believe this approach to CSCW holds implications not only for CSCW, but for the broader field of human-computer interaction. The lack of focus we tend to have in HCI (exhibited by the never-ending stream of “cool toys” presented at conference after conference) desperately needs to be addressed, and the identification of and careful examination through fundamental lines of inquiry could go long way in bringing this focus.

I’m just as guilty of this lack of focus as anyone else. I’ve got what I think are “cool” ideas, and I’ve built up my own research around what I’m afraid are thrown together, not-so-fundamental lines of questioning. It’s difficult for me to backtrack, as I know it would be for anyone else. However, in order to genuinely contribute to the progress of HCI as a field, I must take the time establish my work in such a way that it is both prompted by that work that has gone before, and is at least situated to inform that which may follow. If I and others don’t, then fourteen years after Ackerman wrote his article, we’re still failing our mission.

Picture this…

Does visualization help to more effectively communicate concepts in computer science education to learners? The answer seems, to me at least, to very clearly be yes. It seems as though the jury was out in 2003 when Naps et al. published Exploring the Role of Visualization and Engagement in Computer Science Education.

In this paper, Naps et al. summarize the results of the Working Group on Improving the Educational Impact of Algorithm Visualization. Based on a survey they conducted, the group points out two key reasons that the use of visualization in computer science education may have not yet gained widespread acceptance. These were:

  • “From the learner’s perspective, …visualization technology may not be educationally useful.”
  • “From the instructor’s perspective, …visualization technology may simply incur too much overhead to make it worthwhile.”

They go on to conclude that “learners who are actively engaged with…visualization technology have consistently outperformed learners who passively view visualizations.” And, in fact, that:

Visualization technology, no matter how well it is designed, is of little educational value unless it engages learners in an active learning activity.

I’m fairly confident in stating that this holds true for any teaching materials, not simply visualization technology. Nevertheless, the authors maintain that visualization is more or less worthless unless it actively engages the learner. To be clear, they refer not only to carefully constructed, animated, interactive visualization tools. They also include diagrams–even those found in textbooks–under the heading of “visualization technology”. To this end, the authors provide an exhaustive accounting (or rather, an extraordinarily verbose data dump) of the results of their survey. Based on these results, they both provide a set of best practices for visualization design, as well as a framework for further research around the effectiveness of visualization technology. This framework aims to explore the effectiveness of visualization along the lines drawn by Bloom’s taxonomy.

On the one hand, I’m all for some real science in computer science. Far be it from me to groan when an HCI researcher takes the bold step of demonstrating and calling for additional academic rigor in research. For this, thank you, Naps. On the other hand, I’m stumped by their statements that motivate this call. In their introduction, the authors state that “intuition suggests that…graphical representations would help one” in understanding computer science concepts. Yes, intuition does indeed suggest this. Not only that, the piles of scholarship cited here and elsewhere bear this out–graphical representations do aid in learning. In other words, our intuition doesn’t just suggest this–this is reality. Very little the citations in this paper draw on general education literature. Rather, most of the cited literature deals directly with the use of visualizations in teaching algorithms and data structures. This seems like a non-negligible oversight to me.

In the end, I’m happy to see these sorts of endeavors in our community, dated as they may be. However, their foundational argument doesn’t have me sold. Certainly, the quality of visualization technologies runs a wide gamut. Please don’t exclude diagrams from your articles and books just because Naps told you they’re worthless if they don’t actively engage me–I, for one, appreciate them and find them extremely useful…