Distributed and Embodied Cognition

From CS260Wiki
Jump to: navigation, search

Lecture Slides

File:Cs260-slides-13-cognition.pdf

Extra Materials

Discussant's Slides and Materials

Reading Responses

Kurtis Heimerl - 10/11/2010 19:24:36

On Distinguishing Pragmatic from Epistemic Action This paper argues that there are (at least) two types of actions in completing a task; pragmatic (actually progressing towards the result) and epistemic (modifying state to better judge the appropriate pragmatic action).

I read two pages in and was convinced. The remaining 25 pages were arguments against (I'd wager) some ancient views from cognitive scientists. The chess example given was perfect, I fail to see how you can argue that users occasionally make actions with the intent of garnering more information.

So, convinced, I ask what the value of this work is. It actually seems quite good, you get an understanding of why ubicomp and pervasive computing targets are so highly researched. One might be able to provide graphical cues in reality, and allow for more informed decisions than currently possible.

How Bodies Matter This paper attempts to scaffold the study of human bodies and cognition in the context of interaction design.

80 references! That's one way to head off the classic initial complaint for a paper like this. It's generally hard to argue against, but at there are spots. For instance, pro gamers overwhelmingly use keyboard and mouse instead of more traditional controllers. The skills requires are pin-point and reflexive.. that seems to strongly point that there are solid reasons for the use of these technologies in this context. It seemed like there wasn't enough thought put into these sorts of counter-points, and the general fact that humans are adaptable. I'm sure that others will have counter-points for specific items they know a lot about. Should be an interesting class discussion.

I did find the risk discussion enlightening though. It's quite true that there is less risk with digital items, and sharing risk is the best way to create trust. At the same time, it does seem totally arbitrary to increase risk to increase trust. I'm not sure how to address that.


Charlie Hsu - 10/12/2010 0:17:27

On Distinguishing Epistemic from Pragmatic Action

This paper describes the differences between epistemic and pragmatic actions, using Tetris as the key example. Pragmatic actions are those that directly manipulate the environment, bringing an agent physically closer to his goal. Epistemic actions also manipulate the environment, but may or may not physically bring the agent closer to his goal. Instead, epistemic actions bring a positive effect on the agent himself, by revealing new information about the environment or reducing the mental computation needed by the agent to solve his problem. The paper studies rotation of Tetris blocks as epistemic actions in the context of identifying the zoid, reducing mental rotation computation, and simplifying the process of matching zoid to contour. The paper also looks at translation of Tetris blocks as an epistemic action in determining the relative horizontal location of a zoid.

This paper reminded me of one of Jakob Nielsen's usability heuristics for interface design: visibility of system status. Offering the agent important information about the system can greatly improve user efficiency. Keeping information in the world for simple cognitive recall is a generally accepted good practice for user interface design. Of course, with games, some of the challenge and excitement comes from the user working to gain this information, and the epistemic actions in Tetris were clearly justified as so by the paper.

The paper argues that epistemic actions are pervasive in everyday tasks, I couldn't agree more strongly. Examples that immediately came to mind were graphic design and video editing: graphic designers often receive rich feedback from mock designs and intermediate, experimental steps in design that reveal new information. In my experience with video editing, much of the editing time is spent reviewing work and obtaining information about the current state of the project. If epistemic actions are so important to users performing tasks, how can we better design interfaces to facilitate epistemic actions?

Since information acquisition is a central goal of many epistemic actions, one might consider simply expanding the amount of immediately available to the user. However, this might result in increased mental computation if the amount of information is too large to be easily digested and processed. On a high level, UI design needs to take into account the importance of balancing the increase of available information and the decrease of mental computation. Users should be afforded intuitive and easy-to-use methods of obtaining extra information on demand, much like the rotate and translate methods in Tetris.


How Bodies Matter

This paper explored five themes for interaction design revolving around the use of our physical bodies: thinking through doing, performance, visibility, risk, and thick practice. The paper aims to offer insight into richer interaction paradigms by showing the importance of physical bodies in designing systems that integrate the physical and computational worlds.

I found it interesting that a research paper could simply explore the richness of the human body in its interaction with the environment and open up tons of different avenues for human-computer interaction to further explore. Many of these observations sparked new questions and possible ideas of my own: how can we lessen constraints on interaction to encourage creativity and mobility? How can we integrate electronic enhancements into the physical domain, much like the DigitalDesk (thick practice)? How can we integrate physical, tactile knowledge gathered with our bodies into a digitally enhanced world? How can we expand the traditional GUI's mental model of the user? How can we retain risk in digital communications and force responsibility?

These questions did not require much thought to produce after reading the paper; I personally wished that we had read this paper before proposing a research idea for the class! However, I was also able to reflect on some of the previous papers we've read and connect them to the ideas proposed in this paper. The DigitalDesk was an example of thick practice, working with the traditional office environment and attempting to enhance rather than replace. Many of the crowdsourcing improvements we have discussed in class involve injecting risk into user contribution (Stack Overflow's reputation system, profiles and pre-screening for Mechanical Turk). Visibility and cooperative work has been discussed in our exploration of CSCW, and rich input devices touched on the theme of performance from the paper.


Airi Lampinen - 10/12/2010 15:06:49

In "How Bodies Matter", Klemmer, Hartmann & Takayama present a synthesis of theoretical and empirical work from various disciplines that increases current understandings of both ideation and evaluation of interaction design that integrates the physical and computational worlds. The paper addresses a topic that has received a lot of attention lately: what is the role of our bodies in a more and more digital environment? As the authors state, the desktop computing paradigm has brought about a remarkable transformation by homogenizing the physical performance of work.

The authors question this development by discussing the concepts of thinking through doing, performance, visibility, risk, and thick practice. The first questions the age old (and badly justified) dichotomy between minds and bodies that seemed to underlie also the second reading of the week. The paper talks for a strong integration between thinking and doing and how these to co-produce learning and reasoning. Furthermore, the potential of richer interactions is brought into the discussion - our bodies are capable of much richer actions than keystrokes and leveraging this potential could lead to better results in computing, too.

Next to individual corporeality, social affordances are discussed. More embodied and physical interaction tools and practices may facilitate collaboration as they increase the visibility of people's activities. The authors conclude by claiming that as the reality is hard to simulate digitally, a more embodied path of interaction design might be a better choice. This claim relates strongly to our earlier discussions of augmented vs virtual reality and weiserian ubicomp.

Kirsh and Maglio use a lot of writing in "On Distinguishin Pragmatic from Epistemic Action" to make a fairly simple claim: not all action is strictly pragmatic but people do things also to facilitate thinking and reach new understandings. They are using the classic computer game Tetris as an example and looking into players' practices of rotating and moving ziods around.

To explain our data on the timing and frequency of rotations and translations regularly performed by Tetris players, they propose a new category of action: epistemic actions. Epistemic actions are not performed to get in to a better external position for the game but to advance the player's capabilities of understanding the situation in a useful way. As the authors put it, "epistemic actions are actions designed to change the input to an agent’s information-processing system".

While the authors to some degree attack the model of people as machine-like cognitive processors of information by showing that thinking and action are not separate from one another and that action sometimes is taken to facilitate thinking instead of always thinking first and then acting based on a clearly formulated plan, they go on making a separation between thinking and doing. To me, this was somewhat troubling, as drawing such a line is problematic in a philosophical sense and as the authors at the same time try to undermine the same separation they themselves sustain...


Krishna - 10/12/2010 15:50:49

Epistemic and Pragmatic actions

The primary argument of the paper is that humans perform many actions that do not seem pragmatic (or) necessary given the problems they are trying to solve; these actions, called as epistemic actions, are however extremely useful and necessary as they seem to reduce the cognitive workload - by reducing the space , time complexity of the computation involved and by reducing the unreliability involved with the suggestions coming out of the mental computations.

The authors support their arguments by studying how humans play tetris; their study shows that expert and intermediary game players perform many such epistemic actions to be successful in their game play. The direct consequence of this paper to HCI is that interaction systems should enable users to use their interaction environment in a variety of creative ways. At the least, interaction systems should not restrict users from manipulating their environment.

However, it is difficult to identify what epistemic actions users would do under different contexts; there are other questions as well: will these actions be uniform across users ? how do they generalize ? The authors did a fairly extensive study and such extensive user studies may not be feasible both economically and time wise in most interface design and engineering situations.


Five themes for interaction design

While the physical body plays an important role in human perception and cognition, most existing human computer interfaces do not seem to go beyond 'using the hand' - most interactions are either through the keyboard or the mouse. An important question therefore is whether future interaction systems should make full use of the capabilities provided by enabling bodily interaction. The authors introduce five themes: thinking and doing, performance, visibility, risk and thick practice that would enable designers to cogitate and evaluate ideas on embodied interaction.

In explaining thinking and doing, the authors review the importance of the connections between action and cognition, the point is that humans do a better job when they engage bodily while solving problems - examples: most humans are comfortable solving math problems by writing down the intermediary, and sometimes unconnected, steps than working them out in their head, singers and speakers make seemingly unnecessary gestures but lose comfort levels when their hands are tied, etc. The authors' argument is that Interfaces should facilitate and promote these epistemic actions, or at the least should not constrain users from doing so. In discussing performance, the authors argue that interfaces could be designed for expert performance if users ability to skillfully make use of their body is accounted for. For example, music controllers that facilitate engagement of the entire body, which is a central feature in almost all popular instruments, than just the finger and wrist are presumably more effective - in the sense that users would perceive them as extensions of themselves. Also game controllers and simulators that mimic actual controllers - steering wheel and pedals instead of the arrow keys, increase vehicle control and enhance user engagement.

The authors illustrate the importance of visibility, the extent to which physical activity of connected and unconnected 'other' users can support learning, collaboration and coordination. Examples: listeners prefer a live rock concert than just listening to the music using even the most sophisticated audio systems because there is this experience or sensory feeling of 'collaboration' as a result of seeing the event. The authors suggest that interfaces should make other users' actions visible when it is possible to do so; for systems that depend on synchronous collaboration such as design teams, air traffic control towers, music ensemble, such visibility can distribute the cognitive work load of the individual participants and thus effect better performance.

While discussing risk, the authors mention that though the hallmarks of digital systems are to reduce risk, sometimes it might be useful to preserve the risk and responsibility factors, that physical systems are endowed with, while designing interaction systems. Examples: in computer mediated communication systems as a result of low risk, there is this issue of less thrust and mimicking face-face interactions might actually help here, by increasing the risk levels but increasing thrust at the same time - there are trade offs in terms of openness and emotional ties; another example would be the spell checker, usage of spell checker reduces risks but we also loose spelling practice and hence language skills in the process - it may not be worth to incorporate spell checkers in editing software for children. In thick practice, the authors argue against indiscriminate digital impersonation of physical artifacts. They suggest approaches that directly use the actual physical artifact to the extent possible - this they argue allows user interactions to be more successful by enabling existing practices on the physical artifacts.

A nice read that explains, with many examples and trade-offs, why facilitating bodily interactions may be useful and necessary features to have in interaction systems.


Shaon Barman - 10/12/2010 16:18:29

On Distinguishing Epistemic from Pragmatic Action

The authors discuss how epistemic actions help people solve problems faster by offloading cognitive workload to the environment. As a case study, they analyze how players of different skill levels play tennis, and distinguish between pragmatic and epistemic actions that are taken. They use this study to argue that the models used in computer interaction must include some view of the world and epistemic actions.

Epistemic actions seem quite important when designing an interface. People are limited in the amount of information they can "store" at a certain time and the speed at which they process information. By offloading this information, such as rotating the pieces to calculate where they would fit, the users in this study did better at Tetris. While it seems like all actions the user takes are pragmatic, some actions directly lead to the desired outcomes while others are more indirect. Epistemic actions are those indirect actions. One part that seemed fuzzy was the classical information processing model. While it seems like a plausible model, there doesn't seem to be any information for or against it. It was unclear how exactly to create such a model and the limitations of this type of model. The decision tree is another example of this type of analysis. While it's plausible that this is what happens when people see the piece, there isn't much to support this theory. Overall, it seems important to account for epistemic actions when designing interface, but predicting such actions and measuring the impact seems quite difficult.

How Bodies Matter: Five Themes for Interaction Design

The paper discusses five themes and how they should be considered when designing systems. The themes range from a spread of social science disciplines and move away from the typical design decisions.

I believe the goal for going through these ideas is to explore new fields in HCI by using the work thats already done in other social sciences. The section on thinking through doing focuses on the physical world, and how that effects the user and his interactions. Also, todays interfaces are limited in the use of the human sensory information that is used. It might be possible to get better performance, bandwidth, etc. by bridging this gap. Though in practice, this seems difficult due to a)the cost of hardware and b)the learning curve required to operate new physical interfaces. Most successful interfaces tend to use a metaphor to relate the input to something the user already knows. Visibility and risk seem like they are quite connected, although risk can exist with a single user also. I was not quite sure what was meant by the phrase "thick practice." The example they provided discussed a turntable system which broke down the barrier between LPs and mp3s, and combined the best features of both (and allowed DJs not to carry 30 pounds of LPs wherever they went). The five themes brought up can be used to analyze problems and solutions to find ways to create a more immersive interface and to find problems with existing interfaces.


Matthew Chan - 10/12/2010 16:22:31

===How Bodies Matter: FIve Themes for Interaction Design===

In this paper, the authors explore aspects of "human embodied engagement" and interaction design approaches/evaluations. In some ways, this reminds me of MIT's Tangible Media Group that merges the digital world with the physical world. This paper then presents 5 themes that are valuable for designing/evaluating interactive systems. In a super brief summary, the five themes are (1) Thinking through doing (2) performance (3) Visibility (4) Risk and (5) Thickness of practice.

This paper is valuable because, in my opinion, this is the next frontier. We've mastered the digital world, and now it's time to merge it with the physical world, rather tan keeping the two separate. Moreover, merging the two could unfold extraordinary opportunities and revolutionize the way we live our lives. The results/techniques/methodologies used were observations and analysis of everyday games and tasks such as scrabble or using Montessori blocks physically to change the way how we interact with systems; plus, the brain uses less motor memory and brain power on remember words such as Scrabble where we can manipulate and move letters around.

This paper relates to my work because it could open up new opportunities where i was once limited, such as mobile devices for health care purposes or speech recognition for education use. Perhaps reaching out with hands or other embodiments can even simplify or add richness to the way how users interact with a system. Potential blind spots might be how users with disabilities would use the system. But then we can merely add another design tailored for them.

On Distinguishing Epistemic from Pragmatic Action

In this paper by David Kirsh and Paul Maglio, they explore how Tetris affects certain cognitive and perceptual problems in the real world than by doing it all in the head. In particular, they distinguish pragmatic actions (actions in the physical world) and epistemic actions (actions computed mentally).

This paper is valuable because it complements the previous paper mentioned above and shows who early individuals were exploring the physical aspect with the digital world. Moreover, the techniques/results/methodologies used were mostly based on the game Tetris because it was a fast, repetitive game. The result was that master Tetris players were users who constantly pushed the arrow buttons to rotate the blocks as opposed to newbies who would mentally rotate the blocks in their head mentally.

This paper doesn't directly relate to my work because users for my work don't' need to compute anything mentally nor physically. however, we can explore the realm and see what new opportunies it presents. Of course, from a security standpoint, physical gestures could reveal what a user is doing to an outsider.


Drew Fisher - 10/12/2010 17:02:03

On Distinguishing Epistemic from Pragmatic Action

Kirsh and Maglio detail the behavior of expert Tetris players, noting that they do not follow a strictly divided Observe-Plan-Act agent model. In particular, the players use epistemic action to better understand the falling piece and to reduce mental effort.

This suggests that the Observe-Plan-Act model does not fully represent an intelligent agent, but that planning takes place in conjuction with perception, attention, and reasoning, and that additional effort put toward one of these may significantly lessen the effort required for another, improving the overall effectiveness of the system.

The upshot of this paper is that humans constantly explore their environments to better understand them, and that this technique is important to human performance. Thus, when designing for people, we should make sure to allow the person to explore her environment tactilely if possible.

I liked the fairly thorough logic used to present a set of possible explanations for observed behavior, but it does feel like it could be a false dichotemy. I lack any proposed alternate explanations, though. Some of the diagrams could have been clearer, particularly those indicating contours. Perhaps they would have been more effective in color.


How Bodies Matter: Five Themes for Interaction Design

The human body cannot work without the mind, and the mind does not function separate from the body. This paper describes the importantance of interactions that take into account the entire human body, rather than relegating it to a (comical) finger-ears-eye creature.

This paper's contribution is that it summarizes and clarifies a good deal of other research regarding interaction design, concentrating it into five key fields of focus.

The takeaway is that we should be mindful of how our bodies impact both the way we think and the way we interact with our surroundings, and that our interfaces should be designed with these thoughts in mind.

I hope Bj√∂rn brings in his DJ equipment to class some day and demonstrates the importance of tactile user interfaces. :D


Richard Shin - 10/12/2010 17:33:29

On Distinguishing Epistemic from Pragmatic Action

This paper argues that when performing a task, people do not necessarily take only 'pragmatic actions' to get closer to a goal, but also 'epistemic actions' which change the physical world, in order to aid in the agent's mental computation. This view contradicts the traditional notion that rational agents take only actions that advance them toward the desired goal, and avoid those which do not specifically contribute to advancing toward a particular goal. By developing a model of how people play Tetris that conforms to traditional notions of cognition, and then comparing it to actual observations of Tetris-playing, the authors show that people effect actions onto the game to help them recognize pieces, match contours, and align pieces, even though these actions can detract from the actual goal of optimal piece placement.

While reading the paper, I found the formal model of Tetris cognition the most intriguing; I had never seen an explanation of this kind of how people process information in order to take actions, treating cognition much like computation. Although not directly relevant to HCI, I felt that such models could help predict how people would think and behave when interacting with other computer systems that aren't time-limited or fast-paced. So far, it seemed to me that most of the papers that involve models have largely largely been concerned with the physical interface between the human and the computer, rather than what goes on in people's minds when they use the interfaces, so this paper seemed an interesting new perspective on the topic. I suppose that the primary lesson we can take from the results of this paper is that interfaces should enable users to take epistemic actions, rather than assuming that all actions are pragmatic; supporting 'undo' pervasively throughout an interface, for example, could help users visualize the consequences of taking an action.

Overall, the conclusions drawn in the paper seemed sound given the data that the authors present. Certainly, I found myself agreeing with many of the behaviors that they identified, such as moving pieces in order to match them to the contour down below. Nevertheless, it did seem that the conclusions that they had drawn could have been subjected to greater scrutiny, in order to ensure that the results are not a product of pre-conceived notions on the part of the authors; maybe they could have tested their conclusions separately, after they extracted them from the data that they initially collected.

How Bodies Matter: Five Themes for Interaction Design

This paper discusses the role of the physical body in how humans experience and interact with the world, and suggest themes for interaction designers to follow that take these needs into account. The authors base their argument on the observation that much of human experience is difficult to symbolically articulate, but rather take place through rich actions that our bodies enable. Hence, the paper argues that user interfaces should support 'thinking through doing': allowing use of the body in users' cognition processes. They should leverage the body's richness of physical action, in what the authors term 'performance'. User activity should have 'visibility' in order to enable easier collaboration, and user interfaces should facilitate 'risk', which can engender trust in human relationships, enable focus, and materialize personal responsibility. Finally, the paper argues that user interfaces should support 'thick practice', successfully modeling all of the relevant characteristics of some human activity that the interface enables, and taking existing physical interfaces into account.

Some of the principles espoused in this paper reminded me of previous papers that we have read about ubiquitous computing and direct manipulation interfaces. In a sense, this paper seemed a distillation of some of the directions explored in previous work, explaining what was desirable about these previous techniques—their awareness of our corporeality, in embedding computing in many of the physical objects with which we already interact, or ensuring that user interfaces conform to physical rules that we have come to expect through our bodies. Many of the ideas in this paper seemed intuitively obvious to me after reading them, such as the principle that taking input using more of the body would enable greater fidelity. Nevertheless, this comprehensive investigation into the role of corporeality in interaction design seems valuable as a resource both for interaction designers, and other researchers as a basis for more detailed research that explores some of these themes in depth.

Overall, although the paper was organized along the basis of five themes, I thought that it wasn't substantially much more than mostly a collection of examples and anecdotes, some from previous research. I think the paper overall could have been more convincing, if more quantitative data to validate the arguments presented were available. Also, in some cases, the authors seemed to argue rather strongly for deferring to what is familiar in the physical world when designing interaction. While there is clearly value in designing a system that appeals to what we already know, it also seems difficult to truly make progress in interface design if we remain too tethered to what is already here.


Thejo Kote - 10/12/2010 17:36:10

On distinguishing epistemic from pragmatic action:

In this paper, Kirsh and Maglio introduce the idea of an epistemic action. They define an epistemic action as a physical action that improves cognition by reducing space or time complexity, or by reducing the probability of error. They make their argument by studying Tetris game play.

According to authors, it is important to distinguish epistemic actions from pragmatic actions - those which allow an agent to move towards the completion of an intended action. Their core argument is that human beings rely on external actions to improve their own ability to "compute" an action. An example, is the rotation of "zoids" or patterns in the game of Tetris as a player is trying to find the best fit. Transferring the work of rotation to the external world saves the player the effort that would have been involved in performing the same task mentally.

While the paper makes a strong case for the consideration of epistemic action, an interesting question it brings up is the generality of its application. Is the human mind wired to share workload between epistemic and pragmatic actions in a predictable way?

How bodies matter - Five themes for interaction design:

In this paper Klemmer and Hartmann present five themes which look at how physical actions can be an important aspect of HCI systems. They describe how thought and action work together to co-produce learning in humans. The abilities of the human body and their impact on the performance of tasks are also discussed in the context of the limitations of the current dominant mouse and keyboard interface. The other themes address how physical aspects of our interaction with the world influences our social relationships.

The authors tried to make the case for why this was a useful contribution, but it didn't really matter to me because I'm not really familiar with the related work in the fields of psychology, sociology and philosophy. It was just a very good introduction and argument for why designers of HCI systems need to think beyond just the limited affordances that are popular today. There were some amazing examples like how the introduction of computer systems broke a technician's detection of paper quality through the sensitivity of arm hair to electricity in the atmosphere. Examples like these, I thought, also implicitly made a strong case for the importance of ethnography in the design process.


Luke Segars - 10/12/2010 18:03:37

How Bodies Matter

This paper surveys a list of reasons why digital technology has not been able to totally erode our preferences for physical media. Digital technology offers a number of advantages over physical objects, such as the ability to maintain an object state history, make infinitely many perfect copies of an object, and transmit the object quickly over vast geographic distances. Nevertheless, there are some important situations where digital technology is inferior to its physical counterparts. The first major point of the paper, and perhaps the most impactful, is the tendency that many people have to “think through doing.” Young children do this almost all of the time to learn about the world around them; it remains a critical skill for us to perceive and understand the way that things work as we age as well. Computers are noticeably bad at using this particular human skill; we sit in one place, handcuffed to the table, in order to perform the majority of our digital work. The paper suggests that this can limit our creative potential and certainly limits our ability to ‘do’ things to attain new knowledge. Interestingly, many design studios that emphasize computing have a decent amount of open collaborative space that can allow for thinkers to get away from the keyboard and into a place where their mind can wander more freely. I also thought that it was very interesting how the paper mentions the constraint that using a computer puts on gesturing. I hadn’t thought about this before, but it certainly applies to me; I am a VERY gesture-oriented person and feel that I often have to stand up and walk around to get some quality thinking done. I do somewhat disagree with the position of technology in the final section about risk. The paper argues that physical objects imply additional risk which leads to a higher quality output due to the risk associated with creating or using a particular item. Technology, however, also has a similar risk in that it can often be considered permanent once it crosses onto a network. Due to the low data duplication cost, a statement or action that is performed by someone using technology (searching the internet, viewing a website, downloading a file) is recorded in a variety of logs throughout the internet. If this information can be traced back to a particular person then it becomes risky to perform basic actions online as well.


David Wong - 10/12/2010 18:10:48

1) The "How Bodies Matter" paper discusses how considering embodiment can help influence and evaluate interface design in HCI. The paper explores five themes of embodiment and how it relates to interface design. The "On Distinguishing Epistemic from Pragmatic Action" paper discusses the existence of epistemic actions, which are used to help make a computationally easier decision for a problem, how these types of actions cannot be explained using the classical models of decision theory, and how might decision theory be improved to incorporate epistemic actions. The paper does this through an in-depth analysis of tetris.

2) The "How Bodies Matter" plays devil's advocate in a sense in how it pushes back on the reliance of technological solutions as the primary source of progress. In contrasting how paper medical records have benefits over electronic copies, the paper calls attention to the multiple benefits of utilizing our bodies and senses. The paper takes it further, however, and hints at the progression of technology in conjunction with the utilization of embodiment in interface design as the ideal way to move forward in HCI. In giving ways of deciding and evaluating embodiment in interface design, the paper is inspiring and forward thinking. The theme of the paper is very related to the "Beyond being there" paper in the sense that it pushes the integration of both body and technology.

The "On Distinguishing Epistemic from Pragmatic Action" is a bit dated in content. It brings up a good point, that epistemic actions are taken by users, however, the point proven did not seem too novel now in 2010. Systems that perturb their environment in order to gain a computationally easier problem are utilized in robot planning systems. In regards to HCI, I believe that the paper can naturally lead into the discussion brought up in the "How bodies matter" paper. More specifically, one could argue that with the importance of epistemic actions established, bodily epistemic actions must be considered in UI design.

3) The argument of the "How bodies matter" paper is sound. While it does not critically argue any point, it offers some nice themes of which to use to consider embodiment. These themes are well supported and offer good insight on the topic. The problem is well motivated as traditional UI design is not focused on embodiment.

The argument of the "On Distinguishing Epistemic from Pragmatic Action" paper is thorough and complete. Although the paper makes many qualitative judgments on why something is better than another, the overall argument is well reasoned and supported in the data. The problem at that time may have been well-motivated. However, I do not believe that such an extensive proof of the importance and existence of epistemic actions would be necessary today.


Siamak Faridani - 10/12/2010 18:16:17

Authors of the first article, On Distinguishing Epistemic from Pragmatic Action, use the well known game Tetris in order to define epistemic actions. An epistemic action is an action that rarely seeks to change the outer world. Rather it’s main purpose is to increase cognition. Epistemic actions increase cognition by using the following:

1) Reducing space complexity 2) Reducing time complexity 3) Reducing unreliability (by reducing errors)

In the experiment Tetris players used epistemic actions to decrease the amount of mental computations. They also observed that while elementary level players try to do the rotations in their heads. Most advanced players use the epistemic actions to visually explore possibilities

Relation between epistemic actions for tetris players and NP-hard nature of the problem: Tetris is an NP-hard problem. Even if we do not consider the stochastic nature of future shapes it still remains an NP-hard problem. What I noticed was that perhaps all the epistemic actions are there to visually perform a branch and bound method. Let’s look at the problem like this: each rotation starts a branch (each object can take 4 different modes of rotation) say each rotation mode can have 8 different translation mode. Thus for each object we will have 32 possible placements. A novice player can visually rotate and translate the object to examine all possible 32 options. Perhaps we can encode and NP hard problem in a simple game and ask people to play the game and through their epistemic actions find an optimal solution.

We can extend the observation to other games and interfaces, we use the similar epistemic actions for solving Rubik cubes (although unlike the tetris model, expert rubic cube solvers may not use the epistemic model and may build a mental task tree and just execute that). A more recent example might be (http://fold.it).

In the other article, How bodies matter: Five themes for interaction design, authors borrow from psychology, philosophy and sociology to form five themes for interaction design. These five themes are as following: 1) Thinking through doing 2) Performance 3) Visibility 4) Risk 5) Thick practice

In the section, Thinking through doing, authors point out that our body and mind are connected. They then look at examples of how we learn and show that we can use these examples to improve our interaction design and tangible UIs. It was interesting that even blind children use gestures. Additionally epistemic actions can be useful to reduce mental work.

Performance) how are we capable of performing complex physical actions without thinking. For example musicians are able to play an instrument without thinking through individual action. Perhaps we can use this to build richer UI behaviors and interactions.

Visibility) an interesting example in situated learning is the call center operator, I found this very interesting since I have a similar example about tennis. Tennis is heavily based on motor skills and hand-eye coordination I have surveyed many tennis players and they all believed after they watch videos of good tennis players they play a better game

Risk) physical actions have certain risks associated with them when we abstract the physical word we may lose the risk therefore loose the personal responsibility in our environment. One example of this might be replacing the traffic signs for school zones with drawings of a children on the asphalt. Regardless of whether or not we believe it is ethical it will virtually increase the risk to the driver and will make them alert and cause them to slow down.

Thick practice) intangible UI gives us more flexibility and freedom but should we replace all tangible UIs with intangible? when is it better to use one tangible UI with another tangible UI? The digital vynil is an interesting example since while the interaction remains the same for the DJ he can still bring the richness and flexibility of digital computers into his performance.

I found the article very interesting, it makes us think about how we can make an experience richer and how we can build better interactive systems. When we need to build a tangible UI and when we should replace it with an intagible UI. The article may not give us and answer to what we should do but at least it makes us think about why we are building the UI in some specific way.



Dan Lynch - 10/12/2010 18:17:33

On Distinguishing Epistemic from Pragmatic Action

This cognitive science paper uses the video games Tetris to study the cognitive steps that humans take to achieve a goal. Specifically, the authors want to define the notions of epistemic and pragmatic action. Epistemic actions are actions that we use to make mental computations easier, while pragmatic actions are actions that simply bring us closer to achieving a goal.

Its interesting that they first broke the game of Tetris down into basically 4 components: creating a bitmap, creating chunked representations, determining placement, and computing a motor plan. However, they noticed that people were rotating objects (which is a motor action) before the object even starts to enter the screen!

The basic premise is that people were using rotations and translations to make the game easier. Its very clear when the long zoid is rotated the least before it is dropped, as shown in the diagrams. The most complex shape, the L, was rotated the most before dropped or played.

I think this idea has many great applications... for example, what if google implemented a block-like UI to creating calendar events. Instead of rotations, perhaps using shifts in order to insert dates and appointments into our schedule. This could apply in many areas if there are interfaces that require mental cognition from a user before performing a task which is virtually what all UI tasks involve. However, I do think that finding an appropriate metaphor may be difficult.

How Bodies Matter: Five Themes for Interaction Design

This paper presents an interesting perspective on interaction design--learning by doing. The first example is about how bicycles are multitasking and can articulate the nuances of every task they must perform. The paper discusses the role of the gesture in cognitive development---this is extremely important in today's society with all the touch screen devices and multi-touch applications.

Think about it this way, if you are thinking about an action, then you are not putting 100% of your effort into doing that action. If you allocate more space in your brain to execution of an action, then you should theoretically perform that action better.

This paper motivates the idea of building more devices and gesture-enabled software for people to learn and use more effectively than their traditional counterparts. Think about if a CAD designer could actual draw on a drawing board. Just like the example of the tangible illuminating light table that teaches students about optical systems, it would definitely be a benefit over the mouse and keyboard monolith we have in place.


Pablo Paredes - 10/12/2010 18:19:36

This week I will rather take a mix approach to discuss some underlying subjects brought to light by both papers, Kirsh, D. and Maglio, P. - On Distinguishing Epistemic from Pragmatic Action and Klemmer, S., Hartmann, B. and Takayama, L. - How Bodies Matter: Five Themes for Interaction Desing.

Both papers present an important concept that has raised discussion and lately debate among researchers, but which seems pretty common sense for the common human being, which is: the importance of the body to learn and to gather information.

I believe the notion of importance of this type of research is two fold, first, because it indeed opens to a whole new paradigm of thinking, by integrating several other senses to the HCI and artificial intelligence challenge, and also because the origin of this mind-body separation has a strong modern origin in the Cartesian duality of mind and body theory.

The notion of intelligence as an effect of the brain and its internal processes and the way it was initially embraced by many researchers in computer science, neuroscience, and others although originated from a philosophical stand, also has a degree of simplification, as measuring causal effects is much easier than elaborating theories on a grey mass that is not clearly well defined and that especially has not a clear connection with what we call the mind (the intelligence itself).

From this already biased scenario, it was less risky to assume that the models of cognition are based on elements of memory, rational processing, peripheral processing and other events that do occur in the brain. Much more complicated would have been to "plug" the body in this already nebulous model to actually make it a component that adds intelligence beyond just being an actuator and complex sensing machine.

According to the authors of the papers, it seems that the "reality" in which the body evolves, which is composed of risks, physical laws, chemical reactions and so on, actually provide innumerable ways to process information which traditional brain-centered models could not explain.

It is evident that the different tools and bodies of knowledge presented do carry valuable insights on the value of the body as a means to gather information and to learn, as well as clearly show that incorporating the body in these processes there are advantages in some HCI design processes, however the question of whether we can actually model the underlying systems of knowledge remains open.


Matthew Can - 10/12/2010 18:26:05

On Distinguishing Epistemic from Pragmatic Action

This paper introduces and analyzes the concept of an epistemic action in cognitive and perceptual problem solving. Unlike pragmatic actions, whose purpose is to bring an agent closer to the problem’s goal state, epistemic actions make mental computations easier, faster, or more reliable.

To make the case for epistemic actions, the authors build a classical information-processing model of Tetris and show that it cannot explain some of the actions that players take. One example is players rotating zoids more than is necessary on average to fit the zoid into place. By examining player behavior, they conclude that rotations are also used to uncover new information in the game and to save mental rotation effort, among other things. This leads the authors to distinguish epistemic from pragmatic actions.

In HCI, the traditional model of the user is based on the classical information-processing model. Consider the Model Human Processor, for example. A lot of assumptions about user behavior are based on this model or ones like it. The reason this paper is relevant to HCI is that is reveals the gaps in the foundational classical models. If our assumptions no longer have solid footing, then perhaps we should come up with new frameworks based on a theory of epistemic actions. We might then focus more on facilitating actions that make the user’s mental computation easier as opposed to just actions that allow the user to take steps toward the end goal.


How Bodies Matter

This paper presents five themes for designing and evaluating interactive systems. The themes are based upon theoretical work on human embodiment. As such, they emphasize approaches to design that thoughtfully exploit the best of both the physical and the digital worlds.

One theme I liked in particular was that of thinking through doing. The paper had a number of examples to argue for the value of tangible manipulation in learning. I also found it interesting that gestures can ease the cognitive load during communication. The corollary to this is that systems that prevent people from expressing gestures make communication more difficult.

Something that had not occurred to me before reading this paper was that physical prototypes are not only a milestone on the path toward a finished product, but that they can also be viewed as a way to think through a design by building it. The notion that physical prototypes make design ideas concrete also ties into the theme of visibility, which says that workplace coordination is enhanced by the production and manipulation of visible artifacts.

The only thing I wish this paper would have done is to explain how some of these themes can be put into practice in HCI. They sound great, but it’s not clear to me how to use them. I would have liked to see some sort of applied framework or a case study with these themes in action. This isn’t necessarily a criticism of the paper though because it wasn’t the goal of the paper.


Luke Segars - 10/12/2010 18:32:10

On Distinguishing Epistemic from Pragmatic Action This paper investigates player behavior in Tetris to expose some cognitive ideas related to why people perform actions. The paper begins by explaining previously held theories that state that all physical actions are performed in order to get the performer closer to their physical goal. The paper refutes this statement by studying people playing Tetris who end up performing a number of actions that did not result in a direct furthering of their goal; instead they performed these tasks to aid the task of mentally computing the possible transformations of each shape. The distribution of timing of the rotations among various zoids was perhaps the most interesting part of this paper. There is a significant amount of information that can be gleaned from this alone; it is interesting, for example, that the T-shaped block is rotated later in the zoid’s progression although it is not necessarily rotated less. It’s somewhat unclear what this suggests although it likely has to do with either a lengthened or shortened identification stage before the aligning process can begin. In reference to How Bodies Matter, it’s important to consider that the ability to translate or rotate a zoid is a somewhat low-risk behavior and can be performed unnecessarily without any direct consequence. This particular rule, therefore, can’t be generalized to actions that may have higher risks or higher potential losses since the act of performing a minor permutation in a game is likely far less significant that the outcome of other actions. The mental model of moving a Tetris piece to an appropriate location seems logical but somewhat arbitrary. The paper explores how this model might be innacurate but doesn’t seem to arrive at any new well-defined model to replace it. While a number of the observations move in the direction of creating a new framework and many of their experiments confirm that the previous model is inaccurate, the conclusions of the paper are spread throughout twenty-something pages and never compressed into a digestable summary.


Anand Kulkarni - 10/12/2010 18:41:08

Pragmatic/Epistemic Action


The authors introduce and discuss the importance of epistemic actions in planning and acting tasks, through the example of Tetris.

There are several contributions here. First, though the authors do not emphasize it, the use of Tetris as a benchmark platform for discussing cognition and problem-solving is a relevant contribution of this work. It has since become relatively common in AI in robotics as a tool for testing planners and online reasoning systems. Second, the authors present the notion of "epistemic actions", physical actions carried out to increase understanding, which is a useful observation that can be usefully applied in robotics. Last, the authors argue that understanding internal cognitive processes, particularly around action selection, are important in understanding how to plan so-called epistemic actions (ie, actions which increase computational understanding), which may seem obvious in retrospect but may not have been apparent at the time.

The argument is structured around a set of user experiments and observations playing Tetris. Tetris is a good choice, since it's simple for users to understand and the cognitive processes are relatively straightforward. It's also a good choice because it's easy to develop planning algorithms that can implement various strategies; the information in the planning horizon can be limited (we can only see one piece ahead!). I like the analysis they've carried out on their users' work; it does justify their argument and provides a substantial basis for discussion. One weakness in this choice is that while users may need epistemic actions in playing Tetris it's difficult to believe algorithms would.


How Bodies Matter


The authors discuss several examples of how physical bodies matter in designing and evaluating interactive systems and interfaces.

The authors discuss several loosely linked areas where physical bodies are relevant. I like the notion of thinking through doing, which is the most relevant and obvious way that physical bodies matter. This is doubly important in robotics, where learning by demonstration becomes more important. Visibility is another important focus of physicality, particularly through the example of visual learning. Thick practice doesn't seem to be useful as its own theme here, so I didn't find it as compelling a contribution as the others mentioned.

The argument is loosely structured as a series of themes with supporting examples. This is a useful framework and the stronger earlier themes have several examples that are supported by analysis. I particularly liked the example of the Painstation pong game as an example of physical risk in interface design. The discussion of physical protoyping was another excellent example, if somewhat obvious. Again, I found thick description to be less meaningful, and the instance of the turntables didn't seem to synthesize these notions as well as it could have.


Aditi Muralidharan - 10/12/2010 18:42:29

In "On distinguishing epistemic from pragmatic action" Kirsh and Maglio argue for the existence of a kind of action called "epistemic action" based on a very thorough analysis many users playing tetris. They argue that epistemic actions (actions taken to gain information about the state of the world, rather than in directly in pursuit of a goal) explain the fact that tetris players make more rotations and tranlsations of objects than one would expect, and that they use them to gain information.

Their point is that epistemic actions exists, and that they exist because of specific properties of humans' attention and perception abilities - certain things are easier to conclude by seeing (in the Tetris example) than by imagining.

They did not make it clear how many user they used and how they were distributed by age and gender, and they used the very artificial world of Tetris, in which epistemic actions, if they existed, would be very easy to identify. I am not sure how far their analysis would extend into other domains, in which the deistincion might not be as stark, but apparently, the same applies to rearranging lettered tiles in Scrapbble - i.e. "to see how different options would work".

Building upon the theme that our physical capacities affect our cognition, Klemmer, Hartmann, and Takayama put forward "five themes for interaction design":thinking through doing, performance, visibility, risk, and thick practice. The bottom line seems to be that it really helps humans to be physically engaged with the tasks they are doing, where "physically engaged" means that they produce visible, tangible, permanent results in the real world.This seems like a very important point, with which I agree. It will be useful to me in my own research on search user interfaces to keep turning around these Ideas on visualization and direct manipuation .


kenzan boo - 10/12/2010 18:45:52

How Bodies Matter: Five Themes for Interaction Design

The article goes through examples of the 5 themes around user interface design. The 5 main ones are Thinking through doing. this involves epistemic actions, described as "Distinguishing pragmatic action—manipulating artifacts to directly accomplish a task — from epistemic action — manipulating artifacts to better understand the task’s context". The next one is performance, how fast people can input actions and convey intentions. Then visibility, how clear the the program is, then risk, which affects how adverse people are to changing parts of the system. If there was no risk people would just do it. The final one is thick practice, in practice the alternative of electronic medium may not be helpful yet, eg medical records.

On distinguishing pragmatic from epistemic actions.

Epistemic action is one that teaches us and helps us understand the system. In the tetris example, they directly manipulate the shape instead of thinking it in their heads which is much harder and requires more computation. The article also descibes a mental model: make bitmap of world, chunk it into different things, then position, eg shape, and where to put it in bottom row, then trajectory of where it will be.

The points addressed by thinking through doing were of particular interest. Many of our actions in interaction with a device or system is simply epistemic and do not provide a step towards the end goal. one system i can think of that really helps in the epistemic action in coding is the auto fill feature for coding in systems like eclipse. the drop down menu that provides what methods are available are for the user to discover them. From the epistemic action they can then do more and continue coding. or similarly googles suggested search features which help elucidate what the users intentions may be.

Also, on the aspect of full body motions, microsoft kinect and a lot of other game systems are attempting to gain more information from the user from the whole body rather than the eyes ears and one finger model. In a similar aspects, many phones today have incorporated many other sensors like location and orientation to more fully capture input from the user beyond the finger touch.


Aaron Hong - 10/12/2010 18:51:15

In "How Bodies Matter: Five Themes for Interaction Design" by Klemmer et al. they talk about how there are many aspects of the body that are not utilized for computing. Five themes they believe are particularly salient for interaction design: thinking through doing, performance, visibility, risk, and thick practice. They argue that we can't just imitate the real world, but we must go beyond it: "It is a physical realization of a symbolic reality."

Of the 5 themes I thought was most interesting was the one about risk. In our current medium of digital interaction we have minimized risk. But the paper advocates for increasing risk. The argument is that it brings back some of the social interaction that was once left out: trust, personal responsibility, etc. I agree with that, there are definitely to some degree too many "tolls" out there. But there is some benefit to minimized risk--it gives us an opportunity for creativity even. So on this aspect, we need to be very careful in how we reintroduce back risk. We definitely don't want to use shock disk as in the "pong" example.

In "On Distinguishing Epistemic from Pragmatic Action" by Kirsh and Maglio, they introduce epistemic action, distinguished from pragmatic action in that epistemic action are done to make mental computation easier, faster, or more reliable, while pragmatic ones are done to act on the world. They show this through their experiments on subjects and the game Tetris. This makes a lot of sense since it's like doing mathematics on pencil and paper, which means as we design for humans computer interaction we need to design such pen and paper in that they can perform epistemic actions.


Linsey Hansen - 10/12/2010 18:57:47

On Distinguishing Pragmatic from Epistemic Actions

In their article, the authors argue that performing actions in the real world and learning from them can often be more beneficial than using mental tasks to simulate the effects of an action. Based on Tetris, the authors introduce as a mental model a set of actions meant to be performed sequentially, and then show how this model is not only less efficient, but that it is probably not accurate after comparing it to how real people seem to go about solving problems in Tetris.

The key concept here is of course epistemic actions, which can be used instead of mental processing to improve one's cognition of the the world's current state. This is handy because it is not only faster than mentally altering the world in some cases, but it also reduces some of the mental load when solving complex tasks. Some examples epistemic actions are tying a string on your finger to remember something, preparing and setting out all of the ingredients for a recipe ahead of time, and even young children putting things in their mouths. To get to the point, being aware of these sorts of things can be handy when designing interfaces, since presenting data that can be more easily manipulated by a user (in the event that it cannot be visually represented in some other way) makes it easier for the user to take in. In a way these techniques are already being used in visual drawing programs, that allow a user to physically manipulate something as opposed to needing to calculate the degrees to rotate it by.

How Bodies Matter

In the next article, the authors discuss five things to consider when designing an interface. These things include thinking through doing (similar to the previous paper), performance, visibility, risk, and thick practice.

So out of these, I generally always think of the first three things (thinking through doing, performance and visibility), however I hardly ever think about risk and thick practice. Based on childhood memories, risk makes a lot sense. I remember when I was younger, before I knew about the fact that most drawing programs have an undo button, I would always draw really really really carefully in programs since I would assume that one mistake would ruin everything (this is back when most eraser tools were squares and thus annoying to erase with); when I first used illustrator, I was so amazed, since not only could I undo things, but I could move objects around, and if I drew something, it would not just melt on top of everything below it, but instead be its own separate stroke. As for thick practice, I kind of believe that that can be contradicted eventually. In the example where doctors find paper documents to digital ones superior, which is mostly to their ability to quickly jot down notes on paper, I am sure that there will eventually be some sort of e-ink like tablet with a wacom-like drawing surface that will make it digital, but feel just like paper to move around on. Combined with a touchscreen and the ability to quickly (eventually) sift through other documents, something like this would probably pass up the good old fashioned paper if it were to come out. I suppose that thick practice mostly refers to things that cannot currently be beat, but still, I feel like eventually almost anything can be improved upon.


Brandon Liu - 10/12/2010 18:57:47

"On Distinguishing Epistemic from Pragmatic Action"

The authors first create a model for Tetris player, based on the assumption that all the actions taken by Tetris players are pragmatic and moving towards a concrete goal. They show that this model doesn't work and then propose that some of the actions taken are epistemic, in that they are actions that are purely ‘informative'.

This paper was a lot of fun to read. It is very relevant to HCI since it describes why certain kinds of tangible user interfaces that allow epistemic manipulation are advantageous. It describes the benefit of these actions as ‘offloading mental structure to the world' - which is relevant to how creative tasks are done in the physical world. This could explain why writers and artists prefer physical mediums over working with a mouse and keyboard.

One simple experiment I didn't see in the paper was whether or not putting a square grid onto the entire Tetris playing field eliminated the epistemic action of using the wall to count spaces.


"Five themes for interaction design"

Thinking through doing - this relates to the previous paper on epistemic action. The major area for benefit is computer systems for doing creative tasks. For example, an application which would let writers sketch out ideas and then ‘cross them out' digitally would be an example of thinking through doing.

Performance - the paper made reference to video games for bimanual control. One area of HCI that wasn't really addressed is expert players of mouse/keyboard video games. These professional players use muscle memory in synchronizing keystrokes and mouse clicks. It would be interesting to study in other fields how muscle memory can emerge in an interface using only the mouse and keyboard. One criticism of this approach is that these types of interfaces are more stressful to use. A dimension to this area of HCI is the stress imposed on the user.

Visibility - I found, working in a corporation with a lot of cubicles, that it was hard to understand how expert users do programming. It was hard to observe expert users since most of them had a totally non-obvious environment customized to themselves. There are few opportunities for 'apprentice' style learning in a software development environment. One way this could be improved is through behavior-driven development. If less experienced programmers can observe more experienced workers sketch out their tests and user stories on a whiteboard or a ‘global dashboard', this can improve their effectiveness.

Risk - by nature, all HCI applications attempt to be reversible, so there really isn't any risk. A really interesting application of risk in computing is the "anti-akrasia" experiment, where users have a window to each other's screens. It is impossible to tell if your screen is being watched at any particular time, making it more embarassing/risky to be slacking off or checking Hacker News/Facebook.


Bryan Trinh - 10/12/2010 18:58:51

How Bodies Matter: Five Themes for Interaction Design

This paper describes the implications that prior work on embodied interaction engagement from the social sciences has on human computer interaction and designing interactions. 5 themes are enumerated that, the authors feel, provides a philosophically new way of thinking about interaction design.

The thinking through doing section illustrates a fact about human cognition that is not immediately obvious, despite the numerous times we are presented with examples of this in our own life. The simple act of manipulating objects that we are intending to interact with helps us learn about its use. The bit on expert tetris players is a particularly memorable example of this.

With respect to performance, there are many situations in which mouse and keyboard performance is superior to their more realistic counterparts. Gamers will tell you that there is no better option for a first person shooter game. This is due to the precision that is made available from the mouse and pointer that would not be possible in a simulated gun for instance.

Some of the later points in the paper were seemed like a little bit of a stretch though. The argument on the benefits of putting the user under higher risk of making a mistake seemed particularly reachy. Sure this would better approximate reality, but I would argue that is a really tough sell. The tradeoffs were fully addressed, but it seemed that the disadvantages far outweigh the gains in realism.

On Distinguishing Epistemic and Pragmatic Action

This paper compares the difference between epistemic and pragmatic decision making in the execution of certain tasks and uses Tetris as the illustrative example to their claims. The main idea is that under certain domains of task execution, the user can change the state of the domain as a way to think through the problem and arrive at a superior answer than pragmatic decision making alone. There are transaction costs involved in changing the state of the domain, but it can be outweighed by the benefits in increased cognitive ability. The trick is to properly balance the two.

In my own experiences I find this tradeoff to be true when solving problems in an entirely new domain. Simultaneously testing and thinking through problems, often provides the best solutions. Design consulting firms that are tasked to be innovative and create new products often go through a large number of prototypes guided by pragmatic thinking to arrive at the optimal solution.


Arpad Kovacs - 10/12/2010 19:04:08

The "How Bodies Matter: Five Themes for Interaction Design" paper is concerned with the fact that the keyboard and mouse interface homogenizes the physical performance of personal computing work, regardless of the task. To remedy this shortcoming, the authors propose the following 5 themes that would encourage richer interaction: 1) Thinking through doing - since physical interaction in the world facilitates cognitive development, and gestures lighten cognitive load, communication systems that do not constrain users' interaction styles will improve users' cognitive and communication abilities (eg through reflective practice that lets users work through a design problem, or epistemic action that involves manipulating artifacts to better understand tasks). 2) Performance - existing computing systems do not sufficiently utilitize bimanual input, motor skills, and experiential cognition to create action-centered tasks that leverage kinesthetic learning for unique tasks; instead, overreliance on a limited, universal set of actions requires high context-dependence, and slower reflective cognition. 3) Visibility - a rich interaction system supports collaboration and peripheral participation by clearly distinguishing between different actions, and thus enabling master-apprentice copycat behavior, as well as facilitating coordination through shared information and symbolism encoded in the artifacts. 4) Risk - tangible interaction requires commitment, forcing the performance of irrevokable actions with possibly unknown consequences. The benefits of this may be more personal responsibility and trusting relationships, as well as higher alertness; however high risk is not always a desirable system characteristic. 5) Thick Practice - real-world interfaces capture all of the characteristics that they embody, while a virtual simulation or computerized model may abstract away vital details.

I found the identification of the 5 themes to be valuable; in my opinion, the most useful part of the paper were the concrete examples (such as Final Scratch) where rich interactions provided significant improvements over the traditional mouse/keyboard input paradigm. I completely agree that the keyboard/mouse input does not offer sufficient input differentiation and kinesthetic feedback/learning; unfortunately most desktop computing tasks feel exactly the same, regardless of the actual task being performed. Hopefully, a new breed of tangible interfaces such as Siftables will begin to offer richer and more collaborative interaction possibilities. I for one am looking forward to the wide adoption of such input devices, in particular epistemic action and kinesthetic learning, that would allow me to experience greater input expressiveness, and collaboration with others. However, I feel that these rich interaction devices should augment, rather than obsolete the mouse and keyboard, which are still optimal for certain tasks such as word processing. Rather than trying to find the next exclusive "universal input device" (such as the latest craze of multitouch on cellphones), a combination of multiple input devices (eg keyboard, siftables blocks, light-pen, haptic glove, etc), each optimized for a certain task, would provide for the richest and most efficient interaction possibilities.

The main distinction the "On Distinguishing Epistemic from Pragmatic Action" paper draws is between the pragmatic actions, which change the world, and epistemic actions, which change our mental tasks. The paper studies Tetris, and finds that advanced players improve their cognition through epistemic tasks (effectively rotating blocks to see how they could fit), which reduce the memory (space complexity), steps (time complexity), and errors (unreliability) involved in performing mental computations. The paper asserts that when modelling planning as a state-space search, the nodes should be both physical and informational; in effect a sequence of actions should update the player's knowledge. The paper is mostly concerned with building a model of how an advanced Tetris player perceives the state of the system, processes this information into a salient model, and builds a plan of action, and tries to reconcile and improve these models based on the results of some empirical studies.

Although I thought this paper was a bit long, I found it quite interesting, and was astounded by the depth of analysis. In my own (limited) experience of playing Tetris, I use some of the strategies they identify, such as rotating to find matches; however I had no idea that there is such a deep range of epistemic actions for such a seemingly simple game. At times, I felt like the authors were reading too deep into the problem (eg multiple-perspective representations of zoids appears to be a bit far-fetched). However, the majority of their findings are sound, and the identification of these epistemic actions are the main contributions of this paper.

Since this paper was written in 1994, I would have imagined that there would be a greater emphasis on epistemic action since then. Sadly, the only instances that I am aware of are in WYSIWYG interfaces (eg moving images or text around in a drag-and drop interface). I have not seen epistemic action in use in programming environments, where arguably they could be of the greatest use. It would be awesome if we could reduce cognitive load when designing algorithms just by manipulating artifacts, and seeing how they could fit together.


Thomas Schluchter - 10/12/2010 19:09:43

Epistemic v. Pragmatic Action

The authors differentiate between pragmatic action, activities that are performed to further progress in a goal-directed context and epistemic action that lets individuals temporarily outsource cognition to their environment. Epistemic action is posited to be an important part of task accomplishment in decision-intensive activities.

Highly gratifying to read a paper that advocates to switch from a waterfall model of cognition to an agile model. Basically, previous approaches to explain the link between thought and action go through static phased models, whereas epistemic action introduces externalized feedback cycles into the thinking process that almost remind me of the current view of design processes. Epistemic action is kind of thinking with prototypes.

The study itself is based on a very limited and controlled setting, which raises the question of generalizability. The authors claim that "if epistemic actions are found in the time-limited context of Tetris, they are likely to be found almost everywhere" -- I would have liked to see more reasoning on why we should believe this claim.

While I don't doubt that the distinction between pragmatic and epistemic action can be made I ask myself how this is useful outside of cognitive science? How can we support through design the users' relationship with objects and their experimentation?


How bodies matter The authors argue that current interfaces to computing systems consistently neglect the importance of physicality to cognition and problem-solving. They synthesize research results from neighboring disciplines into five categories relevant to HCI.

I find the synthesis of different approaches to human cognition and action very valuable in the context of HCI. It opens up new perspectives on underdeveloped fields, for example Tangible User Interface that are not just intuitive but actually support expert use. Directing attention to these kind of segments of the field could help to advance the research to real-world relevance instead of producing visionary systems in an extreme niche.

Given that hardly any of the ideas referenced in this paper are completely novel, it is a little hard to understand why computer systems still look the way they do. I believe the point about a certain bias against the analog world in the CS community has a lot to do with this. It seems that the non-engineering disciplines contributing to the field of HCI need to be more assertive about the knowledge they bring to it. It would be interesting to see work that comes to the conclusion that a certain process cannot be enhanced through computation, and I wonder whether that would be an acceptable (and publishable) result.