Distributed and Embodied Cognition

From CS260 Fall 2011
Jump to: navigation, search

Bjoern's Slides

Media:cs260-14-cognition.pdf

Extra Materials

Discussant's Materials

File:RohanNagesh-EmbodiedCognition.pdf

Reading Responses

Steve Rubin - 10/21/2011 17:19:14

The first paper argued for the importance of epistemic actions in cognition models in addition to pragmatic actions. It did this through an analysis of Tetris, noting that experienced players often perform actions that cannot be interpreted as entirely pragmatic. The second paper enumerated five themes for incorporating bodies and physicality into interface design.

"On Distinguishing Epistemic from Pragmatic Action" begins by establishing a cognitive model for how we play Tetris. The rest of the paper--notably, the discussion of the differences between pragmatic actions (actions that lead directly to the desired goal) and epistemic actions (actions that provide more information about the system)--is based off of this model. This begs the question: how much of how we label an action as "pragmatic" or "epistemic" depends on our model of the average user's cognitive process? Without being sure that a model is accurate, any further analysis is necessarily dependent on an assumption. That being said, the upshot of the paper was sound: we should consider things that can be done in our interfaces that are not necessarily intended to get the user from point A to point B, but that can help them understand where point A and point B are with respect to each other.

My main question regarding the second paper was one of a more philosophical nature. In many respects, the best interface for a PC application is likely not one with a mouse and keyboard--Final Scratch is a clear example of that. However, the universality of the mouse and keyboard (and now, touch screens) are an attractive draw for developers. Where, then, do we draw the line as researchers? How much of our time should we devote to cute interactions techniques that lack this kind of universality? I think that if new techniques are experimentally shown to be faster or more accurate, then they are worth publishing, but it may be disheartening to do this kind of research knowing that in all likelihood, the specificity of the technique will prevent it from having impact.


Valkyrie Savage - 10/21/2011 17:48:01

Main idea:

These papers extol the virtues of bringing the physical back into the digital. It doesn’t, they argue, make sense for people to forego all their bodily knowledge to interact with computers, nor does planning in this way improve their chances of doing well.

Reactions:

The paper by Kirsh and Maglio is actually one I’ve come across before. Well, I hadn’t read the paper before, but had discussed the ideas in it. Anyway, I think it’s fascinating stuff! It of course makes perfect sense that people should use external computation when possible: that’s a bit of the idea of the internet, isn’t it? These days there are studies popping up (in pop science news, even) about how children’s brains’ structures are morphing from their uses of the internet, and how it effectively functions as an external storage device for knowledge. That’s not directly related as storage != computation, but I was reminded of it reading this paper. I liked their analysis in this paper, but the authors could have done a better job of describing their experimental setup... I’m not certain exactly what it is that they did: was this on an arcade Tetris machine? Did they modify it in place? Did they build their own Tetris machine? Was it on a regular computer screen? They did do a good job of describing how they drew their conclusions and citing prior work, though!

The other paper was also interesting, and it’s something that I’ve thought about a fair bit. Why is the tendency to digitize everything poorly? It’s hard to say.. it’s easier, I suppose, and the versioning models it introduces (as mentioned) are valuable in their own ways, but sometimes doing things on the computer is just not as awesome as doing them in real life: I played Dominion (card game) online for a while to practice, but the fact that I a) couldn’t socialize with the other players, b) couldn’t get an intuitive sense of how many or which cards I had since they came up randomly automatically, and c) could play over and over and over and over without set-up or clean-up meant that a lot of my time was sucked into playing and I didn’t even really enjoy it that much. So I guess my compliment to this paper is that what it discusses is something that I’ve pondered and all the things in it seem to make sense. Also, I enjoyed that so much out-of-field work was brought in (Piaget, etc.) to lend some credibility to the discussion!

I was a bit confused about parts of the organization of the paper, as the five mentioned areas didn’t seem as much like dimensions as a series of ideas that built on each other. Performance pretty obviously builds on thinking through doing, and visibility is an artifact of performance, etc. Also I felt that thick practice was tacked on at the end and not well-described enough...


Laura Devendorf - 10/23/2011 15:21:30

"Epistemic and Pragmatic Action" offers a study and analysis of the behaviors observed while playing Tetris in order to demonstrate that a "best move" isn't always the one that gets you closer to your goal (pragmatic) but could be one that takes you away from the goal but is more helpful in the long run (epistemic). "How Bodies Matter: Five Themes for Interaction Design" does most of what the title says it's going to do: it defines five themes for interaction design and their associated qualities.

Maybe it was my retro-fondness for Tetris but I did very much enjoy "Epistemic and Pragmatic Action." I was interested in the arguments they made about how best actions can't always be reduced to what makes you closer to your goal. I believe that such simplifications of actions, namely that they occur linearly towards a goal, are suspect. The authors make a very interesting contribution to AI but I have to wonder how much of their finding are based on the Tetris scenario and the organization of the Tetris game. I was also a bit curious where they came up with some numbers, such as the amount of time it takes to rotate something mentally. Since I was inspired by their ending quote, "This same idea, we claim, holds generally throughout all of human activity," I'll finish this thought with an analogy on my own human activity. I took a one day vacation yesterday: no school work, no computer, no email. I bought a bike, rode around town, played tennis, ate lunch outdoors, had a bbq with my neighbors and went to a show to see one of my favorite bands in downtown Oakland. While this behavior may appear to take me farther from my goal of getting a phd, I was inspired by so many things I saw, came up with some project ideas of questionable usefulness, and I'm much more focused on my work today. Did my actions yesterday reduce my cognitive load in a similar way that the Tetris strategies did for the users? Do they illustrate that it is often the case that there is not one exclusive path towards a goal state. Maybe there is not distinction between pragmatic and epistemic actions - it's just a redefinition of a heuristic and path from start to goal.

In "How Bodies Matter: Five Themes for Interaction Design" I particularly enjoyed the discussion on Risk. The authors says, "retaining elements of risk in practice can be beneficial," I agree, I would like to see some examples of interfaces where it is beneficial. While the author mentions that there is a difference, I would be particularly interested in classifying the differences and how it changes work practices. I was confused by Norman's "negative emotional" label of classification as it seems to jump out at me as a "bad" thing. This is troubling because I don't think risk is necessarily bad. I think the foresight required when putting together a letter on a type writer is necessarily negative, I would argue that it is more thoughtful and emotionally positive. I can't conclude what he means for sure and when I checked the reference to look it up, I realized that it's in a book that I'm currently on the waitlist for at the library. Overall, I found the article helpful in finding supporting ideas for some of my own projects. Also, bonus points to the authors for working Milli Vanilli into academic discourse.


Yun Jin - 10/23/2011 17:56:44

The first paper introduces the general idea of an epistemic action and discusses its role in Tetris, a real-time, interactive video game. Epistemic actions -physical actions that make mental computation easier, faster, or more reliable-are external actions that an agent performs to change his or her own computational state. More precisely, they use the term epistemic action to designate a physical action whose primary function is to improve cognition by reducing the memory involved in mental computation, that is, space complexity; reducing the number of steps involved in mental computation, that is, time complexity; reducing the probability of error of mental computation, that is, unreliability. The second paper draws on theories of embodiment—from psychology, sociology, and philosophy —synthesizing five themes they believe are particularly salient for interaction design: thinking through doing, performance, visibility, risk, and thick practice. We intro- duce aspects of human embodied engagement in the world with the goal of inspiring new interaction design approaches and evaluations that better integrate the physical and computational worlds. And this paper presents five themes that they believe are particularly salient for designing and evaluating interactive systems. The first, thinking through doing, describes how thought (mind) and action (body) are deeply integrated and how they co-produce learning and reasoning. The second, performance, describes the rich actions our bodies are capable of, and how physical action can be both faster and more nuanced than symbolic cognition. The first two themes primarily address individual corporeality; the next two are primarily concerned with the social affordances. Visibility describes the role of artifacts in collaboration and cooperation. Risk explores how the uncertainty and risk of physical co-presence shapes interpersonal and human- computer interactions. The final theme, thickness of practice, suggests that because the pursuit of digital verisimilitude is more difficult than it might seem, embodied interaction is a more prudent path.


Sally Ahn - 10/23/2011 18:38:56

Kirsh and Maglio introduces the notion of epistemic action, defined as "physical actions that make mental computation easier, faster or more reliable." They contrast idea with the pre-existing view of actions, which they call "pragmatic actions," by pointing out that the primary focus of pragmatic actions is to "create physical states which physically advance one towards goals." The authors distinguish these two types of actions through experiements conducted with Tetris players, which revealed that expert players performed more superfluous actions (e.g. translating a block to the edge of the screen then back to verify column) that cannot be modeled by behavior theory that considers only pragmatic actions and excludes epistemic actions.

In terms of HCI, I think it'd be interesting to consider how to balance epistemic and pragmatic actions in the design of interactions for specific contexts. The authors themselves note that the difficulty of recognizing epistemic actions stem from the fact that actions can serve both epistemic and pragmatic functions (e.g. rotating a zoid in the direction towards goal helps with identification and physical movement of the zoid). This suggests to me that the ideal interactions one should be aiming for would be those that does precisely that--simultaneously perform epistemic and pragamatic purposes. Of course, this only identifies a challenge and doesn't really ease the difficulty of designing such interactions. Tetris was a good context for illustrating the existence of epistemic actions because the game consisted only of actions whose pragmatic function can be evaluated in a determinate manner (does the action bring the piece closer or farther away from the goal?). I would imagine that in real life, however, the vast spectrum of possible actions make it much harder to distinguish the immediate value of each action, let alone design for interactions that optimize these values.

While the first paper focuses on modeling human behavior, "How Bodies Matter" applies the concepts of such models to HCI by constructing a framework of five themes for designing and evaluating interactions. These five themes (thinking through doing, performance, visibility, risk, and thickness of practice) reveal the richness of human experience with the physical world that is often absent in current forms of interaction with technology. A key contribution of this paper is the comprehensive nature of these five themes and their applicability to interaction design considerations, compared to the more limited scope and abstract ideas presented in papers specific to cognitive science. For example, the first of these themes, "thinking through doing," encompasses epistemic action that was presented by the first paper. While reading that paper, however, I wondered about the absence of discussion about "muscle memory," which I thought may be contributing to the experts players' pre-emptive actions. Thus, I was glad to see a section devoted to "Motor Memory" in the second paper, which discusses pros and cons of designing interactions that leverage kinesthetic memory. Overall, the paper highlights an important challenge that underlies interaction design for any new technology: how to modify existing interactions with the physical world so that we gain new benefits without losing existing benefits? One obstacle of this goal is the difficulty of recognizing the subtle benefits of physical interactions that have indeed been lost, and I think the framework of five themes and their illustrative examples presented in this paper help to overcome this particular obstacle.


Derrick Coetzee - 10/23/2011 21:20:06

This week's readings discussed the role of the human body and physical interaction in cognition and interfaces.

The first study, "On Distinguishing Epistemic from Pragmatic Action" by Kirsh and Maglio, was a focused study that showed how expert players of Tetris rely heavily on epistemic actions that manipulate the game world state to aid cognition - such as rotating pieces to visually evaluate where they might best fit. In a sense, expert players learn to increase performance by converting the problem into one amenable to the efficient primal functions of pattern recognition and hand-eye coordination. This runs counter to the popular intuition that experts are more effective at solving problems because they are able to "do it all in their head." From a materialist point of view, it could be said that the human brain and its environment form a combined computational system with greater overall resource efficiency than the human brain alone, even if the environment has no computational ability of its own and serves only to reconceptualize the problem in a way accessible to specialized brain functions like pattern recognition.

The most immediate objection to the study is that their proposed explanation is speculative and performed after-the-fact based on available data, and might be explained by other mechanisms (some of which might not even involve epistemic action). For example, players may simply rotate pieces many times because they change their mind on where to place it as it falls; or they may simply press the button accidentally. Without the ability of players to introspect effectively, further experimentation is needed to distinguish alternative underlying mechanisms.

"How Bodies Matter" by Klemmer et al. is a higher-level meta-study that leverages empirical studies to build a framework describing the role of the body in cognition and in interfaces. I was surprised by a number of the results, e.g. that merely using a headset could increase user creativity on the phone. I was also intrigued by the laundromat anecdote, in which GUI interfaces frustrated a child's attempt to learn by emulation - this would be an interesting standard by which to judge many interfaces (e.g. video game interfaces can often be effectively learned this way, while most desktop applications can't).

Having participated in many long-distance relationships, I was deeply suspicious of the cited claim that there is a level of trust that cannot be attained through electronic communication alone. While it remains easier to deceive others about things like personal appearance in electronic media, other more salient aspects of oneself become just as or even more accessible, increasing risk and commitment in other areas. I was also suspicious of the claim that negative behaviors caused by online disinhibition could be attributed to "the lack of social context cues" (in reality, lack of consequences is a more important factor, and one that can be mitigated by online community practices). Nevertheless, both types of interaction have independent skill sets that must be learned to fully exploit them - an online newbie is at considerably greater risk of being deceived than a veteran.


Viraj Kulkarni - 10/23/2011 22:48:57

On Distinguishing Epistemic from Pragmatic Action' introduces a new notion of an epistemic action. The actions you perform to complete a defined goal can be classified into two sets. Pragmatic actions are those that take you closer to your goal. Earlier cognitive theories have studied these pragmatic actions in great detail. However, this paper introduces the notion of an epistemic action which does not necessarily take you closer to your goal. The objective of this action is to simplify the problem solving task. These actions are studied through a study on how players play tetris.

The chess example suggests that human actions are a lot more complex and cannot be modeled as state-changes that make progress towards a goal. The example appealed to me because I frequently do that while playing chess. As you play the game, you build your strategies on the existing state of the game. However, as the state changes you continue incrementally building your strategies and, at a point, it can happen that the original conditions you built your strategy upon have been modified considerably. At such a point, you know you have to come up with a fresh approach but you just can't shake off your previous strategy. It helps a lot to physically look at the board from a different angle or vantage point.

'How Bodies Matter: Five Themes for Interaction Design' discusses five themes that should be considered while designing and evaluating HCI systems. The paper speaks about how we cannot separate our physical behaviour from our psychological behaviour - or rather how our mind and body are closely tied together. We learn through the physical actions we perform. It is more effective if the user gets physically engaged with the task he is trying to accomplish. I found the part on instilling risk in an environment very appealing. Higher risk forces people to stay focussed and pay attention to detail while lower risk enables them to be creative and curious. I like the paper's treatment on risk which builds on the point that risk, attention and engagement are intertwined.


Hong Wu - 10/23/2011 23:02:13

Main Idea:

Both papers talks about how people interact with the world.

Details:

“On Distinguishing Pragmatic from Epistemic Action” described the experiment of Tetris and the observation of how players interacted with the video game. The paper discovered that more experienced player were willing to rotate the shape in Tetris more so that they can observe the shape in direct. And experienced player moved the shape to the edge of the screen and moved it back before dropping the shape. The experiment and observation showed that it is fast for observation rather than imaginary.

“How bodies matter” introduced the reaction of human body from five aspects, including thinking through doing, performance, visibility, risk, and thick practice. The paper describe the view of the design of interactive systems.

Both papers are related to HCI research. Human embodied engagement inspires new interaction of computer science. Understanding the behavior and habit of human body will greatly help the UI design of software and IO interface efficiency.


Amanda Ren - 10/23/2011 23:18:05

The Kirsh paper distinguishes pragmatic action and epistemic action and how it relates to the Tetris video game.

We are introduced to epistemic actions, physical actions that improve cognition by reducing steps for mental computation and probably of error from mental computation. This differs from pragmatic actions, which bring you closer to a physical goal. By introducing epistemic actions, it explains why players use extra rotations (such as determining an ambiguous block). I can seen these methods being applied to Scrabble-like mobile games. Like a real game where users can physically rearrange the letters to help them find new words, on mobile games there is usually an option to randomly shuffle the words around. By doing this, users have to do less mental computation to come up with words.

The Klemmer paper discusses how there are give themes essential for interactive design, focusing on the fact that our physical bodies play an important role in our experiences.

The five themes that are important for designing and evaluating interactive systems are: thinking through doing, performance, visibility, risk, and thickness of practice. This paper is important because it gives examples supporting these five themes, supporting the notion of richer interaction paradigms. I thought it was interesting that they brought up the role of gesture - I've experienced how simple hand movements make communicating vocally much easier. Another example I found interesting was the using physical paper medic records to help supply more information about a patient - such as worn records, or pencil recordings.


Allie - 10/24/2011 0:11:46

"How Bodies Matter: Five Themes for Interaction Design" by Klemmer, Prof. Bjorn, and Takayama, introduces 5 themes believed to be relevant for design and evaluation of interactive systems: 1) thinking through doing 2) performance 3) visibility 4) risk and 5) thickness of practice. Unlike theories of information processing and human cognition that focus on thought as constrained within the head; theories of embodied recognition regard physical activity as essential to human understanding of human cognition.

Research has shown that physical interaction in the world facilitates cognitive development, particuarly in infants. It has also been shown that systems that constrain gestural abilities also hinder the user's thinking and communication. Pragmatic action, is the manipulation of artifacts to accomplish a task; versus epistemic action, or the manipulation of artifacts to understand the task's context. In Tetris, players perform epistemic manipulation of the pieces in order to understand how different options work. Reflective practice is the working through, rather than thinking through of design challenges that connect physical action and cognition. Backtalk also provides help to uncover problems or generate suggestions for new designs.

The purpose of tangibility is that these interfaces provide "natural" mappings that leverage our familiarity with the real world. The incorporation of an artifact into bodily practice can be to the point where people perceive it as an extension of themselves; acting through it rather than on it. Experienced cognition, or actions that are learned skillful behavior but not reflexes, encompass driving a car/motorcycle, operating power tools, or engaging in sports activities for which planning through explicit cognition is too slow.

The paper also emphasizes the social life of physical artifacts and their visibility as important facilitators in distributing the cognitive work of groups. Further, one cannot undo a social faux pas in face-to-face interactions; technology mitigates against this risk: sentences can be deleted before being sent to friends in electronic format. In sum, being able to index into past actions through undo/redo and versioning is the most important characteristic that separates digital/physical interactions.

Not sure how much I liked the statement that making one's actions more visible / the risk more salient increases one's sense of personal responsibility for decisions, in particular, helping to overcome the inclination for obedience to authority.

The paper concludes that the value of electronic media should be measured by how they can surpass co-presence and co-location to offer value even when face-to-face communication is available.

In "On Distinguishing Epistemic from Pragmatic Action" by Kirsh and Maglio, cognitive and perceptual problems are explored in the context of Tetris. Pragmatic actions and epistemic actions are further expounded upon. Epistemic goals try to simplify mental computation in order to find the most cost-effective allocation of the agent's time/effort. In Tetris, the best way to interpret the actions are actions that simplify the player's problem-solving task.

Epistemic action improves cognition by 1) reducing memory, or space complexity 2) reducing steps rqeuired in mental computation, time complexity 3) reducing probability of error, or unreliability.

Tetris was chosen as the subject of this study because it requires split-second decisions both percetual and cognitive in nature. Tetrazoids can be translated right/left/rotated/dropt. While the moves are simple and easy to pick up, players must master the mapping between keystrokes and effect, learn to recognize soids quickly regardless of orientation, and acquire spatial reasoning skills.

RoboTetris was implemented to help the researchers in this paper computationally explore cognitive problems underlying Tetris. In so doing, 1) an early bitmap representation of selected features was created 2) the bitmap was encoded in a compact, chunked, symbolic representation 3) the best place to put the zoid is computed 4) trajectory of moves to achieve the goal placement was computed.

In RoboTetris, the process is to search for the largest uninterrupted contour segment that the zoid can fit and weigh against others on the basis of a set of additional factors. Rotations and translations occur quite a bit, but if players wait until they have a plan, the number of rotations average to half that can be performed before an orientation repeats.

Interleaving describes cascade processing, in which each phase in RoboTetris begins its processing before it has been given all the information it would eventually receive. The earlier an estimate is made, the greater the chance it will be wrong, and the more likely the agent will make a false start.

Cognition is necessary for intelligent action, but action is never necessary for intelligent cognition. The paper finds that it is difficult to prove an agent performs an action for epistemic rather than pragmatic reasons.

Orientation-independent repressetation means it should take no more time to judgge whether 2 shapes are the same, however many degrees apart they have been rotated. Players' reaction times on mental rotation are plotted on the horizontal line, and total reaction time should be the sum of the time needed to abstractly encode the first shape, the time to abstractly encode the second shape, and the time to compare the abstract encodings. The more exposure test subjects have to shapes in test orientations, the closer to flat-line performance they display. The 2-depth decision-tree is preferable over the 3-depth decision-tree because probing the same call on most of the successive queries would put less strain on the attentional system. Epistemic payoff helps explain why expert players sometimes choose pragmatically disadvantageous actions within a rational-agent calculus.

The papers are interesting and more relevant than some of the recent papers we've read for class, in particular because it relates HCI research to systems that attempt to simulate human cognition to computer systems. The paper on Tetris is quite interesting, though am not sure its assertion that all human activity can be assumed to make it easy for agents to improve execution.


Shiry Ginosar - 10/24/2011 0:23:45

These two papers stress the importance of physical manipulation of the world as part of computation. While Kirsh et. al. discuss the difference between pragmatic actions that bring an agent physically closer to a goal and epistemic actions that bring an agent mentally closer to a computation, Klemmer et. al. present a survey of the general concept of embodiment and its implication on user interface design.

I was quite impressed with Kirsh et. al.'s ability to study a very non concrete concept is such a scientific and rigorous methods drawing on prior art in multiple fields. Regardless of the validity of their initial mental model behind playing Tetris, their scientific method is something to look up to. I especially liked their choice of Tetris as their point was easily expressed in the game's domain. Klemmer et. al.'s paper was a nice complement to Kirsh's one as it did a good job of extending the ideas presented on the importance of physical manipulation to other domains.

Physical manipulation as an aid to computation is an interesting concept. I wonder how theory relates to general problem solving, and in particular to the kinds of computations we ask workers to do on crowd sourcing systems. For manipulation to be possible, context regarding the problem at hand needs to be available to the person performing a task [Kirsh]. However, in many crowd sourcing systems work is cut up into tiny elements of work that do not allow for any context and may hinder the ability of the worker to perform a high level task. Perhaps this is the reason why high quality work is hard to come by on systems like Mechanical Turk without the use of proper crowd shepherding?

On another note, while I agree that embodiment is important to human computation, I wonder how human performance could be improved by augmenting a system to include feedback loops that would only be possible in the digital world. How can eye gaze tracking, brain signals and other indications of attention not replace the physical manipulative actions of a user but act as additional additive forms of input to a digital system? The simplest example that comes to mind are pop-ups that stay up when you look at them - manipulation is not replaced in this case, it is merely augmented and improved.


Peggy Chi - 10/24/2011 0:31:56

The two papers discussed the importance of physical actions and their implication to problem solving. Kirsh and Maglio presented their study on how actions might quickly assist solving the problems than thinking in user's head alone. Klemmer et al. discussed different aspects of interaction design relavant to embodied engagement.

I found these papers very interesting especially about the model of thinking through doing. This reminds me of playing music instruments, sports, or other activities that we use our bodies to memorize and practice. In the computer world, I often find performing shortcuts (e.g. c&p, open file, etc.) or typing passwords having similar effects. I believe that by properly designing interactions that make use of human's actions and working memory, we might improve performance and user experience. However, to what degree can we rely on users' physical practices? When an action cannot be easily learned or mapped, will it brings more negative effect to the experience? How do we define the actions that can be accepted by most users, and those that are limited?


Alex Chung - 10/24/2011 1:02:35

The “interface” part of human-computer interface is not restricted to intangible software. HCI includes any artifact that mediates the interaction and communication between human and computer or human and human through computer as a medium. The authors of “How Bodies Matter” provides five themes for interaction design as well as evidences to support their inclusion.

Thinking through doing is a key component of brain development during childhood for all animals. Without the concept of language and threshold for trust, the brain uses the body sensors to interact with the environment and to learn the risks and rewards of various actions. Dr. Eric Kandel (2000 Nobel Prize in how human memory is physically stored) explains that the difference between short-term and long-term memory is the number of neural connections between the neural network and the memory location. Thus it is possible that learning through action triggers more stimuli to the brain than just reading or listening. This is a crucial concept for HCI design because a better design would convey richer meanings to the user without the burden of demanding cognitive effort.

The highest form of praise to an engineering design is where people perceive that artifact as an extension of themselves. “They act through it rather than on it” is a phrase that is often used for sporty automobile design and engineering. Mazda’s famous little roadster MX-5 was designed around the philosophy of Jinba Ittai – a Japanese phrase describing unity of horse and rider in mounted archery to describe the driver’s connection with the car. Force feedback, vibration, and noise are purposefully sent to the drivers as physical cues that can be tacitly understood. Since the perception of directness is dictated by the amount of feedback to user’s action, physical cues such as haptic technology and audio recordings along with visual can improve the user’s understanding of the actions performed by the computer.

However, I am skeptical that mapping motor memory with dedicated actions would constrain the design space. While dedicating physical movement to interface functions affords learning and memorization, it might not the most efficient and functional design. For example, the original Window Tablet affords users to write on the screen with a stylus like paper and pencil. However, from personal experience, the process was slow and the scribbling was hard to read. Human brain is flexible enough to learn and to adapt; therefore, HCI design should not sacrifice function for the sake of natural movement. User-friendly design might attract more audiences at first but a useless design would lose audience just as fast.

Then the paper got preachy and took a turn toward philosophical discussion on risk, trust, and responsibility. The inclusion of Milgram’s studies seems out of place and the whole section about personal responsibility does not sound genuine or convincing. It would be better if removed.


The second paper by Kirsh and Maglio of UC San Diego reports on their studies of epistemic learning by studying the behavior of Tetris players. Their studies are important to HCI research because it illustrates the design and analysis of experiments to study the cognitive process of human.

Positive: The decision of using Tetris to explain the motivation behind epistemic actions. Games such as Tetris are repetitive and the settings are clearly defined to limit the number of independent variables. Sometimes researchers can leverage existing software for studies instead of building everything from scratch.

Positive: Cognitive process cannot be directly measured. Researchers must infer the process from interpreting the results. Thus the study environment must be well controlled to prevent ambiguous results.

Negative: The author asked the question of “how context affect performance?” However, the study did not remove the color scheme to ensure that the users were only studying the shape of the blocks.


Donghyuk Jung - 10/24/2011 3:11:06

  • On Distinguishing Epistemic from Pragmatic Action

Argument: In Tetris (a real-time, interactive video game), certain cognitive and perceptual problems are more quickly, easily, and reliably solved by performing actions in the world than by performing computational actions in the head alone.

The authors found that Tetris experts perform a variety of epistemic actions (physical actions that make mental computation easier, faster, or more reliable) to reduce their internal computational effort. While regular players might rotate the blocks in their head first, advanced players might rotate them physically. According to the paper, Epistemic actions improve cognition by doing the following: Reducing the memory involved in mental computation (space complexity); Reducing the number of steps involved in mental computation (time complexity); Reducing the probability of error of mental computation (unreliability).

Their idea that theories focused almost exclusively on pragmatic actions in cognitive science are not enough to explain the full extant to which humans utilize the external environment. I totally agree that both pragmatic and epistemic actions can be embedded and embodied in each other. However, I think that epistemic action can be explained well when we explain it by using the concept of tacit knowledge.

  • How Bodies Matter: Five Themes for Interaction Design

“A synthesis of theoretical and empirical work (drawn from psychology, sociology, and philosophy) that provides insight for both ideation and evaluation of interaction design that integrates the physical and computational worlds.”

“Synthesizing five themes we believe are particularly salient for interaction design” 1. Thinking through doing (Individual corporeality): The connection between thinking and doing as uncovered by educational theorists, gesture researchers, and cognitive scientists. 2. Performance (Individual corporeality): An action of the body’s ability for skillful performance. 3. Visibility (Social affordances): How practices that are physically distinct support collaboration and coordination. 4. Risk (Social affordances): Risk is having to choose an action which cannot be undone while the consequences of the action are not fully knowable ahead of time. 5. Thickness of Practice: Designing interactions that are the real world instead of ones that simulate or replicate it hedges against simulacra that have neglected an important practice.

This paper comes for me at a moment when it is useful because we just submitted the abstract for our final project. I need to go through these five themes in order to examine a hypothesis. In addition, it gave a chance to look up many different viewpoints of other fields and hunt out the knowledge in literature.


Suryaveer Singh Lodha - 10/24/2011 7:55:41

Epistemic and Pragmatic actions

This paper describes the differences between epistemic and pragmatic actions, using Tetris as the key example. The main argument of the paper is that humans perform many actions that do not pragmatic (or) necessary given the problems they are trying to solve; these actions, called as epistemic actions, are however extremely useful and necessary as they seem to reduce the cognitive workload - by reducing the space , time complexity of the computation involved and by reducing the unreliability involved with the suggestions coming out of the mental computations. The study shows that intermediate and expert game players of tetris perform many such epistemic actions to be successful in their game play. The direct consequence of this paper to HCI is that interaction systems should enable users to use their interaction environment in a variety of creative ways. At the least, interaction systems should not restrict users from manipulating their environment. The cost-benefit model explains why players physically rotate to save mental rotation- They can attain the same knowledge faster and with less effort than by mentally computing the image transformation. Rotating to facilitate matching has a favorable cost-benefit spread because matching via perception is fast, reliable, and uses less resources than matching in working memory. Without a notion of epistemic payoff, its difficult to justify why expert players sometimes choose pragmatically disadvantageous actions within a rational-agent calculus.

Five themes for interaction design:

This paper explored five themes for interaction design revolving around the use of our physical bodies: thinking through doing, performance, visibility, risk, and thick practice. In explaining thinking and doing, the authors talk about the importance of the connections between action and cognition. Humans do a better job when they engage bodily while solving problems - example: most humans are comfortable solving math problems by writing down the in-between, and sometimes unconnected steps rather than working them out in head. In discussing performance, the authors argue that interfaces could be designed for expert performance if users ability to skillfully make use of their body is accounted for. For example, music controllers that facilitate engagement of the entire body, which is a central feature in almost all popular instruments, than just the finger and wrist are presumably more effective - in the sense that users would perceive them as extensions of themselves. Also game controllers and simulators that mimic actual controllers - steering wheel and pedals instead of the arrow keys, increase vehicle control and enhance user engagement. User activity should have 'visibility' in order to enable easier collaboration, and user interfaces should facilitate 'risk', which can engender trust in human relationships, enable focus, and materialize personal responsibility. Finally, the paper argues that user interfaces should support 'thick practice', successfully modeling all of the relevant characteristics of some human activity that the interface enables, and taking existing physical interfaces into account. The bottom line seems to be that it really helps humans to be physically engaged with the tasks they are doing, where "physically engaged" means that they produce visible, tangible, permanent results in the real world.


Vinson Chuong - 10/24/2011 8:46:03

In "On Distinguishing Epistemic from Pragmatic Action", Kirsh and Maglio argue that not all physical actions are intended to directly bring the actor closer to his goal, that some physical actions are intended instead to improve cognition. In "How Bodies Matter: Five Themes for Interaction Design", Klemmer, Hartmann, and Takayama combine this theory with many others to arrive at a set of common themes for how our mental representation of the world interacts with the physical representation.

Supported by observations of the various actions players take in playing Tetris, Kirsh and Maglio argue that there are two distinct types of actions people take with respect to a goal: pragmatic action, which directly moves the actor closer to his goal; and epistemic action, which assists in cognition by simplyfying a data-processing task. They challenged classical information-processing models by observing that some of the players' actions cannot be explained by such models and suggested that those actions were taken to extract information or to speed up mental computations. However, they stop short of offering an updated model and instead just advocate the need for rethinking why people take certain actions. In short, all they've done is identify a type of action that cannot be described as pragmatic and offer one possible explanation as to why such actions are taken.

Klemmer, Hartmann, and Takayama take these ideas further by taking more theories and examples and arriving at a comparison of the affordances offered by action and physical representation vs. thought and mental representation, and how they play off of each other. They take these comparisons and make them concrete by discussing their implications and impacts on interaction design. Over the course of the paper, they seem to draw an interesting analogy between cognition vs. action and digital representation vs. physical representation. It suggests that a lot can be learned about designing digital interfaces to augment or replace "analog" alternatives by taking a closer look at the interplay between cognition and action. It suggests that designing an interaction involves something deeper than matching or accounting for a user's mental model of the task, that it also involves accounting for how the physical representation becomes an extension of the user's mental model.


Manas Mittal - 10/24/2011 8:54:49

In the Klemmer et al. paper, the authors give an example of the tangible illuminating workbench as an example of 'learning by doing'. It seems that people are, in this case, learning about models of scientific thinking (i.e., angle of incidence = angle of reflection, light travels in straight lines) rather than observing physical phenomena (i.e., the bigger pale is twice as wide, and contains four times as much sand). Making a distinction lets one think in terms of models and experiments.

Thinking through prototyping Another key aspect of prototyping is to use it as a communication tool to concretely communicate our idea. Also, When the authors talk of 'knowing to ride a bike', knowing in this context is used as a proxy for being able to enunciate/vocalize something.

The Tetris paper is super interesting. Tetris has been (recently) shown to be a NP-Complete problem. I wonder if that has a role in determining the usefulness of physical manipulation in determining the right 'fit'.


Rohan Nagesh - 10/24/2011 9:00:21

These two papers concern themselves with comparing and contrasting pragmatic with epistemic actions. Pragmatic actions help towards achieving some goal whereas epistemic actions help us compute some computation that later saves time, but at the moment doesn't seem to directly be helping towards the goal.

The main gist of the first paper is that experts actually make more rotations in a game of Tetris (especially on the more complicated pieces) than beginners do. This indicates that the human cognitive cycle is not just about pragmatic actions but also heavily composed of these intricate little epistemic actions which help us visualize the world around us and later save time in our mental computations.

The second paper also touches on richer interaction designs. Thinking through doing or more tangible representations like in Tic-Tac-Toe enable humans to really feel and predict what the results of their actions will be. It mitigates risk, another concept the paper describes wherein one doesn't know the outcomes or risk associated with a physical social interaction and the fact that digital media mitigates this risk.


Jason Toy - 10/24/2011 9:01:04

On Distinguishing Epistemic from Pragmatic Action

This paper is about epistemic actions, a type of action that does not get the user closer to the end state like pragmatic actions do, but rather change the world in order to make problem solving easier. A study on the game of tetris is used to show that players make moves that make little sense under the current model, which is only based on pragmatic actions.

The authors propose a new framework where not all actions are directly used to solve a problem: by introducing epistemic actions which make problem solving easier. In the game of tetris, this includes actions such as the player rotating a block before it fully comes into view. This does not quite fall under the category of a pragmatic action because rotations may cause a false start (you have overshot the optimal placement position by rotating too far). However, it may serve as an epistemic action as you may be able to determine the type of block by process of elimination, before you are able to see the block fully on the screen. This addition to problem solving may influence future software such as photoshop, where developers could consider tools that help the user think about the problem rather than just accomplish the goal.

The paper does a good job considering counterpoints to its arguments, such as adhoc attention and motor control. One problem with the paper however is how its discussion on tetris will generalize to problem solving in general. The entire paper takes examples from the game of tetris to prove the existence of epistemic actions. Other examples are from other games such as chess: viewing a chessboard from different angles is considered an epistemic action as it doesn't directly help a player make a move. The paper does not discuss how games are related to problem solving in real world applications like software programming. Knowing that we can rotate a block to gain early knowledge of its type might be helpful in programming a tetris AI, but how does this help us in another problem, without having to resort to observing human actions and mimicking them?

How Bodies Matter

"How Bodies Matter" is about how physical bodies play a central role in shaping human experiences in the world.

This paper provides a framework of five themes which describe the importance of physical action as part of our conception of the world around us. Many of these themes are related to real world practices. For example, one of the themes talks about situated learning. This is true in the practice of pair programming. In pair programming, one programmer can learn from another based on suggestions and tips from the other's experience. The idea that visibility facilities coordination and that you cannot ignore someone walking into your office like you could ignore an email and that distance may affect social interactions such as trust and truthfulness are ideas that were discussed in a previous paper: "Beyond Being There", about forms of communication that did not require users to interact by being physically next to each other.

The paper does a good job bring up points which are grounded in various experiments and products seen by the authors. The general theme of "we know more we can tell" is definitely visible throughout the paper. However I disliked the structure of the paper as a brief overview of all five themes. Because it was so brief, no single theme had an in depth study that explored possibilities deeper, each had a brief sentence or two about its uses, if anything. In addition it was sometimes hard to see how some themes related to others. Is risk related to visibility, to actions that your hands make? Should a distinction have been made here? For example, should an HCI researcher who is developing joysticks care about how the device he is creating helps your ability to collaborate with others?


Ali Sinan Koksal - 10/24/2011 9:04:45

"On Distinguishing Pragmatic from Epistemic Action" draws attention on distinguishing between pragmatic actions which visibly help one move towards a goal state, and epistemic actions, which are performed to gain a better understanding of the task in hand. Models that ignore this second category of actions and that do not consider that external actions can effectively help in improving internal state cannot explain some observations, such as rotations and translations performed by advanced Tetris players and that do not aim to execute a task that is fully planned.

"How Bodies Matter" gives us an interesting perspective on how to design interfaces and evaluate them, revisiting a lot of the work that we have studied so far in class, including the other assignment for this week. Among the concepts that were discussed, I was particularly interested in seeing the importance of differentiation of interfaces, and the impact of visibility of interaction (how visible actions bring a sense of commitment that can help in focusing on work).

The choice of Tetris as a game to study for exploring epistemic actions was brilliant in that the consequence of the actions performed could be clearly categorized as helping move physically towards the goal or not. This insight on epistemic actions can help in designing new interfaces where this kind of actions are eased by the interface (or even explored and presented to the user by the computer).