Design Methods

From CS260 Fall 2011
Jump to: navigation, search

Bjoern's Slides

media:cs260-09-designmethods.pdf

Extra Materials

Discussant's Materials

Reading Responses

Hanzhong (Ayden) Ye - 9/26/2011 0:20:11

Reading Response for -Dilemmas in a General Theory of Planning. Horst W J Rittel, Melvin M Webber, Policy Sciences, Jun., 1973, vol. 4, no. 2, p. 155-169. -Getting the Right Design and the Design Right: Testing Many Is Better Than One, Maryam Tohidi, William Buxton, Ronald Baecker, Abigail Sellen CHI 2006: ACM Conference on Human Factors in Computing Systems, pp. 1243 - 1252.

The two papers for Wednesday lecture cover several aspects of design methods used in human-computer interaction area. The first paper, although being an discussion on a topic in policy science, lends me deep insight on the dilemmas in a general theory of planning. The second paper, which draws conclusion from a real experiment on interface evaluation, shows the power of multiple design in the process of interface evaluation, compared with single design.

The first article, which holds an pessimistic attitude towards the procedure of planning, points out that the search for scientific bases for confronting problems of social policy is bound to fail. The authors believe such planning problems have a ‘wicked’ nature, which can be explained in many aspects. These problems are wicked because they lack a definitive formulation and have no stopping rule, and we cannot find a true-or-false solution, while no immediate or ultimate test for such problems. The authors also believe that every planning problem is essentially unique, and can be considered as a symptom of another one. Although to some extent the authors are right about their points of view, I still think their attitude is too pessimistic and should be more constructive. While it is true that the planning process is a difficult one and not easy to be analyzed as traditional scientific research objects, I believe it is more important for us to use methodology as a powerful tool to overcome it.

The second paper, which is more close to our topic in human-computer interaction, launch a research to investigate the difference of design evaluation in terms of multiple/single designs in testing process. The UI evaluation for HCCS shows great difference in many aspects when different number of designs are given to the participants. The test is very scientific and the data warrants some credential conclusion, such as the effects of faint praise, increased criticism and increased participation of construction for new interaction prototype. The paper also shows the effectiveness of paper prototype in the process of UI design. It is an obvious conclusion from the paper that we should present more models of interface at an early phase of design and collect widely from potential audience, rather than doing iteration from a single model and generate some result that could be wrong from the beginning.

-By Ayden (Sep 26th, 2011)


Derrick Coetzee - 9/27/2011 14:37:54

This week's readings focused on the role of prototypes in design and what makes them successful. Houde and Hill's "What do Prototypes Prototype?", from Apple in 2004, effectively conveyed the idea that a prototype should be described primarily in terms of what aspects of the design it is intended to explore. Particularly instructive were the brick and pizza-box examples, prototypes which were extremely cheap to construct, yet effectively explored a critical aspect of the design (size, shape, and weight). It was also successful in debunking the idea that prototypes are built sequentially and with greater refinement as design progresses, e.g. in the sound browser example.

On the other hand, the work was also self-limiting in that it used a fixed model with three different aspects that a prototype could explore. Although they allowed mixes of the three, they did not explore or even mention the possibility of other aspects (e.g. a marketing or a public policy directed prototype). The work also grew rapidly repetitive, having made its main point in the first page: examples are illustrative, but reiteration of the main idea is not.

Tohidi et al's "Testing Many Is Better Than One", from 2006, was a focused statistical study that set out to test how user feedback changed when multiple alternative user interface designs were presented. Because of its detailed focus, the work was limited in scope, focusing on only one application with three low-fidelity look-and-feel prototypes, and with a relatively small number of test users (12 per test group).

Its heavy reliance on statistical significance testing at the p=0.05 level for well over 20 tests is highly problematic: in such a large group, at least one test is likely to be significant by chance alone. Moreover there is no statistical measure of effect sizes, the size of the difference between groups, other than the raw data, which may be misleading due to small group size. The p=0.05 was treated as a "magic number" in discussion, supposing that a p=0.064 indicated no difference, while a p=0.045 indicated a significant difference, although these probabilities are actually quite similar. Even their strongest result, that significantly less positive comments were received, had relatively high p values of 0.045 and 0.025. In all, I think the conclusions are tainted and reduced to speculation by this poor use of statistics.


Amanda Ren - 9/27/2011 14:51:24

This prorotyping paper aims to describle the problems with the current terms being used to describe prototypes and propses that we focus on the purpose of the prototype to make decisions on the types of prototypes to build.

This paper is important because it shows that prototypes are modeled after three questions: implementation, look and feel, and role. By making separate prototypes, it is easier to address specific design questions, leading to a better overall design. I thought it was interesting how the extent to which role prototypes were embellished depeneded on the audience. Apple made a highly produced video when demonstrating a vision to the public, but paper storyboards work very well among the design team. As for look and feel prototypes, the designer could get valuable information from something that takes no time at all to build. Even though implementation prototypes take more effort to build, they have the benefit of being used in the real system. The hard part comes in building an overall prototype to use to demonstrate all three models.

The Tohidi paper discusses how using three functionally equivalent designs allows for better feedback from user testing than just one design.

This paper is important because it points out that just using one design in user tests may not bring you the best feedback, and offers another solution. It does not surprise me that positive comments decreased since what the paper said about people feeling more uncomfortable and feeling the need to impress will affect them. Although the researchers expected the participants to come up with more creative ideas when shown more designs, this did not happen. I think one reason is because the participants are given more than one design, so maybe they figured the researchers already went through many other designs so they were not motivated to come up with more ideas. Although for the single design experiment, I'm wondering if the researchers should have stated that the design would be their final design. It implies that much testing and decision making has already been put into it.


Steve Rubin - 9/27/2011 16:47:23

Both of the papers "What do prototypes prototype?" and "Getting the right design and the design right" emphasized the importance of certain practices is design work. The upshot of the first paper is that designers can do distributed design work by considering prototypes that cover only a specific part of the design spectrum, from role to look-and-feel to implementation. By considering these separately, designers can make more informed decisions when ultimately building an integrated prototype. The second paper suggests that usability studies benefit greatly from the existence of multiple designs.

While the examples listed in the first example were a bit superfluous, the key idea of the design space (Figure 1, the triangle) is a great concept. As a computer scientist, I'm tempted to get caught up in the prototype-as-implementation idea. To be fair, I've never actually built a prototype in the iterative development environment that would happen at a company. At times, design may seem to be a single-person task. This paper assures that it isn't, and that there are principled ways to avoid this.

In light of the first paper, the second paper was a provocative read. The authors performed _one_ study and proceeded to make sweeping declarations about usability studies. Even though their results were limited to a single study, I do believe that we would get better results from a usability study by testing multiple designs. My real qualm with this study is that the paper's conclusions are based on a study that did not fully explore the prototyping model given by the first paper. Paper prototyping completely neglects the "implementation" side of things, which can have a serious effect on usability. Paper is also a bit lacking on the "look and feel" side of things. So, while I intuitively believe the paper's conclusions, I think that more care should be taken to see how other kinds of prototyping affect usability studies (and, perhaps, participatory design).

I generally enjoyed these papers because even though I had no prior experience in the area, I think I now have a good grasp of the design considerations necessary for creating successful prototypes.


Valkyrie Savage - 9/27/2011 16:48:10

Main idea:

Prototypes are an important step on the path from idea to product, but their properties range across kaleidoscopes of possibilities. There are problems for which there are no prototypes! And the approach for getting a prototype’s ideal form can be as difficult in and of itself.

Reactions:

Beginning with “What do Prototypes Prototype?” I felt that their model was reasonable, and I was surprised that design teams work in parallel as they do. I haven’t really had many opportunities to work in larger teams, and certainly not large enough teams to necessitate having multiple prototypes in the pipe at once. I have to say that I felt the approach was somewhat unscientific (their arbitrary placements of dots across the board seemed... arbitrary), but also that that was perhaps appropriate for the situation, since it’s sort of the idea of a prototype that it isn’t precisely what you’re looking for, anyway.

“Getting the Right Design and the Design Right” was a paper that didn’t hold that many surprises for me... it intuitively seems as though people should feel more okay about criticizing something when they are given a choice. I did wonder about the fact that no one seemed willing to make the step to designer: though they listed a selection of the majors that their subjects hailed from, it seemed strange that they didn’t mention or seem to ask anything related to design experience from the subjects. Presumably we have all been designers at some point in our lives, e.g. when we created lives for our dolls, missions for our soldiers, or games for playing with our friends. It’s a wonder that we’ve all lost it. I’m also curious about how such a result set might change if the experiment were repeated on members of e.g. the maker community.

I really liked the third paper. It approached the problem of interaction design from an entirely different perspective, and I appreciated reading something which was more of a sociological treatise on person-problems as opposed to system-problems. Obviously people are half the core of our industry! Also, the paper’s historical groundings made it interesting for other reasons. It gave me a lot of thoughts to chew on, and sadly I don’t think I can synthesize them as well as I’d like in this short window. Anyway, I’ll take a stab at it: HCI people are not just computer people. We have to appreciate many of the things in this paper about how the addition of people often makes problems unsolvable, and all we can do is move towards better solutions iteratively. This discussion makes me think back to the groupware paper: there was no perfect solution for a company, or even for a small group, since everyone brings different expectations to the table. It really requires a measure of, tritely, thinking outside the box to arrive at a solution which is, if not optimal, at least acceptable, and it takes open-mindedness (at least, it seems open-minded when a computer scientist does it) to recognize that not all data comes from computers, or even from testing, but instead from users. But, tying back to the other paper, users can’t solve their own problems. It seems that we’re a sort of crippled society in that way...


Laura Devendorf - 9/27/2011 18:07:13

What do Prototypes Prototype discussed a number of concepts to consider in order to develop and utilize prototypes in designing an interactive application. Getting the Right Design and the Design Right discussed how participants in a usability may provide more acurate ratings and constructive feedback when presented with multiple designs rather than one.

I found the paper to be very helpful and I like how they viewed the prototype as a construction that takes multiple parties into account. I was particularly interested in their discussion of the linkage between the literacy of the audience, in terms of the project, and the resolution required of the prototype. I was a bit confused about their taxonomy and how and why they grouped particular projects into particular regions. I found the "look and feel" prototypes to be awfully similar to the "role" prototypes. I am not sure if the language they are using is obscuring their goal. To me, the role of an object in one's life sounds a bit more involved in interaction than the term "look and feel." I also would have liked to seen a discussion about how to avoid research tunnel vision. By this I mean, the fact that you're so involved with the project makes it difficult to see the product as an outsider. How might prototypes be able to encourage an outside perspective? I would also liked to have seen more of a discussion about creating just the right amount of abstraction in prototypes in order to encourage new designs and thoughts and how to accomplish that. Their suggestion seemed to be, try lots of things, but I would have preferred a more methodical approach. Once one person sees and image of an idea, it becomes more difficult to think beyond the image.

The paper presented evidence to support their claims relating to testing one prototype versus another. The cited the fact that users may "feel badly" if they criticize one single design and "less badly" if they are presented with multiple since it signals to them that the developers haven't decided on one yet and put work into a single interface yet. While I found the results compelling, I'm interested in two things. First, what does the psychology community say about this topic? I would assume someone in behavioral psychology may have already looked into this phenomenon. Second, the scenarios they described were tested entirely on paper prototypes. Clearly, not a whole lot of work went into developing them. If they were to compare three fully developed interfaces, each of which implied lots of time to develop, would they be more sensitive to giving negative feedback?


Yin-Chia Yeh - 9/27/2011 20:52:08

Two papers related to prototype testing are read. The “What the prototypes prototype?” paper proposes a new terminology to describe/classify prototypes by their purpose, instead of using current popular terms of material, fidelity or resolution. It also broadens the definition of prototype not to limit it to a narrow range of something with form factor. The “Testing Many Is Better” paper claims that having multiple designs tested in a survey will yield to more (critical) information. However, it does magically making people proposing new ideas, so it is a means to identify problems instead of providing solutions.

In the “What prototypes prototype” paper, I like the way the authors explain the concept of three different purpose of prototype, first using a unique project with three different purposes, then giving examples of different categories. Another important thing noted is that other than purposes, the audience is critical to decide what kind of prototype you want to build, and beyond that, it is also important to make your audience understand the scope of design questions this prototype aims at. I’ve been writing software tools and I found it is often misleading just describing the software you are going to build without giving a sample input or sample user interface, and it is so true that you need to provide different level of details to different audience. As for building multiple prototypes, I feel a little bit unconvinced because I am still not familiar how designers can really do “Role”, “Look and Feel”, and “Implementation” stuffs in parallel. It will be nice if they can provide more concrete example.

In the “Testing Many Is Better” paper, I am most interested to the experiment design part because instead of doing some math or algorithm, they aims at testing people, which is uncommon in my own experience. One question I have is while they converting the recorded video into classified statements. How do they prevent human bias? Is there any golden rule to follow in this kind of experiments? The other question is that though I agree their hypothesis that people are trying not to be negative and hurt others, is there any other alternative explanations that could affect the way they interpret the experimental results? Anyway, from the practical point of view, as long as it can solicit more critics on single design, testing multiple designs is a good idea.


Viraj Kulkarni - 9/27/2011 21:39:52

The first paper, 'What do prototypes prototype', is about directing the thinking behind what prototypes are while the second paper, 'Getting the Right Design and the Design Right: Testing Many Is Better Than One' is about conducting usability testing by providing multiple designs rather than a single design.

'What do prototypes prototype' suggests a change in the way designers think about prototypes. There is no traditional definition to the word prototype and it is used very loosely. This paper attempts to formulate a common terminology that designers can use to talk about prototypes and the object they are prototyping (which they call the artifact). They also present a model which designers can use to classify the approach they are using in developing the prototype. The paper then goes on to list comprehensive examples for each of the categories of prototypes it lists.

'Getting the Right Design and the Design Right: Testing Many Is Better Than One' states that when subjects of a user study are presented with a single design, they give the design a higher rating than what they would have rated it if they were presented with multiple designs. The paper states that if users are shown only one design, they are hesitant to critisize it lest it disappoints the people conducting the test. This makes them rate the design higher than what they would really rate it. The way around this is to ask them to rate multiple designs to allow them to critisize without being negative. The paper investigates the impact on ratings by simultaneously evaluating three designs instead of one and presents the results and analysis of this investigation.


Yun Jin - 9/28/2011 1:00:31

Getting the Right Design and the Design Right: Testing Many Is Better Than One: In this paper, it compares the results of presenting a single design and presenting the same design in a group of three. As a result, users would give higher ratings and more reluctant to criticize upon the former one. Moreover, it also demonstrates that identifying problems are more useful than providing solutions based on the usability testing by itself. From this paper, I find that multiple alternatives are only to be considered in the beginning of the process and then designers would focus on one chosen from the many. Thus, testing many at first is more helpful for designers to explore in the further step. Moreover, it is the norm that people getting together simultaneously but independently would help the whole progress of the project. Finally, this paper also has some implications for participatory design. It is important to adopt approaches beyond standard usability testing techniques for generating redesign.

What do prototypes prototype? This paper proposed a change in the language used by designers to think and talk about prototypes of interactive artifacts. And it introduced a model and illustrated it with some initial examples of prototypes from real projects. Designers could use this mode to benefit for prototyping. This paper described that there are some suggestions for designers. They are defining “prototype” broadly, building multiple prototypes, knowing audience, knowing the prototype, and preparing audience. The purpose of the prototype is what is the prototypes, and with such clear purpose, designers could use prototypes to think and communicate about design better.


Allie - 9/28/2011 2:00:30

In "What Do Prototypes Prototype", Houde and Hill discuss the attributes of prototypes, as well as the various challenges they face in a HCI context. The 3 main audiences that review prototypes are: 1) users of the artifact being designed; 2) their design teams; 3) supporting organizations they work in. Two related terms that are often used to describe prototypes are "resolution" and "fidelity", which mean "amount of detail", and fidelity means "closeness to the eventual design". The triangular model of "role", "look and feel", and "implementation" is used to describe the functionality of the artifact; its sensory experience; and its function, respectively. We introduce these vocabularies in order to evaluate various prototypes by where they fall under the triangle. Applications evaluated include, if not encompass the 3D space-planning application, a portable notebook computer, an operating system user interface, the knowledge navigator, and the integrated communicator using cited criterion.

Designers often feel they invest quite a bit in making the prototype but was only able to support a narrow role. In conclusion, the paper makes 4 suggestions for changes in designing prototypes: 1) define "prototype" broadly 2) build multiple prototypes 3) know your audience 4) know our prototype; prepare your audience.

The Houde/Hill paper is very abstract, and its adopted vocabulary very much like a business case study. It is useful in introducing some interesting paradigms for evaluating different types of prototype design. The paper concludes that by focusing on the purpose of the prototype, we can build better ones. I tend to disagree with this statement, since sometimes the prototype that do not set out with an intended audience may find utility elsewhere by chance.

In "Getting the Right Design and the Design Right: Testing Many is Better Than One", Tohidi, et al propose that usability by itself is a way to identify problems, not solutions. The paper asks: "can exposing users to multiple design alternatives also help us in getting the right design"? The research conducted the same test on 3 alternate designs, where different groups saw one of the prototypes in isolation, and then were presented with all 3 prototypes. The system is a House Climate Control System that regulates the temperature inside a house by controlling its heating and cooling mechanisms. The designs were 1) of circular variation 2) tabular 3) linear. Participants were asked to express their likes/dislikes of the designs, and provide suggestions for improvement. The findings present that Circular and Linear designs were preferred over Tabular. Results were different when the participant was presented with all designs rather than a single design, for when participants saw the same design in the context of its alternatives, the balance of positive/negative comments changed. For the Linear Prototype, the number of negative comments was higher when it was seen in a group of 3 than in isolation; and being exposed next to its alternatives removed participants' inhibitions about making negative comments.

As expected, usability tests are vulnerable to overly positive results. Tohidi, et al challenges traditional thinking, as their research finds that low-cost techniques, such as paper-prototyping allows multiple alternatives to be explored beyond the initial ideation phase. Overall, the results seem to provide some useful insight into the different scenario of multiple alternatives seen by individuals or in group settings; but am not certain the conclusions drawn from these results are necessarily on target.


Ali Sinan Koksal - 9/28/2011 2:22:40

This week's first paper, "What do Prototypes Prototype?", establishes a model for a better understanding of the function of prototypes, and how they should be conceived and presented in order to convey ideas in three distinct dimensions:

 - "role": the focus here is on presenting new functionalities, and which role the artifact may play in everyday life.
 - "look and feel": here, the goal is to show concretely novel ways in which a known functionality can be expressed
 - "implementation": in this case, prototypes should focus on technical aspects, to assess implementability of envisioned designs.

Each prototype may explore one or more of these dimensions. "Integrated prototypes" address each of these dimensions and are typically produced at more advanced phases of design.

I think the model of prototypes proposed in this paper can be very helpful in organizing one's limited amount of resources such that no effort is wasted on developing high-resolution prototypes that address all the above dimensions in each prototype that is considered. A very interesting, extreme example is the pizza box prototype of a computer for architects. This example shows how very simple prototypes can give insight in one particular aspect of design.

I am however not very convinced that prototypes in each of these dimensions can be developed in a fully parallelized fashion. One typically thinks of use cases/scenarios in order to define clearly which is the problem in hand and how it could be approached. I also think that it might be quite difficult to decouple "role" and "look and feel" fully. Indeed, all of the examples that explored "look and feel" were also related to the "role" dimension.

The paper by Tohidi et al. presents the benefits of considering multiple designs in usability tests, instead of just one. When users are presented with multiple choices, they are less prone to give high ratings that do not actually reflect their judgment about the product. This approach also helps in receiving more constructive criticism, offering a setting which lets the users articulate better their opinions about designs.

The general lesson of being "open-minded" and considering a number of alternative designs and not committing to just one seems very valuable to me. One should not be reluctant in research to try out different possibilities to be able to compare those. As the authors point out, rapid, cheap prototyping of artifacts helps both in refining a single design and choosing among many, while it is still possible to not have to commit to one single approach. There is certainly value in taking lessons from the well-established methods that designers, architects etc. have been using for a long time.


Galen Panger - 9/28/2011 2:45:51

The CHI piece this week highlights a common and kind of alarming problem seen in many CHI papers. The authors build a pretty cool new system, then find 6 users to evaluate it and—surprise!—everyone thinks it’s pretty cool. They then conclude that the technology is promising and deserving of further exploration. Great. Has science been done, though? Or is this more like a glorified homebrew computer club?

Unfortunately, the CHI piece highlights this problem with its own methodological issues. Their results have low or no statistical significance almost across the board—don’t you think they could have included the other half of the video footage to boost significance? They also failed to implement a pseudo “double-blind” design everywhere, most especially where it’s most crucial: in interviews. If the interviewers know the hypothesis, that’s a problem.

Also, I thought the discarding of the 11-question scaled was a bit hasty. They reasoned that the average was correlated with the overall rating, so they chucked the more granular data out. Hmmm... are they sure there aren’t any interesting subtleties in the data? Maybe there aren’t any—and given the overall disappointment of this part of the study, perhaps it was best for the authors not to spend so much time on those results.

That said, I do think their larger point is helpful, warts and all, though it’s fairly common sense. Never hurts to re-emphasize common sense, I guess, especially if an entire forum like CHI seems to need that kind of reminder. But, come to think of it, perhaps the reminder is a bit empty given the piece’s lack of implementation recommendations. Most CHI-published projects don’t involve mundane thermostat design; most are much more complex explorations of role/implementation/look and feel. Where and how could the exploration of multiple alternatives be most helpful in CHI-type projects, not to mention cost effective? The authors don’t seem to think about that.

Speaking of the Houde/Hill piece, I was intrigued by one thing—their emphasis on the role of new functionality in people’s lives. I really liked how that was framed, and I think that gets at much deeper questions about the what and why of a new artifact than questions of look and feel and implementation. How will people use this and how will this affect their lives? That’s a super important question for any designer. And I don’t think enough attention is paid to that when evaluating prototypes. Consider me a “yes” vote for more storyboarding and in-situ observation.


Apoorva Sachdev - 9/28/2011 3:14:41

Reading responses for the week:- This week’s readings were about design methods -- in particular proto-typing and performing user studies. In the paper “What do Prototypes prototype Charles Hill and Stephanie Houde, describe what a prototype is and how most prototypes can be classified into 4 categories: - role, look and feel, implementation and integration which each focus on a certain aspect of the idea and are used to test and improve that specific aspect, except in the case of integration where the prototype is relatively close to the final product.

I thought this paper was very clear in distinguishing the different prototypes and their purposes. I agree with them that sometimes users are confused while interacting with the prototype on what has been accomplished and what is actually wizard of Oz and this can greatly affect the amount of substantial input they can provide to the study. I also feel that sometimes the wizard of Oz approach is not very effective if one has not researched the technically side well to make sure that it can be implemented eventually. Additionally, going directly to implementation may also be limiting since instead of modifying the technology to suit the interface, we might modify the interface more to suit the implemented technology.

The other paper did an analysis of how people respond to prototypes present in user studies if they are presented with just one sample and are asked to criticize it versus when they are given multiple samples. I feel most of the papers finding very pretty common sense, since people inherently tend to be nicer if there is only one sample presented because they don’t want to hurt the feelings of the interviewer. Although the paper was able to support this quantitatively and this is useful. However, I feel the paper didn’t account for things like the user’s background (have they been exposed to similar ideas in the field, interacted with contemporary solutions if there are any) as this would influence the way they criticize and provide suggestions. Also, the way the study is conducted affects the way people respond. For instance, people tend to be more truthful if anonymity is exercised. More often unless people are specifically asked to brainstorm for ideas, they would not suggest radical changes to the current approach.


Peggy Chi - 9/28/2011 3:21:09

How do we efficiently test design ideas before putting all the efforts on implementing computer systems? Houde & Hill (1997) from Apple proposed a model that included role, look and feel, implementation, and integration to help designers answer important questions. Several case studies and examples were presented to support this multi-dimensional space. Tohidi et al. (2006) argued that multiple designs should be provided to help users decide and criticize when conducting a study. This might allow users to compare between the alternatives and identify the problems.

I really like the ideas of exploring design space before implementation. I'm especially a fan of paper prototyping and storyboards. By putting designers and users in a context without technology, they could focus on the design concepts without getting into too many tech details. However, I'm not sure to what degree we should consider prototyping first. When a system is relatively complicated, should we simplify or should we build a basic system directly? What are the constrains and boundaries of prototyping? To a specific domain of problems, can we apply design patterns to avoid the efforts on building prototypes?

Moreover, when presenting multiple designs for participants to compare, do we need to consider the "stereotypes" of interfaces? For example, to redesign a new ATM system, if the users see a thoroughly new system with different layout and interaction styles that they are not used to, can they accept the new ideas, or most of people prone to criticize new things and thus bias the results? This might lead to micro-improvement instead of jumping out of the box.

One more thing, people said the current Apple doesn't believe in usability and user studies. It is interesting to see the paper from the company that shared the insights of prototypes way back to 1997. Is it because there is unavoidable problems lying beneath this methodology so that they don't adopt usability test anymore? Or usability is another thing that serves a different purpose? (before vs. after design)


Cheng Lu - 9/28/2011 4:02:58

The paper “Getting the Right Design and the Design Right” presents a study comparing usability testing of a single interface versus three functionally equivalent but stylistically distinct designs. The author found that when presented with a single design, users give significantly higher ratings and were more reluctant to criticize than when presented with the same design in a group of three. The results imply that by presenting users with alternative design solutions, subjective ratings are less prone to inflation and give rise to more and stronger criticisms when appropriate. Contrary to traditional expectations, the experiment results also suggest that usability testing by itself, even when multiple designs are presented, is not an effective vehicle for soliciting constructive suggestions about how to improve the design from end users. It is a means to identify problems, not provide solutions. The most important contribution of this research is its implications regarding usability engineering. One of the standard texts teaches that multiple alternatives are to be considered only at the very beginning of the process. From then on, one is taught to work through successive iterations of the one design chosen from the many. What the paper findings suggest is that low-cost techniques, such as paperprototyping enable multiple alternatives to be explored beyond the initial ideation phase. More to the point, they suggest that doing so can enable us to obtain a less inflated subjective appraisal of the designs, as well as obtain more critical comments that help identify problems.


Alex Chung - 9/28/2011 4:06:36

What Do Prototypes Prototype?

Summary: Different kinds of prototype can used to answer different design questions and to speak to various types of audiences. Instead of going for the designated prototyping tool, one should consider the expectation of the audience and the purpose of the prototype. “With a clear purpose for each prototype, we can better use prototypes to think and communicate about design.”

Positive: I love how the author describes prototypes as tools to communicate with the audiences about the design concept and direction. Generally, less expert audiences require a more detailed prototype (high resolution) to explicitly illustrate the point. On the other hand, more knowledgeable audiences can easily interpret abstract prototypes. Both audiences are equally important in answering different set of design questions and there is a trade off on how much effort required visualizing the design.

Positive: Another great point made by the author is the importance of stating the intention of the prototype. It helps both audience and designer to set the framework of what question the prototype is intend to answer. The digital movie editor example perfectly illustrates that misunderstanding could lead to ineffective discussion. In my experience, project managers without technical development background often take the prototype designs at the face value. It would then lead to an unrealistic project timeline or a badly designed product. The iteration process is vital to integrate lessons from each dimension of prototyping including role, look and feel, and implementation.

Negative: The chapter gives me an impression that any product/service design can be split up into three parts and each dimension has only one question requiring only one prototype. I believe that some of the prototyping models such as sketch, storyboard, and physical mockup can be combined to find the product’s role in user’s life.

Negative: The chapter failed to address how to compromise conflicting values from different design dimension. What if the answers from “look and feel” and “implementation” could not be integrated at all? The author did not discuss who is the decision maker to find judgment call. In the case of digital movie editor, the author discusses how the communication broke down because the audience’s frame of mind wasn’t correct. Yet sometimes, there is power dynamics and the boss just won’t focus on one aspect of the prototype discussion.

Comment: While I understand the value of migrating successful implementation designs to a new integrated prototype, it is difficult for most engineers to treat working implementation prototypes as disposable.

Getting the Right Design and the Design Right: Testing Many is Better Than One

Summary: Designers in HCI should practice generating multiple alternatives for each project. One reason is that multiple designs allow end users to compare and contrast in order to provide constructive feedbacks on the designs. Thus a low cost, fast turnaround prototype technique such as paper prototypes is so valuable in early exploration of design alternatives.

Positive: The experiment design was detailed and it clearly showed user feedbacks changed as the number of alternatives differed. And there is a trend where the number of negative feedbacks increased along with the number of alternatives available. The experiment was well designed to provide some interesting answers to three unique hypotheses.

Positive: The surprising result of not getting more suggestions for design improvement in the Multiple condition. And I agree with the author’s assertion that increased experience from exposure to three different designs would not suddenly boost creativity in the participants. On the other hand, increased experience has provided the participants more confidence to give negative comments regarding the designs.

Negative: I wish that the paper had done more qualitative interviews with the participants. Many of the author’s conclusions on participants’ behaviors include language of uncertainty such as “it seems” or “we believe”. Was it possible to confirm their conclusions by asking the participants directly afterward?

Negative: Maybe because the small sample size. The number difference is subtle but not significant. However, it is very difficult to calculate scores that are based on subjective opinions. People have different ways for measuring how a system is “friendly” or not. Also, the use of usability testing mainly for error detection should based on the expertise of the participants. I thought the author generalized the conclusion without providing enough information about the level of expertise of participants in this study.


Suryaveer Singh Lodha - 9/28/2011 4:28:14

I wish to pass on this reading assignment.


Hong Wu - 9/28/2011 7:25:53

Main Idea:

“What do prototypes prototype?” shows the principles to build prototype.

Interpretation: “What do prototypes prototype?” focuses on the purpose of the prototype. The paper argues that the success of the prototype lies on clarifying what aspects the prototype can do or cannot. The author divides the aspects of the prototype into three regions ------ “roles”, “look and feel” and “implementation”. The function of “role” prototype is to investigate what an artifact could do for a user. The function of “role” prototype is to investigate what an artifact could do for a user. The function of “look and feel” prototype is to show the appearance of an artifact and to concrete the experience of an artifact. “Implementation” prototype mainly tries to answer whether a technique is doable or not. The combination of the prototype has the property of all of the three kinds.

Besides the three kinds of the prototypes mentioned in the paper, we can also design and examine a prototype from scale of the prototype and other aspects. Doing prototype is a necessary process of the development of software. The concept of the paper is very useful for my future career.


Vinson Chuong - 9/28/2011 8:30:50

In "What do Prototypes Prototype", Houde and Hill address common issues and misconceptions about the concept of prototyping and present a new conceptual model that they hope help to clarify and fix some of them. In "Getting the Right Design and the Design Right", Tohidi, Buxton, Baecker, and Sellen study the effect of presenting multiple prototypes (instead of a single one) on the behavior of usability study participants.

When building a prototype, Houde and Hill stress the need to define its purpose---what kind of information is being gathered, what questions are being asked, etc.---and to account for the knowledge and preconceptions of the target audience. An interactive system involves many interrelated components and features whose importance vary between different types of people. Different kinds of prototypes emphasize different components and features. A common occurance when presenting a high-fidelity prototype, as opposed to a low-fidelity paper prototype, is that the audience tends to focus more on the look and feel of the prototype, which may skew or confound other factors that designers are trying to assess. Houde and Hill argue that pinning down the purpose of a prototype enables designers to determine what aspects of their projects they want to express, emphasize, and therefore, to assess, and that such a concise prototype would be more easily understood by the audience.

In "Getting the Right Design and the Design Right", the researchers saw a a decrease in the number of positive comments, an increase in the number of negative comments, and no change in the number of suggestions when comparing the behavior of user study participants presented with three vs. one prototypes. The purpose of their prototypes, in context of their "usability testing" and not the comparison between number of prototypes, was not clear. Had these been actual usability tests, what types of information would they have tried to gather (other than the subjects' opinions on usability)? Would number of prototypes presented have an effect on these types of information? I feel that the purpose of the prototypes confounded their results, as evidenced by their surprise that showing multiple prototypes did not cause more suggestions. What kind of suggestions were they looking for? Were these prototypes built to elicit design such suggestions? Although their results were clear, that increasing the number of prototypes had a psychological effect on study participants, I doubt the usefulness of this result.

Houde and Hill's paper offers a useful model to compare and to think about prototypes. They bring up compelling issues that may be easier to fix in context of this model. However, they concede that building an effective prototype is more of an art than a science and that dealing with the preconceptions of the audience can be very difficult.


Manas Mittal - 9/28/2011 8:58:14

The prototyping paper tries to describe the objectives and mechanisms around building prototypes. It attempts to introduce a vocabulary (like that detailed in the triangle figure). The paper attempts to develop a taxonomy around prototypes.

Engineers tend to think of prototypes as a tool to decrease risk, often in context of feasability of implementing something, and learning about how to do something by doing it. I think this paper doesn't spend much time on that discussion, but its an important one. The whole field of systems is about building prototypes that teach us how to build solutions. At some level, all research is about building prototypes - we are trying to learn how to build, and what to build.

The second paper - Tohidi et al., has a serious problem - all the subjects were within a narrow age range, and in general, were very homogeneous. It is likely that older people are more direct, are not as affected by insecurities 'i am a negative person', etc. Based on this, I find that making any generalization based on their study is inappropriate.


Shiry Ginosar - 9/28/2011 8:58:15

These two papers discuss the usage of prototypes as a tools designers use to test and refine their artifact designs. The Houde and Hill paper discusses the process of choosing a prototype to use in a user study and presents a very reasonable claim that prototypes should be defined and used for a particular purpose and with a particular research question in mind. Furthermore, the purpose and research question are the most important things that should be communicated to the audience (rather than the tools used to create the prototype etc). To this end, the writers propose a model which allows one to converse about prototypes as explorations of one of three design questions - role, look and feel and implementation. This model strikes me as a very reasonable one and one that would serve its purpose well in allowing designers and audiences to align their frames of references before and during a user study.

The Tohidi et al paper tests the hypothesis that showing users more than one prototype to choose from and critique elicits honest opinions and criticisms from them and helps them not fall into the trap of being over positive about the design. Moreover, it makes the point that one should not expect the users to provide new design ideas or ideas for alternative designs as a followup on a user study.

Personally I was surprised that the authors were surprised about either of these questions. Both of these behaviors seem to follow a very common human communication protocol - people try and not hurt each others' feeling most of the time so it is only natural that they would not be negative about a single design one invested in. Moreover, novices in a field would rarely intervene and express ideas for improvement in a field other than their own as they would be worried that they do not have the tools to do so.


Donghyuk Jung - 9/28/2011 8:58:15

What do Prototypes Prototype?

The authors argued that it might be difficult or impossible to create prototypes of a whole design in the formative stages of a project because interactive systems are complex. So, they proposed that each prototype requires a clear purpose in order to use prototypes to think and communicate about design better. It is largely because: complex prototypes means that user experience is therefore complex; prototypes are not self-explanatory; every stakeholder including designers, users, engineers has different expectation of what a prototype is. Fore these reasons, the main goal of this article is to establish a model that describes any prototype (= any representation of a design idea, regardless of medium) in terms of the artifact (= interactive system) being designed, rather than the prototype’s incidental attributes. They represented a three-dimensional space (role, look and feel, and implementation), which corresponds to important aspects of the design of an interactive artifact.

As they mentioned in the article, there are various types of interactive systems as well as different level of complexity. I think that this article utilized a lot of examples of prototypes in order to make readers understand the model they proposed. In addition, the authors tired to cause a paradigm shift from a simple prototype as a preliminary model of something toward communicating audience with its specific purpose. All in all, they proposed a shift in attention to focus on questions about the design of the artifact itself: What role will it play in a users’ life? How should it look and feel? How should it be implemented? The model can be used by designers to divide any design problem into these three classes of questions. They will benefit from a different approach to prototyping.

Getting the Right Design and the Design Right: Testing Many Is Better Than One

In this paper, the authors conducted usability test between a single interface and three functionally equivalent with stylistically distinct designs and compared them. The underlying question in our research is “ Can exposing users to multiple design alternatives also help us in getting the right design?” In their experiment, they used paper prototypes with low-fidelity to investigate the impact of simultaneously evaluating three designs compared to just one during early usability testing. The basic assumption of this experiment is that users who are participating usability tests feel compelled to impress experimenters and to under report errors. However, being presented with multiple alternative designs may allow for a more accurate comparative evaluation.

As the result shows, I agree that presenting multiple alternatives enable usability testers to obtain a less inflated subjective appraisal of their designs, as well as obtain more critical comments that help identify problems. However, I doubt that paper-prototyping enable multiple alternatives to be explored beyond the initial ideation phase. In this experiment, they proposed a simple HCCS system and testers are all highly educated people so that they well recognized what the prototype is and its usage. My rationale behind this is from our first reading, which mentions about complex interactive systems not simple ones. If they used much more complex prototypes such as user interface for business applications or operating system, it might be impossible for experimenters to identify design problems with critical comments.


Rohan Nagesh - 9/28/2011 9:00:45

The first paper, "What do prototypes prototype," discusses an ideology change in the reasoning about prototypes--rather than focus on implementation details or tools used, etc. exclusively, the authors wants to focus on the purpose of a prototype and builds a model including role, look and feel, and implementation under which all prototypes fall into. The second paper "Getting the right design and design right" explores the possibility of stronger criticisms and better eventual designs when presenting user study subjects with a multiple designs rather than just one.

With regards to the first paper, I absolutely agree that a structured framework for thinking about prototypes would be hugely beneficial. Ultimately at the end of the day, we do think about "what does this prototype prototype" as our main question, and I very much agree that prototypes fall into the 3 categories the authors mentioned. However, I believe that an integration prototype as they described earlier in the paper seemed to cloud a bit of the delineation for me, and I feel like an integration prototype is much later-stage in the design process and perhaps shouldn't be considered with the role, look and feel, and implementation prototypes.

With regards to the second paper, I absolutely agree with the authors that the more designs you present to someone, the more criticisms you'll get. When you present just one idea, the subject might get wedded to that idea and focus on particulars of the design. Whereas with more ideas to critique, the subject will first focus on high-level issues before delving into specifics. I've had this happen to me in my case studies and focus groups for my club Berkeley Consulting.


Jason Toy - 9/28/2011 9:01:02

Dilemmas in a General Theory of Planning

Professors Ritel and Webber discuss the problem of trying to depend on professionals to give scientific answers to public policy problems. According to them, public policy problems never can be completely solved because they are ill-defined, and a solution cannot please everyone in a pluralistic society.

The paper answer the question of why there is a movement against the modern professional: that they failed to solve the problems of society using scientific methods, and why this is not possible. For years, progress was determined by efficiency which was easily obtainable by increasing output or reducing input. However, given the many subdivisions of society we have today, there is no right answer: cutting down a tree grove near Memorial Stadium for stadium expansion might please sports fans, but would anger environmentalists. In addition, opportunities we are allowed in science, such as the ability to repeat an experiment, or be incorrect are no longer plausible when solving social problems. The paper does not suggest a new approach, noting that whoever is in charge will be guilty of perusing his own personal agenda of what is right for society. However, by bringing these problems into the light, the paper may allow for discussion of how we can alleviate these problems.

I liked the non-partisan stance the author was able to take on the various subgroups in a pluralistic society. He doesn't claim that any number of them are right and wrong, but acknowledges their wants as a part of a society that is shifting away from the focus on efficiency. One of the best points is his first "There is no definitive formulation of a wicked problem". "The information needed to understand the problem depends upon one's idea for solving it." In science, there is usually an idea or theory you are trying to prove correct or wrong, whether in physics, or math, etc., and you have the information necessary to do do your experiment. However this is not true in the real world. Maybe you have to do a poll to find out what the problem is, and if you never ask the right questions, you will never get the right answers. One thing the paper could have done better was to attempt to provide some possible improvements on the status quo. For example, a continued focus on engineering and subway systems might lead to a future where no one would have to think hard about transferring a solution from one city to another, solving this problem from a scientific approach rather than a social one. It is facts like these that make me believe that the scientific approach is not completely useless in the world of public policy.

Getting the Right Design and the Design Right

This paper is about doing testing on multiple equivalent interfaces instead of one in order to generate more critical and truthful responses from users.

The paper presents a system for testing, arguing that presenting multiple interfaces reduces the negative feelings in users when they criticize a design, allowing for more accurate responses and a better design. This system relates to marketing research and focus groups, where different product designs, interfaces, commercials, etc. are presented to users and their responses factor in to which is actually picked for consumer use. The PlayStation One controller had four prototypes which eventually lead to the iconic design today. Adopting this policy would definitely change the design of future products. Designers would have to spend extra resources building additional interfaces and testing them, increasing the product creation cycle.

The problem the authors try to solve is a well-motivated one, allowing users to reject certain designs outright, similar to what is done in various industries today. A weakness of the paper is that they assume that is always possible to create a low cost testing framework. While it was true in this case, there might be software or hardware products where users would not have the same experience playing with notecards as they would touching and interacting with the real thing. To get accurate testing impressions, prototypes need to be built and tested, which might be a costly task, especially considering that multiple designs must now be created and produced. Another problem is that they do not acknowledge other possibilities for gathering feedback from the community about good design. While the testers were able to choose from three designs, these were three designs that the designers thought were best. What if a user thought that none of the designs were good at all? What they would have liked was never factored into the options, which could have been remedied by getting the opinions of users before building the three interfaces.