Input II: Surface Computing

From CS260Wiki
Jump to: navigation, search

Bjoern's Slides

File:Cs260-slides-05-input2.pdf

Extra Materials

Additional reading:

Videos:

Discussant's Slides and Materials

File:SurfaceComputing ArpadKovacs.pdf

Reading Responses

Airi Lampinen - 9/13/2010 14:08:58

In "Interacting with Paper on the DigitalDesk", Wellner discusses the potential to leverage the advantage of both computer-based interaction and paper documents. The presented vision of bridging between the digital and the physical is the DigitalDesk.

The DigitalDesk makes the desk more like a workstation, instead of trying to make the electronic workstation more lie the physical desk. DigitalDesk, the concept around which the paper is built, is listed to have three important characteristics: it projects electronic images down onto the desk and onto paper documents, it responds to interaction with pens or bare fingers, and it can read paper documents placed on the desk. Next to these, possibilities for collaborative work over distance are discussed.

The paper ponders in remarkable length on the difference between integrating the world into computers and integrating computers into the world. While only fairly vague notions from user studies are presented, the author makes a clear statement, situated in the office context, in favor of enhancing paper with better access to computers rather than the other way around. Thus, Wellner questions the merits of virtual worlds and puts forward a different vision of how computers can be integrated to our working days in a useful way.

The second article, Han's "Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection", dates from a bit more than a decade later than Wellner's discussion on the DigitalDesk. The paper focuses on a technique for enabling high-resolution multi-touch sensing on rear-projected interactive surfaces based on frustrated total internal reflection. The author describes an implementation of the technique, along with considerations of its drawbacks.

The scarcity of information concerning the tactile image is mentioned as perhaps the biggest problem: Since, the system provides no proximity (i.e. “hover”) information, nor any ability to label or classify each point of contact, it cannot determine, for instance, whether something was caused by one or two persons, using fingers from one or two hands.

Both papers mark steps in the development of surface computing, as does Buxton's overview of different multi-touch systems.


Kurtis Heimerl - 9/13/2010 17:08:35

Interacting with paper on the DigitalDesk Interesting work. I've been involved in paper-based computing for a while, specifically working with annoto pens and their applicability to the developing world. That's somewhat relevant to this discussion, though I feel that time has shown that these desktop questions are not that important. I know there's still work being done (mostly MSR, I think) so I could be wrong.

Anyhow, the big thought here was to switch the "desktop computing" metaphor around, and actually have a computing-enabled desk rather than a desktop on your computer. This, of course, requires a lot of hacks and a lot of specific technologies (OCR, cameras, so on). I remain somewhat unconvinced of the utility of the system, I don't think they have a great intuition on what the strengths of paper are; portability, price, and ease of use. They leaned heavily on resolving the final issue, but did not discuss the other two in depth. Given the advantages of computers (spell check as an example), I think traditional desktop computing already wins on usability/functionality.

That's not to say that there are no wins here. They clearly improved something, I just don't feel like it was the core issue required for such a system to take off.

On another note, I want to posit a thought I had while working in this space: A printer is a really low refresh rate monitor. If you had an annoto pen, you don't need a scanner any more, and if your updates are orthogonal and rare, that's a pretty powerful paper-based system.

Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection Yay! 4 page paper. That was a pleasant surprise. Anyhow, I don't have much to say about this paper. I mean, it works, but this was a pretty technical discussion. I buy everything I understood. The diagrams helped with that. Seems like a pretty clear benefit to the HCI community.

The only real thought I had is the following: It's pretty hard to CSCL on touchscreens, and this is a problem for me. I really wish there were a way to identify more than finders, but users. It was nice to know that there's some literature on this particular complaint of mine.


Charlie Hsu - 9/14/2010 1:17:24

Interacting with Paper on the Digital Desk

This paper described the DigitalDesk, a desktop enhancement focusing on bringing electronic enhancements to the physical environment of the desktop, instead of creating a virtual desktop environment on a computer. The author's group believed that paper as a physical tool was extremely hard to replace; paper is cheap, universally accepted, tactile, and portable. However, electronic documents have a strong set of convenience features, and can be integrated into the physical working environment via the DigitalDesk, thus ensuring the best of both worlds. The DigitalDesk encompassed a calculator, copy/paste drawing, and a document collaboration application.

The key idea of using computers to enhance the physical environment instead of providing a virtual environment brought back the vision of ubiquitous computing. This sounds like a perfect application of ubiquitous computing; bringing digital enhancements into the already existing physical world. However, as a consequence, it suffers from the same concerns ubiquitous computing brings as well. Privacy may become an issue for confidential physical documents being constantly photographed.

I feel that key idea of switching the context, as in ubiquitous computing, is the main research contribution of this paper. The paper provides a concrete example of converting something virtual (the virtual desktop environment) into a virtual enhancement of the physical. If we think about other things that have been converted to virtual representations, how could we possibly use new technologies of computer vision and other sensors to possibly turn these into enhancements of physical entities? What about things like virtually enhanced games in the real world? (I am imagining Scrabble with an electronic dictionary built into the board, professional sports with electronic officiating equipment, like line calling and baseball strike zones. Tennis already has electronically monitored lines for slam tournaments.)

Why hasn't an electronic enhancement of the physical desktop environment caught on? I feel that many of the applications and the concerns they address are valid: eliminating the need for switching between viewing a paper and entering numbers into a calculator was immediately enticing to me. Hardware has come a long way since July 1993, the publication date of the paper; digital cameras have become ubiquitous and high resolution cameras are not a hardware limitation at all. The problems of printing and scanning are still present. Perhaps user adoption of personal workstations has become so strong that users are reluctant to halfheartedly digitize with the DigitalDesk. Or perhaps digital workstations have established themselves as important for certain tasks disconnected to paper. Computers may be good for calculations, document creation, collaborations, etc. while paper remains good for reading. Perhaps a DigitalDesk-like implementation would be hard to sell and integrate with the millions of different types of desks in the physical world. Ultimately, I feel that electronic methods of maintaining files and documents beat out physical methods in almost all metrics but cosmetic, and digital enhancements have become powerful enough to emulate nearly all the physical advantages offered by paper. For the cosmetic advantages of paper, printing is enough.


Low Cost Multitouch

This paper details the implementation of a low-cost multi-touch sensor using "frustrated total internal reflection." Using machine vision and image processing, the implementation takes advantage of the "frustrated" refraction of light caused by an extra "material at the interface" (a finger), where the interface is a medium with a lower index of refraction (i.e. glass to air). Image processing translates the data into light points representing each of the user's digits on the interface.

This paper's research contribution to HCI is solely as an introduction to one possible implementation of a low cost multitouch input device, as stated in the introduction. It is interesting that when an idea like multi-touch sensing has a strong enough grip, even a paper on simply one cost-effective method on how to implement it can be considered HCI research, even though the usability of the multi-touch sensor here is not explored in detail. The merits / possible uses of multi-touch are not even touched in this paper either! I am certainly not reluctant to call this HCI research however; through the readings in the class, I am beginning to understand the different sorts of forms HCI research can take, and the collaborative nature of publications, revealing findings as they come in the hopes of inspiring more discovery.

I was a little unsatisfied with the paper's description of the actual implementation, however. It seemed that although most of the physical details were described, the paper lacked a more holistic view of the implementation; a larger resolution real-life picture would have been helpful, as would a larger schematic overview. I also felt some of their concerns about their approach were unwarranted: hover seems to be a relatively difficult goal to attain with a touch-based sensor, due to occlusion. Determining left vs. right hand touch input could much more easily be done with two touch-sensors, one for each hand, and the same is the case for multiple user touch sensing.


Anand Kulkarni - 9/14/2010 11:27:30

Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection This paper describes a new low-cost, low-complexity multitouch sensing platform using infrared light and a camera.

The paper here describes a substantially cheaper, easier way to construct an interface technology that had already existed for some time at the time of publication. While the paper does not have much impact in terms of analysis of a novel mode of interaction-- the user experience and interface in TFIR are not terribly different than other multitouch platforms, just the underlying technologies -- the likely impact is more fundamental and perhaps more important practically. The technology described allows multitouch technology to be used within a much wider range of other applications, both commercial and research-oriented, which can have a broader impact on HCI than proposal of a new kind of interaction. For example, this method permits cheap construction of wall-sized multitouch displays. The fact that we're constructing IR-based multitouch devices over the course of a weekend with nearly off-the-shelf parts should give a clue to the strong potential influence of this paper. Of course, our multitouch devices are based on ordinary reflection, not frustrated total internal reflection, but surely the simplicity of our in-class multitouch systems owe something to the central concept of this paper.

The problem the paper sets out to address is the high complexity and cost of multitouch sensing devices, and it solves this problem well. There's no user experiment, but they do describe several of the perceived benefits and drawbacks of the technology, including the potential degradation of the interface surface during use. Only a limited amount of time is spent discussing the drawbacks of this interface; this is fair, given that the authors were under space constraints. I'm not sure they motivated the final drawback they stated well enough -- hover information, and user discrimination, are not self-evidently so important that the system could be considered lacking just because it doesn't offer it. The authors should have justified this better or described it as future work of interest.


Interacting with Paper on the Digital Desk The authors describe a physical desktop with a mounted camera/projection system above that allows users to work simultaneously with physical and electronic documents on its surface.

The paper suggests that the joint physical-electronic model for the desktop takes advantage of many of the (purportedly) desirable properties of physical interaction, such as tangibility, touch-based control, and manipulability, while capturing desirable properties of electronic documents, such as portability, concurrency across multiple users, and automation and ease of processing. The authors view their main contribution as the invention of a unifying interface for this purpose. However, it would be difficult to argue today that this was the most important contribution of the paper to the development of HCI; though I know of more recent research work in digital desktops (for example, "Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops", by Maneesh Agrawal here at Berkeley), these systems have yet to gain traction in any user environment compared to distinct electronic and paper desktops. This isn't to say that the technology demonstrated in the paper hasn't made an impact, however -- I'd argue that the real contributions of this paper have proved to be in its foresight in the context of augmented reality environments, which is becoming a topic of tremendous research and interest on mobile platforms. The authors, remarkably, had built a touch-based augmented reality desktop in 1993, which is something that I expect we're likely to see in some form on mobile platforms in the near future.

The authors cite little external work in support of their arguments that certain features of paper or electronic documents are superior, but these claims are mostly plausible despite this fact. The real drawback in my view is that the evaluations provided in the paper are not formal user studies, but simply remarks made by informal users providing casual comments; I would prefer to see a formal evaluation of whether the hybrid environment offers the advantages the authors claim it does over paper and electronic documents respectively. However, the authors do have some impressive photographs of technology demos running on the system, including a painting program projected onto a real sheet and a touch-based system for pulling numbers off of a paper document and into a calculator, which have served to make up for much of the lack of formal evaluation -- the technology's rich potential for supporting new kinds of interactions at least, speaks for itself.


Thejo Kote - 9/14/2010 14:49:49

Interacting with paper on the Digital Desk:

This paper discusses the DigitalDesk system created by the author. The main motivation behind the work is his opinion that much effort in the area of HCI research had been spent in integrating real world behaviour into computers, for example in the form of the desktop metaphor. He wanted to bring the capabilites of a computer into the real world process of interaction with documents.

The paper discusses three applications developed to showcase the concept and the challenges faced by the author in implementing them. I thought this was a really nice idea and implementation given the technology available at the time. As the author notes, paper is not going away anytime soon. Almost 20 years after this work, paper still plays a big role in our daily interactions. The PaperPaint application especially resonated with me. I am very bad at drawing with a mouse using currently available tools. I just prefer to draw on paper, take a digital photo, transfer it to a computer, crop it and use as needed. This is exactly the kind of workflow that DigitalDesk prevents.

But, the reality is that such sstems aren't really popular today. Maybe some of the implementation challenges discussed can't be solved cost effectively yet. There is also the problem of portability and generality of use cases for this type of system. DigitalDesk works great for manipulation of documents, but it does not address many other applications that computers are used for. That is probably a reason why such systems are more popular evem with the availability of improved technology.

Low cost multi touch sensing through frustrated total internal refelction:

This paper describes a low cost way of achieving multi touch sensing in rear projected surfaces. It uses frustrated total internal reflection to implement the sensing. The author provides some history of the use of this technique in the biometrics and robotics communities. The end result of this approach seems to be similar to that achieved through Rear Diffused Illumination, which we will use in the next assignment. I'm not aware of any commercial applications of this technique, but it's definitely seems to be a good low cost technique for multi touch sensing.


Dan Lynch - 9/14/2010 14:58:22

The Digital Desk

This article discusses the notion of human-computer interaction in terms of virtual and physical interactions with objects and electronic documents. In particular, the author argues that "[p]hysical tools can be hard to replace", and the author goes on to say that the reason people have not completely gone digital in the workplace is because people like physical objects.

The main object that hasn't been replaced but was thought to have been is paper. It seems as though focusing on this particular object would be extremely relevant. The current interface between paper and the electronic document is the printer and scanner, which as the author mentions is not convenient.

The idea of "Computer Augmented Environment for paper" was brought up and claims to be a solution to the minimization of printer/scanner conversions while enabling user interaction with important paper documents. The goal of the DigitalDesk is thus to make the desk more like a computer instead of completely digitizing the desk.

This is a very important idea because abstracting complex interactions with a mouse and virtual menus can be replace by a physical behavior and/or interaction with a real object. This enables people to pick up and learn interfaces rapidly. In addition, paper is ubiquitous and statistically has been shown to be resilient to the digital realm. Thus, a way of interacting with natural objects as normal, but benefit from computation is something of great value.

One question that I do have is, how can this technology be improved using today's technology? In the discussion of implementation issues, the author probably had no concept of what computer vision is today and underestimates (in my opinion) what can and what cannot be recognized.

Frustrated Total Internal Reflection

This paper describes the methods and implementation behind FTIR in context of creating multi-touch devices. A particular perspective is given and focuses on creating a robust system at a cheap cost and little effort.

Related work was discussed in terms of differences in implementation methods. The first is the use of discrete sensors, the second utilizes force sensors in a matrix configuration, the third uses video cameras. They argue that the video camera is a very good method.

This paper overall is implementation-specific more-so than philosophical. Nonetheless, it is important to those who develop these technologies---especially cs260 students.

It was also interesting how they solved sensing pressure, by utilizing the contact area of the finger. Also, the edge-lit infrared LED array was also a deviation from the proposed interface that we build in this class... I wonder which is better? I'm guessing its easier to implement the latter, however, I wonder how much benefit is really given by choosing the best of the two, whichever one it would be.


Siamak Faridani - 9/14/2010 15:50:20

All three articles follow a similar line of research but from different perspectives. The first article Interacting with paper on the DigitalDesk builds an interesting application on top of a relatively cheap input method. Along the same line Jeff Han proposes a new and affordable method by moving the sensing device and projection to the back of the screen and finally Buxton studies over 30 years of research in multitouch and shows us that innovations that we see in new devices are in fact not new, rather they are built upon a long history of research in multitouch.

Paper I: Interacting with paper on the DigitalDesk The author with comparing the old model of using documents (hard copies) with digital displays. It is always a trade off to go from hard copies to digital copies. On hard copies we can easily brows and read while on digital copies we have much more functionalities (like search, edit, multimedia, etc) at the same time we lose the convenience of paper copies in digital environments. Digital desk is the author’s solution to these problems. While digital desk still lacks the most important feature of hard copies (i.e. portability) it provides a lot of functionality that is impossible to have in traditional document editing environments among them collaboration might be the most interesting one. Digital Desk is a new model beyond what has been traditionally known as “Direct manipulation”. I found the section on implementation problems rather interesting since most of those technical difficulties are now solved by advancements in computer vision, image processing, and display technologies. Author correctly suggests that he looks at this research not as an end product but as a research toward understanding better interactivity techniques. In the other two papers we have seen that many other researchers have built upon these products and extended the boundaries of our understanding of interaction techniques.

Paper II: Low-cost multi-touch sensing through frustrated total internal reflection This paper was to me the most exciting one among the three, I had watched Jeff Han’s TED talk (url: http://www.ted.com/talks/jeff_han_demos_his_breakthrough_touchscreen.html) and also I had seen some of the products from his company but after watching this video on his academic website I was under the impression that perhaps he has figured out a complicated way to get some type of feedback from typical LEDs (url: http://cs.nyu.edu/~jhan/ledtouch/index.html) It was exciting to see that the technology that he is using is affordable and can be implemented by cheap parts and a little bit of coding (obviously they have put a lot of thought in it to make it affordable)

Paper III: Multi-Touch Systems that I Have Known and Loved In this paper Bill Buxton from MSR explains to us that although the new interfaces for iPhone are exciting there is nothing new about them. He tells the story of multi touch and how it has started. Additionally he goes on to explain to the reader that with all the excitement, multitouch is still lacking a number of features. For example what about accessibility in multitouch environments? (a.k.a multitouch for the blind) or what about tactile feedbacks. What about sun shine? One aspect of the paper that I didn’t like about his report was that he does not look at these shortcomings as more opportunities to do research. An example of the research that can be done with multitouch is to extend that stretching cloth concept (sensing from the back of the device) with multitouch. The article has a wonderful informal survey of research. He sometimes forgets to include the exact date but the chronological presentation helps the reader to understand the amount of the research that has gone into development of multitouch.


Drew Fisher - 9/14/2010 15:59:01

Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection:

This paper is useful because it presented the idea powering most large-scale multitouch displays today, including products like Microsoft Surface. Notably, the hobbyist community has used this technique extensively for creating multi-touch tables due to its low cost and simple implementation.

I liked the connection of the use of a camera and video tracking with a rear-projection system - it makes the system effective and relatively cheap to implement for large screens. I also appreciated the observation of factors affecting robustness - notably, the difference between dry and moist skin. My observations using FTIR screens provide corroborating evidence for this note.

I would have liked to see in the "Future Work" section a technique for creating a much thinner system; one that doesn't require rear-projection. Making a multitouch screen possible to retrofit over an existing LCD screen would make it much easier to deploy to many more people who don't have the space for a full projection system.


Interacting with paper on the DigitalDesk:

I particularly liked the autocalibration routine - this is intelligent, and a feature that would be neat to incorporate into a multitouch table. As an ambidextrous individual, I also appreciated the attention to handedness. I wonder if perhaps the system could dynamically figure out which hand is where, based on relative sizes and positions of the fingers, which could be tracked. While such tracking was probably intractably computationally intensive at the time of publication, I wonder if such a setup would be feasible now.

The specific "window" for high-res camera usage makes it clear that even in this system, it's the human that does all the adapting, not the system. This is a common theme whenever humans and systems try to collaborate. I question whether such a system successfully approximates the intuitive use of the body the way the standard papers-on-desk metaphor does.

I was also amused that the practical constraints of this system including human pride - "...some executives may be reluctant to hide their polished mahogany desk under a piece of paper or other screen."


Arpad Kovacs - 9/14/2010 16:02:23

The Wellner article introduces the Digital Desk, which attempts to combine the advantages of the electronic workstation (with conveniences such as cut-and-paste, spell-checking, and search) as well as the physical desk (which provides tactile feedback, easy legibility, and portability), without any of the inconvenience of photocopiers or scanners. The Digital Desk allows the user to place physical paper documents on the desk, which are read into the system using optical character recognition and image processing technology, and may then be manipulated through pen or finger-based input; it is also possible to retrieve electronic images, and project them onto paper documents. This ability to seamlessly transfer and manipulate documents in both electronic and physical format, and recombine them with physical and/or electronic figures allows for novel interaction techniques such as tracing a projection of an electronic image onto a paper document, or selecting data from a paper report and effortlessly copying it into an electronic calculator or spreadsheet.

I found it remarkable that the researchers found a way to create a working prototype, even given the limited technology that the researchers had available to work with in 1993. In particular, I thought that using sound to track the taps on the table, as well as prescanning the documents, and then simply recognizing the particular document on the table to overcome the lack of modern capacitive-touch grids and high-resolution cameras were brilliantly creative solutions. The main contributions of this paper are the identification of various obstacles that the authors encountered during testing of the Digital Desk. For example, they mention the Midas Effect, which occurs when users rest their hands on the touch screens and cause the system to interpret the touched objects as input. This is still a significant issue in modern multitouch devices (eg touchscreen phones and trackpads on laptops), since clearly determining user intent remains an unsolved problem. I also found the discussion of occlusion to be interesting; although the authors claim that users learned to interact in ways that kept their selections visible and were not bothered by the shadows cast by their hands, the overhead projection design used by Digital Desk is clearly not optimal and has since been replaced by back-projection techniques for newer systems such as Microsoft Surface and NanoTouch; the latter also eliminates the "fat-finger" problem that the authors anticipated in this pioneering work.

Unfortunately, the DigitalDesk paradigm did not come to fruition (except perhaps for a few CAD drafting tools), which is a shame since it would have eliminated the large divide between the paper and electronic world that remains even today. It seems that people were so enamored with the vision of a fully electronic, paperless world that they preferred to shrink their workstations, and give them additional properties of paper, such as portability, annotations, and improved readability, as exemplified by the laptop, stylus-input tablet-PC, and e-ink displays respectively. The PDF format is the closest approximation we have today to a document that bridges the digital-physical divide, but even that does not have all the affordances of physical paper such as weight, width, and texture (you can instantly estimate how much of a book you have read, but you have to lookup page numbers of a PDF eBook to tell how long it is and how far along you are). Therefore it seems that the need for a modernized Digital Desk is still present, and we still have much to learn from this device which was ahead of its time.


The Han paper describes the Frustrated Total Internal Reflection method for adding an inexpensive, yet high-resolution multitouch interface to rear-projected interactive surfaces. The principle behind this technique is that when light passes from a material (A) that allows it to travel quickly to another material (B) that impedes its progress (measured by the index of refraction), it will bend in a phenomenon known as refraction; if the refraction angle reaches a certain threshold, then the light ray will be unable to escape from the first material (which is known as total internal refraction). However, a third material (C) with a higher refractive index than the second medium (B) can alter the refraction characteristics when it is close to the interface, and can cause light to escape the first medium.

The main contribution of this paper is describing a practical implementation that leverages these optical laws, and discussing how it compares to traditional multitouch systems. The setup consists of a sheet of acrylic lighted from its edges using infrared LEDS. Upon pressing a finger onto the acrylic, light is scattered due to the frustrated total internal refraction, and this can be captured by a video camera that is orthogonal to the screen. The advantages of this solution are that the video cameras used in this approach are cheap and high-resolution; in contrast, the conventional approaches utilize more costly sensor-matrix-based sensors that have limited resolution, are opaque, and require many connections and elements. The author also identifies the drawbacks of the system, namely inability to sense pressure, space required behind the screen for the placement of the camera, and potential vulnerability to ambient light as well as contamination of the surface.

I found the most interesting part of the project to be its potential for future improvements, and possibilities for widespread deployment. The authors mention using a holographic film to "see through the screen", but I think that a simpler approach would be to add a second camera and with a different IR filter/threshold. With two cameras, it should be possible to perceive depth via stereoscopic vision and thus distinguish hover from touch. I also think that the idea of using an LCD backlight to reduce the volume of the device would be very worthwhile, since it could then be fitted to various consumer devices rather than just demonstration tables or boards. It would be awesome if in the future I could retrofit my TV or computer monitor with multitouch by just buying an IR camera and a special backlight or edgeperimeter-LEDs.


Thomas Schluchter - 9/14/2010 16:03:17

    • Interaction with Paper on the DigitalDesk (Wellner)

Wellner describes a desktop system that augments the physical artifacts on a surface with access to computing powers.

The DigitalDesk is an interesting change in perspective. Instead of asking how the computer can be likened to the physical world to flatten learning curves and to take advantage of everyday knowledge, Wellner reverses the question: How can the physical artifacts that we know how to interact with on a very deep level be computationally 'activated'? Wellner's system has an advantage in that it addresses a very well defined and controlled environment, the desk, but this has traditionally been the domain that application-oriented computing has tried to model.

It is very instructive to read the DigitalDesk description in the context of our discussion of Direct Manipulation. As Wellner points out, the direct manipulation that the authors of the 1960s paper describe is in fact not so direct. The interface introduces a layer of abstraction. With the DigitalDesk, the interface becomes the surface, and not in the sense that the user touches an abstract digital representation of the artifacts (s)he is working on, but the surface of artifacts is literally made intelligent. On the other hand, in those cases where the system creates a representation and projects it onto the surface, there is still more directness to the interaction from the user's perspective.

It is interesting to think about what this means for the concept of the affordance. Clearly, paper has a number of affordances that we are all acutely aware of, and that we take advantage of with great ease. If these affordances are augmented, how can the user learn about it without experimenting? Is there a way of representing the added capabilities of the digital-analog-hybrids that speaks as clear a language (in Norman et al's sense) as a sheet of paper?


    • Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection (Han)

This paper describes a scalable multi-touch sensing technique that relies on the refraction of light in situations where materials meet that transport light in different speeds.

The paper's relevance is in enabling anyone with moderate technical knowledge to take advantage of multi-touch sensing on a large scale. It makes accessible a heretofore extremely complex and buggy mode of interaction in a fairly reliable way. The key advantages of this approach are the clear distinction between hover and touch, the ability to recognize multiple different blobs on the surface and treating them individually and the possibility of back-projection.

In comparison to capacitive touch displays, this technique is more cost-effective to deploy with very large surface areas.


David Wong - 9/14/2010 16:44:21

1) The "Interacting with Paper on the Digital Desk" paper discussed a new device called the Digital Desk that, instead of focusing on virtual reality, tries to combine the physical environment with the digital environment. The paper brings up three sample applications on the Digital Desk and follows up with a discussion on the implementation. The "Low-Cost Multi-touch Sensing through Frustrated Total Internal Reflection" paper revisits the concecpt of frustrated total internal reflection and applies that to a multi touch device. It does not propose any new innovations in technology, but illustrates that a highly accurate multi-touch sensor can be built from existing technology.

2) The "Interacting with Paper on the Digital Desk" paper focuses on the idea that there needs to be research done to increase computation in the physical realm. This is an important focus for HCI research as it is, at least in my opinion, forward thinking, although it is a challenging research area. I believe that their attempt with the Digital Desk was okay at best, but the system was implemented in 1993. Given the advances in hardware and software from the previous decade, this area can be very exciting and add a lot of value to society. The "Low-Cost Multi-touch Sensing through Frustrated Total Internal Reflection" paper does not offer any new innovations in HCI, meaning that it doesn't introduce any new concepts or technologies. Rather, it offers a unique perspective on how a highly accurate multi-touch sensor can be created. The paper adds value to the research community as it can inspire new and creative ways of creating multi-touch sensors.


3) As a research paper, "Interacting with Paper on the Digital Desk" paper was quite poor. They did not run any formal experiments and had no way of evaluating the results of their informal experiment. At best, it was merely a proof of concept and a way for them to publish the idea. Nevertheless, the problem is well motivated as there needs to be more research towards a computerized physical environment. The "Interacting with Paper on the Digital Desk" paper did not really present an argument or conclusion. The paper merely discussed an implementation given existing techniques. I think, however, that the problem they are inadvertently solving is a needed one: how to create a cheap, precise multi-touch sensor. Also, having done so in a creative manner is inspiring.


Shaon Barman - 9/14/2010 17:15:57

Interacting with paper on the DigitalDesk

The authors of this paper explore ways to get past the inherent limitations between physical and digital documents. To do this, they use a variety of sensors which enable users to manipulate physical papers digitally and vice versa.

At this point in time, printing and scanning were relatively lengthy tasks (and even today they are time consuming). The authors tried to circumvent this process by automatically capturing images off paper using a video camera. In addition, to give the appearance of an augmented reality, they broadcast the captured images using a projector. In addition, they use a novel integration of tracking movement along with sensing sounds caused by vibrations to allow the user to interact with "digital" paper. The applications they show demonstrate a wide range of tasks, such as number recognition, user collaboration, and image editing. Overall, this method frees the user from the inherent limitations of the mouse and allows him to use all fingers when manipulating the "paper." In addition, the difficulties they encounter seem easy to overcome, except for the obscuring selections which is a problem even today with small devices.

While the authors demonstrate a clear understanding of the technical issues with the "digital desktop," they do not discuss many of the social issues. There seems to be privacy and security issues if there is always a video camera tracking your desktop, seeing and reading all papers on the desktop. In addition, because the desk has intelligence built into it, due to the video recognition of text and gestures, there seems to be a large chance of false positives. It is not clear how users would interact with this new interface where things may not be exact. And even though the user can manipulate papers on the desk, to transfer the paper to somewhere else still requires a physical copy. Although it seems like a more "integrated" desk would be a beneficial, it seems the digital-physical interface was too much of a barrier and it was simpler to move towards an all digital system.

Low-cost multi touch sensing through frustrated total internal reflection

This paper discusses how to create an inexpensive, scalable multi-touch device using an IR emitter (LEDs) and an IR sensor. They delve into the physical and technical challenges associated with it.

This paper only explores the "how" to create a multi-touch device and says nothing on the uses or implications of such a device. Technically, its quite interesting. I would have hoped they explained the issues involved with calibrating the device, and gave some ideas of how to reduce the size of device (distance from camera to screen).

They could have also discussed some of the social implications. Only a few years ago, Microsoft released a similar device for commercial sale. At first there was much buzz about the device. But it seems like the device has made little impact so far. One limitation it seems is that people are not used to sharing a computer. In the current model, a single user interacts with a computer and most programs/models are built for this setup. Having a device like this would greatly increase the number of users who could interact with it, but most programs do not know how to use the extra info. Also since the device itself is relatively cheap, I would have thought it would have made a bigger impact. This might have been offset by size restrictions or calibration requirements which make it hard to put in just "any" spot.


Krishna - 9/14/2010 17:30:56

Interacting with Paper on the Digital Desk

The paper describes DigitalDesk, a technology that provides access to information present in the physical paper. What is interesting about DigitalDesk is its primary focus on being a direct manipulation interface. This is quite contrary to current trends in this direction where the focus is on digitizing information present in the paper and providing alternate schemes of access by analyzing the paper’s content.

The author wonderfully summarizes HCI research as that which tries to enhance a computer by providing it with desirable physical properties from the physical world. He argues that the desktop metaphor is only an approximation and lacks many attributes associated with the physical paper - e.g. the tactile nature of the paper. However, the author acknowledges usefulness of the digital representation and thus the existence of trade offs. He promotes the DigitalDesk as the technology that provides the best from both the spaces; specifically, he wants to retain the tactile nature of the paper and facilitate computational access to the information present in it - an important learning, as a HCI researcher, would be to retain the physical attributes of the object we want the users to directly manipulate while enhancing its abilities computationally.

Though the DigitalDesk’s system architecture and hardware setup looks quite trivial, it would not have been during the times it was conceived and built. The technical architecture reveals the emerging confluence of different technologies and ideas like gesture recognition, motion tracking, OCR, remote collaboration, etc. It was quite instructive to read about the challenges faced while designing the system and how they were handled, especially about system calibration and on how it could be done, without user involvement, by solving a system of linear equations that relate points on the desk to pixels on the screen.

There can be many criticisms of the technology mentioned in the paper : its sheer size, economic infeasibility, lack of rigorous user studies and market analysis etc. It could also be argued that the author‚Äôs arguments on the portability and economic aspects of using physical paper are no more relevant and that we certainly see a push towards ‚Äòpaperless‚Äô office these days; also, the tactile feature provided by the physical paper may not outweigh these other factors.

However, I believe that the paper should be read from a different perspective; the creative aspect of the described technology is not trivial moreover, if the authors had worried about rigorous analysis and user experiments, they would have never started to build the technology and we, as HCI researchers, would have never known the technical feasibility of such interfaces.

Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection

The paper describes a low cost multi touch interface, an interface that enables users to use multiple fingers simultaneously while interacting with the system. The author briefs about existing technologies which predominantly seem to use an array/matrix of sensors. Most of these existing expensive interfaces are opaque and would require top down projectors. This, as we know from the challenges mentioned in the DigitalDesk paper, may lead to problems due to the shadows created by the user’s hands.

The central idea is to use frustrated total internal reflection. Light from a set of infrared LEDs are made to undergo total internal reflection, this is done by projecting the light from the LEDs at a certain angle on a material with a suitable refractive index - acrylic paper. When a finger touches the paper, this total internal reflection is disrupted (or frustrated) and the light is made to scatter or diffuse out through a diffuser, the light that escapes is then captured by a digital camera placed orthogonally to the table that holds the setup.The resulting images from the camera can be computationally processed using techniques from Machine Vision. For example, the fingers can be identified as blobs and their position can be tracked by subtracting successive frames.

The author mentions various drawbacks of the setup. It does not measure pressure, which might be important for certain applications - for example touch based musical instruments. The heavy usage of image processing makes the system susceptible to lighting conditions and dirt. The system is not capable of distinguishing between fingers from different people.

However, the key take away is the fact the author had creatively used a relatively obscure idea from the 70s to construct a trivial, inexpensive and yet accurate multi touch device.


Luke Segars - 9/14/2010 17:44:46

Interacting with paper on the DigitalDesk

This paper describes a device called a DigitalDesk, a physical desk that is designed to bridge the reality gap between the surface of physical desks and digital documents. The paper describes a number of potential applications that allow information to be copied to and from physical and digital mediums through a series of cameras and projectors suspended above the table. The applications also mention character recognition techniques so that number and words can be converted into their digital equivalents.

This paper is relevant for a very interesting reason: digital technology has, in many ways, made physical paper functionally obsolete. Many of the most obviously useful functions of paper can be performed more effectively on a computer than they can on paper (adding content, erasing content, editing, etc). In addition, it is also much easier to make copies of digital documents and share them with others. Nevertheless, paper is still a very relevant tool for today's world. It's hard to place your finger on exactly why this is so, but I myself am an avid paper / whiteboard user and often prefer to work in physical space instead of digital spaces. An idea's technical superiority, even if it is hugely superior, doesn't guarantee that it will make the previous technology obsolete -- there is very often room for both technologies to coexist for different scenarios where they may be superior.

Last week we discussed the relative merits of many interaction devices given the correct context. Even the ridiculed headmouse had a place once a fitting application was found. Similarly, a class of technologies can coexist that have similar purposes but different scenarios of usefulness. Nevertheless, there also seems to be a limit to the number of technologies users want to deal with to accomplish a set of tasks -- mental saturation is most certainly possible. Determining the different between space for expansion and no space is difficult to distinguish.

I thought that this paper was organized very well for discussing the merits of the DigitalDesk. The example applications seemed very realistic and useful (although very simple) to a range of users. In addition, the authors spent a significant amount of time discussing the issues they came across during implementation. They don't claim to have any experimental evidence with users, but their detail to problems with the implementation of this technology make this paper very interesting and worthwhile to read.

It would have been very helpful to hear some user feedback about particular use cases of the table. I suspect that people would at least initially have trouble merging the paradigms of digital and physical objects and that much of the user experience would seem unrealistic. It would be interesting to see how a user might use this device for something like editing a document from a peer or another realistic office task that users of this system would be interested in. It is unclear whether technology like the DigitalDesk would actually expedite this process or throw the user into a sea of technical troubles to solve.

I generally found the paper to be very insightful. Bridging the gap between physical and digital realities is a substantial field of research today and there is still a lot of progress to be made, but a number of the implementation issues that the authors faced are still very significant today.


Bryan Trinh - 9/14/2010 18:00:30

Interaction with Paper on the Digital Desk

Pierre Wellner makes an important realization that, despite the growing usage of digital paper, there is still a large number of people that prefer physical paper. This paper introduces a new human computer interaction paradigm that attempts to give physical objects, primarily paper, digital capabilities. This is fundamentally different, Wellner argues, than previous work that made digital objects more physical. The concept is tested using a projector for output images and a camera as the input.

Although I think that Wellner was right in thinking that we should move away from the completely digital desktop, I think that the physicality of the objects in his computing environment were not convincing enough. There were too many constraints in the workflow of his system. One example is the inability to move a piece of paper that contained digital and analogue media on it. Without this, the user loses the perception of working with a physical object. Even if the user could move the pieces of paper, they would still be confined to the 2d plane. Once the physical piece of paper is moved to the third dimension, the metaphor breaks.

Johnny Chung resolved this issue by using a wiimote and a projector. In his system objects could be picked up and moved around, and the projected images would track with the paper. The problem with this device is that the pages needed to be outfitted with sensors, which is not always practical.

I understand that he Wellner wanted to create one system that could add digital functionality to physical objects, but I think a better approach might be to just build physical objects that serve as metaphors for objects that we work with. An example of this would be the iPad or Kindle. These separate systems can then be networked together for computing continuity. This is a much less restrictive model to work with and retains many of the physical qualities of physical paper.

Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection

In this paper Jeff Han introduces a method of implementing a multi-touch interface in a cheap and reliable way. It is very similar to the method that we are using to build the multi-touch sensing surface in that it uses computer vision to analyze light patterns. His method however, will not register false positives from other items placed on the surface as ours would. It also seems to be not as strongly effected by ambient lighting because of the larger contrast between light and dark that the FTIR system has.

This paper is instructive because it illustrates the same shortcomings that we will encounter when we build our multi-touch devices. For instance, there is no way for us to detect the differences between our different fingers. Other than that the paper just provides us with some details on how they constructed the device.


Bryan Trinh - 9/14/2010 18:10:23

Interaction with Paper on the Digital Desk

Pierre Wellner makes an important realization that, despite the growing usage of digital paper, there is still a large number of people that prefer physical paper. This paper introduces a new human computer interaction paradigm that attempts to give physical objects, primarily paper, digital capabilities. This is fundamentally different, Wellner argues, than previous work that made digital objects more physical. The concept is tested using a projector for output images and a camera as the input.

Although I think that Wellner was right in thinking that we should move away from the completely digital desktop, I think that the physicality of the objects in his computing environment were not convincing enough. There were too many constraints in the workflow of his system. One example is the inability to move a piece of paper that contained digital and analogue media on it. Without this, the user loses the perception of working with a physical object. Even if the user could move the pieces of paper, they would still be confined to the 2d plane. Once the physical piece of paper is moved to the third dimension, the metaphor breaks.

Johnny Chung resolved this issue by using a wiimote and a projector. In his system objects could be picked up and moved around, and the projected images would track with the paper. The problem with this device is that the pages needed to be outfitted with sensors, which is not always practical.

I understand that he Wellner wanted to create one system that could add digital functionality to physical objects, but I think a better approach might be to just build physical objects that serve as metaphors for objects that we work with. An example of this would be the iPad or Kindle. These separate systems can then be networked together for computing continuity. This is a much less restrictive model to work with and retains many of the physical qualities of physical paper.

Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection

In this paper Jeff Han introduces a method of implementing a multi-touch interface in a cheap and reliable way. It is very similar to the method that we are using to build the multi-touch sensing surface in that it uses computer vision to analyze light patterns. His method however, will not register false positives from other items placed on the surface as ours would. It also seems to be not as strongly effected by ambient lighting because of the larger contrast between light and dark that the FTIR system has.

This paper is instructive because it illustrates the same shortcomings that we will encounter when we build our multi-touch devices. For instance, there is no way for us to detect the differences between our different fingers. Other than that the paper just provides us with some details on how they constructed the device.


Brandon Liu - 9/14/2010 18:17:19

“Interacting with Paper on the Digital Desk”

This is a proof of concept paper that extends the work of Mark Weiser on the “Computer for the 21st Century”. It justifies its own work by using Weiser’s points about virtual reality vs. embodied virtuality, and also describing the importance of paper-related tasks to computer users in general. The paper is intended only to show one possible implementation of ‘embodied virtuality’, and does not present a rigorous user study.

My major issue with the paper was when it described how “user testing revealed’ that people created purely projected images instead of mixed images. My complaint is that if the users gravitated to that solution, it indicates a problem with mixed projected/physical images. The authors claim correctly that both purely digital workspaces and purely physical workspaces have their unique benefits. But this doesn’t imply that a mixed workspace is as good as either. I think there is an implicit ‘friction’ in the combination that is not addressed by the authors. I feel like this is valid criticism (of a paper without a rigorous user study) since the point made is against the concept of mixed projections, and not the author’s specific implementation. It seems like this hypothesis (that a mixed environment is just as good as a pure projected or pure physical one) should be tested by a rigorous user study.

A similar criticism I have of the paper is that it doesn’t justify why translating computer tasks are necessarily efficient. I believe the authors when they say that taking advantage of our habits with manipulating paper in the real world is A Good Thing. They then describe a system that lets users do selection with marquee boxes on the table. This kind of selection isn’t in our toolbox of normal interactions with physical paper, so it isn’t obvious to me that it is necessarily better. The authors need to explain why this is so; since it isn’t common knowledge that just involving tactical manipulation is always better, when the ultimate goal is usability.


“Low cost Multi-Touch sensing through Frustrated Total Internal Reflection”

In summary, the paper described a specific implementation of multi-touch sensing using low-cost components. The author gave an overview of the problems with other implementations of multi-touch. The paper then described the physical phenomena behind doing multi-touch sensing using off-the-shelf parts. Then, it went into detail about the shortcomings of the current device and how improvements can be made with better materials and components.

One thing I really liked about the paper was that it brought up and responded adequately to many possible criticisms of the device. It even describes how richer functionality than basic multitouch can be implemented (for example, pressure by size of the finger impression). The author even considered the cost of the device.

This paper is relevant to HCI in that it is a “golden nugget” that complements more conceptual work, such as the previous reading. While the previous reading brushed away a lot of the technical details saying that “they will be possible soon”, this is the kind of paper that actually realizes those goals. I’m surprised that such a device was published through an academic conference instead of being filed as a patent (I looked up the author and he holds a multitouch patent on a different kind of device).

Also, the implementation described is novel since it shows how novel HCI devices can be created from off-the-shelf hardware and sophisticated software, instead of always needing to make new hardware.


Matthew Can - 9/14/2010 18:23:10

Digital Desk

This paper describes the DigitalDesk, a device that augments the physical desktop with the power of computer technology. The device projects virtual objects onto the desk and paper documents, and it allows for the tactile manipulation of both the paper and virtual documents on the desk. The paper describes three applications built for the device and implementation issues that were encountered.

The main contribution of this paper is the development of a direct manipulation interface that integrates paper documents and the digital world, but does so using a new paradigm. Instead of extending the desktop metaphor of workstations, the device presented in this paper extends the capabilities of the physical desktop by giving it properties of virtual objects and documents. This opens up a big avenue for research in HCI.

The PaperPaint application in the paper was a good example of what this system does well. It allows users to manipulate virtual documents the way they manipulate paper documents. In addition, it allows users to edit paper documents with the ease of editing virtual documents (though the edited paper documents are actually mixed paper and electronic documents). This application illustrates how the DigitalDesk can provide integration between paper and virtual documents that is more seamless than a desktop workstation.

I like that the paper addressed the implementation issues with this system, because there are several, and they are nontrivial. What seems most problematic to me is the video-based finger tracking and audio-based tap detection. It does not easily support multi-touch interaction and leads to false positive taps. I think a touch sensitive surface is preferable to the implementation in the paper (the touch input can be filtered to reduce the “Midas Effect”).


Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection

In this paper, Han describes how frustrated total internal reflection (FTIR) can be used to build a simple and inexpensive multi-touch sensor. In doing so, he raises issues with the approach and explains how they can be addressed.

This paper’s contribution to HCI is a new kind of multi-touch sensor technology that is based on FTIR. This is important because, as the author emphasizes, the technology is simple, inexpensive, and scalable, which means it can lead to further commercialization of multi-touch devices.

The implementation section of the paper presented a good level of detail on the multi-touch sensor in the paper. It gave me a good understanding of how the device works. The author also explained and justified his design decisions. Furthermore, he sufficiently addressed the potential drawbacks to his approach. The discussion of the sensor response being dependant on the object being sensed and the explanation of surface contamination were both thoughtful.

One thing lacking in this paper is a comprehensive comparison of the FTIR-based multi-touch sensor to other types of multi-touch sensors. I would like to see a comparison of cost per square inch, sensor resolution, space requirements, etc. In addition, a comparison of qualitative differences would be useful too.


Luke Segars - 9/14/2010 18:23:28

Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection

This paper describes a method of creating cheap multi-touch surfaces that can be used for boards, walls, tables, and other interactive technologies that can accommodate rear projection. The technique described in the paper is identical to our current assignment, and the author discusses a number of details that go above and beyond our requirements such as how rear-projection can be used to add a visual output component to the device.

This is one of the hardware components that I feel most optimistic about in terms of its usefulness and realizability in the current social climate. Relative to technologies like the DigitalDesk, it doesn't take a huge conceptual jump to move from a physical whiteboard to a digital one. Many of the tools will be the same and the behavior of the system will be identical to a normal whiteboard except for some additional features like saving, loading, and potentially multiple users interacting on the same surface. In addition, the whiteboard will generally look the same as it has before -- all of the hardware is hidden behind the device and ideally out of sight of the user. It will, in essence, have a very low barrier of entry to someone coming from a traditional whiteboard background.

The author doesn't go into a lot of detail about how a rear-mounted camera and projector can both be physically incorporated into the area behind the screen. Both devices would ideally be centered behind the screen to minimize distortion on the input and output, but it may be necessary to place one beside the other and correct for the offset / distortion in the software layer. Outside of this single issue, the authors description makes it easy to reconstruct a similar device.

Perhaps the biggest benefit of the approach this paper describes is that there is a very low-cost method of building the technology ourselves. Not only is this great from an interaction and research standpoint, but it also provides an outstanding opportunity for pedagogy and advertising the merits of the field of HCI. Overall I am a huge fan of this idea.


Linsey Hansen - 9/14/2010 18:25:17

Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection

In his paper, Han, explains a new method to create cheap multi-touch sensing devices using frustrated total internal reflection (FTIR).

Though FTIR itself is an old technique that has been used since the 60s for fingerprinting, Han decided to take further advantage of it. The way it works is by using cameras to detecting scattering of light from it's normal refraction within a glass-like material- in Han's case it was acrylic, though he listed other capable materials. This scattering is caused when another material (ie a finger) is placed on the material, thus frustrating the internal reflection. There are some issues with this method in harsh lighting environments, or when the screen gets dirty (though I don't know of a touch device that doesn't have trouble with a dirty screen), but nothing crippling. One other issue is that since this uses a camera, the camera needs to be positioned far back, so there cannot be small devices.

As the author noted in the beginning, the point of his paper was merely to show a cool new way to create multitouch devices, not to show how to use them- though he did mention a number of ways in which other researchers could apply this method. To add to that, I feel like this sort of technology could be used to create something like a touchscreen overhead projector, the only problem is that there would probably not be a place for visual feedback aside from the place you were projecting to, which could be annoying, or there could be a separate LED to give feedback, but then that would just be expensive, but it might be reasonable within a few years. This would also be great for placing giant touch screens on a desk, for say, artists or something, but artists might require a stylus, and creating a stylus that works well enough could be a problem.

Interacting with Paper on the Digital Desk

In his article Wellner describes a “digital desk” which is a way to add the computer to the real world for users in workspaces. This digital desk would be able to lessen the gap between actual paper and digital documents by adding digital properties to any regular piece of paper without requiring the user to actually digitize it.

This article mostly serves to share the idea of a digital desk, which is defined by three characteristics: it can project electronic images onto paper, it responds to both styli and bare fingers, and it can read paper documents that are placed on it. This is accomplished by using a (or multiple) cameras and a projector, and he went on to talk about three examples for how this could be used for calculation, painting, or interacting with other people. In relation to today, almost all of these tasks are easily done on computers, though there is no physical piece of paper to keep track of everything on unless one is printed, so this would be another way of doing that if someone wanted that since it would save them a step.

Considering that this came after Weiser's article about tabs, pads, and boards, this concept seems less fancy, but is still pretty imaginative and has goals similar to those others are still trying to accomplish. Also, looking at today's technology I feel like there were some blind spots. For starters, while participants did not seem to mind the projector, that still seems rather annoying and distracting whenever your hand is under it. I still do not understand how someone could have thought it was an actual screen since the projection would have been on his hand, so maybe most people really just would not notice. Plus, when I think of projections, I still think of something that, while not like staring at an LCD, would be really bright, and not as easy on the eyes as paper. Then, since one still needs to use actual paper in order to be able to draw on it, I feel like that is a waste, but then if something like it were implemented today, I am sure that a stylus or finger-only one could be created. Looking more into what is available today, I feel like a similar effect could be accomplished by creating an actual “paperless” environment, where papers become interactive eInk tablets (for those that prefer paper, but even normal tablets would suffice) then what is being displayed could be clever controlled with some other device (ie little tokes of some sort for representing different papers to display, where the tablet adopts the contents of the closest token). Or, another alternative would be to just have a digital paper place all along the surface of one's desk (so the desk would be a giant screen), where papers could be looked at and moved around at will.


Richard Shin - 9/14/2010 18:44:43

Interacting with paper on the DigitalDesk

This paper describes a computer-enhanced desk named 'DigitalDesk', which enables its users to interact directly with electronic documents on a literal desktop, while imbuing paper documents placed upon it with the advantages of electronic documents. The system consists of a projector, combined with a number of cameras, pointed at the desk and connected to a computer, so that it can display things on the desk surface as well as read paper documents placed upon the screen. This enables applications such as calculators which read numbers from paper, drawing on paper with electronic copy and paste, and simultaneous collaboration on paper documents with other users.

The paper's main contribution seems to be a novel user-interaction paradigm that applies direct manipulation to its logical extreme, at least in the manipulation of documents. Instead of simply focusing on adding tangibility to electronic objects, the authors focused very much on making physical paper a part of the computer's user interface. Text and images, whether on paper or projected electronically, can be manipulated consistently, blurring the boundary between the physical and the virtual. Given that the computer gains all its input through vision, the system seems easily extensible to supporting other types of tangible objects without, e.g., adding special radio tags to them.

Unfortunately, due to the system's limitations, I wasn't convinced that it would be practically useful. While information on paper can be easily manipulated electronically, there exists no good method to reflect the changes back onto paper, defeating one of paper's listed advantages of portability. Similarly, the inability to manipulate the paper physically, rather than electronically manipulating the information, seems a major limitation that greatly reduces the intuitiveness of the system.

Low-cost multi-touch sensing through frustrated total internal reflection

This paper presents a new sensing method (or, rather, a new application of an old method) for multi-touch interaction. By illuminating a waveguide (a transparent panel made out of acrylic in the paper) at its edges with IR LEDs, the placement of fingers upon the panel causes light to escape at those points, which can be detected by a suitable camera, and the resulting images processed with computer-vision techniques to detect the locations of fingers. Combined with rear projection onto this panel, a display supporting multi-touch interaction can be built.

The paper doesn't describe the benefits of multi-touch interaction, but rather only describes a technique for sensing multiple touches on a surface at once. Nevertheless, since the ability to sense multiple fingers is a strict improvement on the ability to sense only one finger at once, the benefits are not difficult to imagine. The paper hints that previous research had explored the potential of multi-touch but actual multi-touch sensors had been largely unavailable; given the low cost of this technique, it seems poised to enable future research into the topic.

As far as I can tell, the mechanism as presented in the paper seems sound and adequate for the intended use. However, the technique doesn't seem easily applicable to how most computing is conducted today---at the desk or with a laptop---but rather useful mostly for large, rear-projected screens, which tend to be expensive on their own and inaccessible to most people. While great as a technology demonstration, I'm not sure how useful the technology will be, given its limited applicability.


Aaron Hong - 9/14/2010 19:00:03

In "Interacting with Paper on the Digital Desk" Wellner describes research at Xerox PARC (haha, so much of it was developed there) of bring back the tactile interaction of pencil and paper to the computer. It is a matter of perspective he says that drives our goals in combining the digital and the physical. As for the Digital Desk, it is more on the computer workstation side of the paper-computer spectrum: "The goal is not to enhance computers by giving them better access to paper; it is to enhance paper by giving it better access to computers."

One of the most important things about computer interfaces that Wellner mentions is the idea of tactile interaction. Objects in the world have certain affordances, the computer is limited in this respect. Current technology is bringing back the affordance of touch, but lacks the feedback of real buttons often time because often these touch interfaces are just flat screens. That's were the idea of haptics may be explored as a solution to the missing "tactility" of physical buttons, knobs, and other mechanical devices.

Finally another thing talked about in this paper was occlusions and shadows. What was not talked about in this paper that has probably been discussed (almost certain someone thought of it) is the combination of front-projector and back-projection. This way you can project onto physical objects, yet see what is being occluded (to some degree). Also eliminates shadows and more feasible for difficult lighting situations (lamps, etc.).

In the other paper, "Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection" by Jefferson Han talks about a cheap technique for multitouch, Frustrated Total Internal Reflection or (FTIR). It uses optics to sense the touch, and can do so relatively well with inexpensive equipment. The main problems discussed is finding a suitable screen material and ways to mitigate the needed distance in the back of the screen needed for the technique. One interesting question that he brought up at the end that has not been discussed in many of the papers I've read on multi-touch is the use of hover. I've used hover on my Wacom tablets and find it to be a useful state to have that is reminiscent of the added functionality of a mouse.


Kenzan boo - 9/14/2010 19:00:26

Interacting with Paper on the Digital Desk

The article describes the merger of physical paper and the digital media. It describes a projected desk onto the physical desk so we can have the benefits of both electronic media (spell checking, etc) and the benefits of physical paper, the form and familiarity.

I thought this was interesting as there has been a better attempt at this in the modern day. This design is no longer productive and is way too big and clunky to be used well. However the idea is still very attractive, we want both the free ability to write anything we want on physical paper but also have that in digital sharable and reproducible format. One modern day attempt at this is the light scribe pen that records everything you write onto a pen that can later be transported to the computer. This brings a digital aspect to the physical paper.

One of the biggest limitations to the computer is the ability to have free form writing and drawing, with the detail and physical aspects of paper. A drawing on paper with colored pencils or pastels has much more color and feel to it than its parallel on screen.


Low cost multitouch sensing.

The acticle is a brief introduction into what we will be doing in our project. It details how light is refract in Frustrated total internal reflection. It is caused when a new object is pressed onto the surface and changing the refraction between the prior material. This is what makes our hw2 work.

The ideas are interesting, especially the one with a rear projector but I feel that the device would be much to immense to be used as anything more than a demo or proof of concept. I would like to test it if we could get a hold of a projector. This was a good read for our hw2.


Kenzan boo - 9/14/2010 19:00:26

Interacting with Paper on the Digital Desk

The article describes the merger of physical paper and the digital media. It describes a projected desk onto the physical desk so we can have the benefits of both electronic media (spell checking, etc) and the benefits of physical paper, the form and familiarity.

I thought this was interesting as there has been a better attempt at this in the modern day. This design is no longer productive and is way too big and clunky to be used well. However the idea is still very attractive, we want both the free ability to write anything we want on physical paper but also have that in digital sharable and reproducible format. One modern day attempt at this is the light scribe pen that records everything you write onto a pen that can later be transported to the computer. This brings a digital aspect to the physical paper.

One of the biggest limitations to the computer is the ability to have free form writing and drawing, with the detail and physical aspects of paper. A drawing on paper with colored pencils or pastels has much more color and feel to it than its parallel on screen.


Low cost multitouch sensing.

The acticle is a brief introduction into what we will be doing in our project. It details how light is refract in Frustrated total internal reflection. It is caused when a new object is pressed onto the surface and changing the refraction between the prior material. This is what makes our hw2 work.

The ideas are interesting, especially the one with a rear projector but I feel that the device would be much to immense to be used as anything more than a demo or proof of concept. I would like to test it if we could get a hold of a projector. This was a good read for our hw2.