Surface Computing

From CS260 Fall 2011
Jump to: navigation, search

Bjoern's Slides


Extra Materials

Discussant's Materials

The presentation by Alice Wang and Hong Wu

File:Surface Computing 0914.pdf

Reading Responses

Derrick Coetzee - 9/13/2011 13:55:36

Wellner's DigitalDesk was the first prototype of a surface computing platform, using an overhead camera and projector to add computing ability to a tabletop surface and integrate well with physical objects like paper. The ability to interact with portions of paper documents on-the-fly without a time-consuming scanning process is valuable, and is even more feasible today in the presence of high-resolution low-cost cameras, and depth cameras for distinguishing a hand from the background (also making the touch problem easier to solve). Like ubiquitous computing, it shares the vision of augmenting the ordinary physical world with computation.

In many ways, however, the system shows a lack of imagination: it's envisioned solely as a document processing system, and paper documents are the only physical objects it deals with. The ability to compute with arbitrary objects opens up many other possibilities: for example, one could cook on the table while a recipe is projected, or construct a 3D model on top of a digital blueprint. By sharing arbitrary objects, users could collaborate on assembling of objects, as in building a puzzle together.

A limitation of the system is that virtual changes to real objects, such as new text projected on paper, cannot be preserved when removing the object from the table, nor can any material on real objects be erased. The use of a printer to do this is awkward. A new device that could alter the real object from afar would be valuable.

It would be interesting to consider a hybrid system in which an emissive display is combined with a projector, allowing modifications to real objects to be projected while still clearly showing virtual objects and avoiding many shadows.

Han's work focused on the particular problem of multitouch sensing on projected surfaces, which is valuable for large surface computing systems, especially multiuser systems. His solution relied on frustrated total internal reflection (FTIR), a phenomenon in which the total internal reflection of light hitting a boundary between mediums is prevented by the presence of another material at that boundary (in this case, a finger), causing the light to escape and shine out of the material at points where it is touched. This is inexpensive compared to alternatives, requiring only a few basic commodity parts (a camera, a piece of glass or acrylic, light sources) and fits well into large surfaces that already use rear projection.

The use of a high-end workstation for image processing gives little insight into how much computing power is required: could this device be scaled down to low-power applications like digital boards with embedded processors, or even portable devices? One interesting path to making such a device portable would be to place fiber optic cables or a slanted mirror behind the surface, directing all light from FTIR into a much smaller, closer sensor.

The system has a number of disadvantages: it requires periodic cleaning, it has to be used with either the fingers or a special stylus with the desired optical qualities, and above the surface interaction is fundamentally impossible. Unlike the depth-camera based system however, it is able to distinguish touching and slight hovering with extremely high precision, suggesting a possible combination of these methods.

Steve Rubin - 9/13/2011 13:58:19

The two papers for this class were about surface computing. The first is Pierre Wellner's "Interacting with Paper on the DigitalDesk," which illustrates a desktop system combining a projector and cameras with physical objects like pieces of paper and pens. The second paper is Jeff Han's "Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection," which gives a brief overview of a technique for constructing multi-touch displays.

Multi-touch displays were a novel idea in 2005--no one had really perfected the technology yet. Han's intention in writing this paper was to convey the technology he was using, but not to discuss new multi-touch interaction techniques. In that sense, he achieves his goal. The paper is succinct and (I assume, as I'm not a physicist) complete. The technique unfortunately suffers from requiring that displays have a certain depth. He claims that this is ok, as the technology will primarily be used for large displays like interactive walls and tables. This was, to put it lightly, a bit shortsighted on his part. The majority of the multi-touch displays in use today are, by necessity, thin (smart phones and tablets).

Wellner's "DigitalDesk" is difficult to reason about in a modern context. He wanted to bridge the gap between the computer and the desk. His route involved giving the desk computing power rather than giving the computer desk-like facilities. The reason that this is difficult to assess is that he is operating under a huge assumption that does not seem to hold any longer: that people like paper more than computers. The rapid adoption of the Kindle and other e-readers are one example of this faltering assumption.

The example applications in "DigitalDesk" are largely trivial, except for the collaborative document-sharing system. Sharing physical space remotely is an interesting idea, but its implementation in "DigitalDesk" has some serious issues. One issue is that users of DoubleDigitalDesk are given the illusion of shared documents, but they cannot physically manipulate the objects on the other person's desk--they can only write on documents that the other user has decided to show. The bigger issue is one that applies to all of the example applications: there is no sense of permanence in the system. If my colleague digitally writes on my document, I have no way of retrieving her updates at a later time. If I draw something using the copy/paste facilities of DigitalDesk, I have no way to access a physical copy of that drawing. Unless the system incorporates these notions of permanence, the DigitalDesk will only be able to have superficial interactions with paper documents.

All things considered, though--Wellner took a unique approach at bridging the desk/computer divide, and I applaud him for that. Even if his ideas were not successful, the paper encourages us to think about what kinds of interactions we really want between our computing devices and our physical world.

Valkyrie Savage - 9/13/2011 14:22:17

Overall idea:

Multitouch is a more, dare I say it, “natural” way to interact with computer systems; humans were given 5 fingers, so why should we not use them?


The “Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection” paper that we read (by Jefferson Y. Han) was intriguing to me, since it did not seem so much to portray a novel interaction technique (Han in fact mentioned the plurality of papers on the topic already) as it described an implementation of an implementation of a technique. Han discussed at length his influences from papers that people had more or less forgotten about for years. From what I could tell, he was just revisiting them because of the recent advances in computer vision technology, but there did not seem to be evidence that it had helped him; he still admitted that there was no concept of right or left hands, and that there was no concept of which fingers/touch points belonged to which users. There were also other drawbacks to the system, including its not registering gloved users (alas!), its failure to note hover interactions (entirely fair, given the design, to be sure), and its difficulties with dry skin and dirty screens. It also sounded (from the description of the failure with dry skin: that it wasn’t registering a large enough point) as though the screen would have difficulty in registering the touches of children, who are quite a fair chunk of market these days, what with the way that iPads are becoming babysitters.

Before I complain too much about this paper, I would like to say that its title is, reasonably, correct. It doesn’t present anything new, but it does offer a low-cost option of something which is (for all intents and purposes today, though perhaps not in 2005) already fairly cheap. It doesn’t provide an estimate of the total system cost, sadly, which makes it difficult to judge, but the parts he was using did not sound like they were specialized or expensive in particular. He even discussed the repurposing of a fairly ubiquitous vinyl rear-projection screen material.

The DigitalDesk paper was significantly more motivated and less-similar to existing technologies. I liked its ideas about giving paper the power of computing rather than giving computers the power of paper. It seemed very close ideologically to the ubicomp paper we read for Monday that came out of XEROX PARC, in that it hopes to integrate computing into the user’s environment instead of forming the user into something unnatural. The three applications that they selected seemed variedly useful. The calculator app was neat! It combined computer vision (pointing), OCR (transference of digits), and UI (pointing) design aspects, and it seemed like it would be a) fairly intuitive and b) a more rapid way to enter digits for calculations; assuming that the OCRing and computer vision were reasonably accurate, anyway.

I thought that some of their choices were sort of questionable in design, though, too; for instance, this quote made me laugh:

More cameras could easily be added to cover the entire desk, but this has not yet been necessary, because the applications tried so far only use small parts of a document at a time, and sliding a piece of paper into the camera’s window is so easy.

Really? How interesting. I guess it’s great that it’s so easy to interact with this system, but it seems unnecessarily limiting to restrict it to an area (whose size they sadly don’t mention) that is smaller than even a full sheet of paper. As far as other things discussed in the paper, they did mention the limitations related to the computer’s not registering new papers as they were added to the table, and the issue about needing to keep the papers perfectly still. Sounds cumbersome to me, but based on the July 1993 date for the paper, it is perhaps okay. It sounds actually like it could be an interesting direction for current research, now that our computer vision techniques and all are much better.

Yin-Chia Yeh - 9/13/2011 15:05:40

The two papers today are about surface interaction techniques. The DigitalDesk paper presents the idea of providing paper documents some nice properties of electronic documents via a combination of camera and projector. Example applications include calculator (OCR), paperPaint (graphic tablets), and DoubleDigitalDesk (shared editing.) The FTIR multi-touch paper introduces a new way to create an inexpensive, scalable multi-touch surface. Moreover, this panel allows using rear-projection devices so it can avoid the occlusion problem of top-projection devices.

The DigitalDesk paper is really a precursor of other interactive surface papers we’ve read last week. In my opinion those applications are not so attractive because we have other alternatives to perform the same tasks today. However, the concept of providing electronic properties to paper documents is very interesting. I didn’t think of that interpretation of interactive surface before. However, as far as I can think of, most of best properties of electronic documents, such as quick editing or searching, cannot be ported to paper documents. On the other hand, porting the merits of paper documents to electronic documents seems to be easier, especially when considering the technology development progress of electronic paper. Moreover, the cost and technical difficulties of such system is just too high that these applications cannot justify. I believe this is why other alternatives dominate current world instead of this work. Anyway, it’s still a good paper for discovering interesting possibilities and my point of view might be proven wrong in the future.

The FTIR multi-touch paper, on the other hand, aims to provide a very practical technology solution. The system can be easily scale up, sensing is accurate, and it’s inexpensive. It also allows rear-projection so no occlusion problem of top-projection. I have only one question about this paper, how much space they need to scale up the system to a large size. Obviously it depends on the angle of view of camera and projector. One possible approach to mitigate this space problem is to use multiple cameras and projectors to cover the whole area of the sensor; then it becomes a tradeoff between vertical (to the sensor plane) spaces and horizontal spaces.

Viraj Kulkarni - 9/13/2011 16:05:39

The first paper, 'Interacting with paper on Digital Desk', talks about a technique for combining computers and real world desks as an attempt towards integrating computers into our real physical world. The paper, 'Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection' presents a multi-touch sensing technology based on total internal reflection.

The paper treats the subject quite comprehensively. It explains what the Digital Desk is and also lists a few examples of the interactions it supports. Along with the technical details, it also presents observations on how users interacted with the system. It also talks about some implementation issues and drawbacks of the system. There are many other systems similar to the Digital Desk. However, the unique thing about the Digital Desk is the way it approaches the question of integrating paper documents within the system. This is something I did not see in the other similar systems. Another distinction is that the Digital Desk is an attempt to integrate computers into the real world rather than it being the other way round. Although this is a conceptual distinction rather than a real one, it is an approach that is different from the ones that other similar systems adopt.

'Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection' presents a simple and inexpensive technique for sensing multiple touch inputs. As I see it, the major advantage of this technique is its lower cost and scalability. Increasing the size of the interactive surface will not increase the costs as dramatically as it does when it comes to capacitive (or related) touch screens. Another attractive property for this system is that its cheaper in terms of computational power required to detect touch. As per the paper, a touch to the precision of 1 can be detected from a 30fps video on a Pentium 4 workstation - which is low computational power as compared to other camera capture based touch interfaces. One drawback of this approach, however, is the large space required behind the interactive surface. This technique is fit for large interactive surfaces like touch-walls but it won't be a good choice where smaller interactive surfaces are needed.

Alex Chung - 9/13/2011 16:34:54

“Interacting with Paper on the DigitalDesk” explores the application of leveraging the distinctive strength of traditional paper and computer. Traditional paper has the “first to market” advantage with loyal followers who loves to use paper as ways of putting ideas into form. While paper is portable within a short distance, computer is more superior in long distance communication. Computer is also more adapted to making exact copies for iterative development.

There really isn’t any seminal idea from this paper. The system configuration is similar to Wilson’s paper “Depth-Sensing Video Cameras for 3D Tangible Tabletop Interaction”. Instead of detecting 3D objects, it captures images and uses image analysis to detect for characters and symbols using OCR technology. This system also uses project to provide feedbacks to the user.

With a wide range of objects scattered randomly on the DigitalDesk, it is difficult to use shape recognition to distinguish user’s pointing motion with hands and fingers. The paper suggested looking for motion as a more effective strategy, assuming that most objects seen on the desk do not move. They should call it the T-Rex Vision.

While the author believed more sophisticated techniques to track multiple fingers and to recognize gesture control, it has not gotten more sophisticated and similar calibration issues continue to exist today. Generally, the idea is transposing the action in the real world onto the virtual world. The technique requires a reference point and a detect system that has high sensitivity and processing power. I like the proposed calibration technique but this requires the table surface to be cleared for each calibration. Overall, I cannot see any significant contribution to HCI from this paper. Also, I don’t see the difference between “working primarily in the physical world but with access to computer” and the other way around. After all, it is the electrical impulses on our brains that make the physical world feeling tactile and manipulative. Someone with brain disorder would think otherwise.

“Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection” discusses various forms of multi-touch sensing technologies and their technical implementation.

Han then introduces his implementation using rear-projection display and video camera capture. I believe the original Microsoft surface uses something similar. The author believes his contraption has high potential and the only thing that is holding it back is more research on material science.

The positive of this implementation is the lack of shadow casting but the down side is the thickness of the display as the original Microsoft Surface was a giant table. Yet miniaturization is a relatively easy problem to solve.

Microsoft Surface2 has since improved its original design by using IR projection and camera capture to detect hovering and identifying multiple touching points with the individual user.

Laura Devendorf - 9/13/2011 17:50:49

Welner's piece discussed the implementation of a digital desk as a manifestation of a physical object with digital capabilities and Han's article discusses an implementation for multi-touch sensing that can measure the force of the touch as well as allow for rear-projection.

The motivation behind Welner's paper was to make physical objects take advantage technology instead of making technology more like physical scenarios. I think this motivation is very interesting and something that I've been thinking about lately. As a conceptual idea, the digital desk is successful and implementations of similar ideas could easy cite this paper as inspiration, the live scribe smart pen comes to mind. In discussing the limitations of the system at length, he also guides future research in the area of tangible manipulation and computing.

One of the points I would have liked to have seen in the paper is a discussion of the benefits to physical and digital environments and when we might actually prefer one medium to the other. In the calculator example, he assumes that the desk user is slow and clumsy on a push button calculator and thus, pointing at the number would be easier. For someone doing data entry, their muscle memory and familiarity might make them extremely fast on the traditional calculator as they have learned it over time. I think it would be important to discuss the limitations and advantages that physical and digital objects afford and how they relate to various types of users.

While I can't honestly say that I fully understand the FTIR technology that allows for multi-touch interaction in Han's paper, I can say that this particular implementation of it would be very useful in many HCI scenarios. Most promising was the ability to detect touch and force, construct large and small scale implementations and to use rear projection. While it wasn't mentioned explicitly in the paper, it seems like this could implemented on a horizontal surface as well which would allow someone to interact with the multi-touch with less fatigue than a vertical arrangement. In both articles, I enjoyed the extent to which the authors discussed the limitations of the system. One of the most interesting limitations mentioned in Han's paper was the fact the issue of dry skin and dirt. While Han's technological reasoning seemed sound, it could have benefited from some data supporting his statements. I'm particularly interested in the qualitative aspects like how the screen feels or how much it is actually depressed when force is exerted upon it. He mentions it takes about a second to fully restore after touch but I don't recall a mention of whether or not the user would feel the screen give way underneath their fingers. I would also be curious to see what would happen if someone pressed their entire hand, not just fingers.

Amanda Ren - 9/13/2011 20:53:12

The Wellner paper introduces the DigitalDesk, which aims to bring the electronic workstation to the physical desk.

This paper is important because it focuses on a different view of things. It chooses to integrate the computer into the physical environment, instead of improving the electronic workstation to be more like the physical world. The DigitalDesk will project electronic images onto the desk and onto paper, it can read paper documents on the desk, and it can respond to interactions from pointing devices.

It was stated by the paper that two of the issues would include problems with the projected display and problems with shadows cast by the user. It was interesting how the users in the studies were not bothered by it. Another interesting case was with right and left handed users. It agree that it would be a good idea by observing the user's 'handedness' to then adjust projections to avoid obscuring things for the user. when they talk about obscuring selections, however, they mention that they might not need a solution because the users would automatically adjust. It was interesting to see the degree in which users were willing to adjust to use the system. I would have thought these would be issues users would've complained about more, yet they were not bothered.

The Han paper covers technique for multitouch sensing by using frustrated total internal reflection.

This paper is important because it explores a simple and inexpensive way for multitouch sensing. This leads to allowing multiple users to interact at once with a display system. One drawback is that although it senses multitouch, it cannot tell if the points of contact are from the same user, or even if they are from the same or different hands. I thought it was interesting how the dryness of the user's skin greatly affects the response of the sensor. The effectiveness of the system also decreases through extended use because of the contamination of the surface, which seems like an inconvenience.

Allie - 9/13/2011 22:20:20

Jefferson Han from NYU’s Media Research Lab wrote a paper on multi-touch sensing in 2005, 6 years before multi-touch mobile and electronic devices became commercialized. Similar to the In-Depth Cameras papers from two lectures back, frustrated total internal reflection (FTIR) describes the technique in which light encounters a medium, usually glass, such that it becomes trapt at a certain critical angle and enables the finger to perform and manipulate tasks on the glass. Where applicable, compliant surface overlay makes possible sensors responsive to force rather than touch. The technique is imperfect as of ’05, of course, where the medium can be contaminated; or dry skin makes it difficult for the medium to be responsive. I found the Han paper more relatable as the technology described is more palpable to the present day, but its use of depth cameras similar to that of the technology described in the Wellner paper, below.

The digital desk paper by Wellner , published in 1995 speaks of various types of digital desks that enable a ubiquitous experience between the user and the digitaldesk/calculator/paint program/doubledesk. The makers of the digitaldesk attempted to make the desk like a workstation, which is more tactile than “direct manipulation” with a mouse. The paper advocates a non-traditional UI experience: rather than working with the computer which in turn allows us to access the physical world; the digital desk and its various programs help the user with the physical world via its functionalities. The paper relates with the Seminal Ideas “Direct Manipulation Interfaces” paper; but hones in on a single ubiquitous user experience with digitaldesk and its peripherals, rather than ubicomp in at large.

16 years after the publication of this paper, many of the touchpad functionalities as described have since become reality. The digitaldesk calculator, the paperpaint, and the doubledesk are all possible via today’s technology, i.e. Google Wave and other similar collaborative environments may be analogous to doubledigitaldesk. Wellner may not have prophesied solutions to the issues raised, such as adaptive thresholding and calibration, yet what he has envisioned in 1995 has since omniscient.

Hanzhong (Ayden) Ye - 9/13/2011 23:17:52

Reading response for: Interacting with paper on the DigitalDesk. Pierre Wellner, Communications of the ACM 36, 7 (Jul. 1993), 87-96. Low-cost multi-touch sensing through frustrated total internal reflection. Jefferson Han. In Proceedings of UIST 2005. p.115-118. Short paper

In response for Wednesday’s reading materials, I would like to make a summary about the main points of two papers first: the first paper about DigitalDesk showcases an interesting design of office interaction tool, which emphasizes on augmented reality as well as multi-user environment; the second paper introduce a novel approach to realize multi-touch sensing process, which is featured by its several notable advantages.

The notable notion in the design of DigitalDesk is its idea of putting computing devices into office environment to augment the capacity of traditional office desk, rather than the traditional notion of introducing entities in physical world backwards into computers. Such notion is built upon the concept that computing devices should help in physical reality, but not in an opposite direction. Several interesting applications such as virtual calculator, PaperPaint and DoubleDigitalDesk have been introduced, and related implementation issues are discussed as well. Actually, many technological barriers frequently concerned in the paper have been well solved by some recent technological breakthroughs such as depth camera, etc. which makes most of the application prototypes described in the paper not a problem to be implemented. Moreover in my point of view, many other interesting applications should be put forward and implemented to increase the features of DigitalDesk.

The second paper introduces a new way to realize multi-touch sensing process which takes advantage of FTIR (frustrated total internal reflection). Such approach has the features such as simple to implement, inexpensive cost and capacity in scalable installation. Because of its unique visually transparency, when combining with rear-projection display and using complaint surface to overcome several existing shortcomings, FTIR multi-touch sensing technique shows promising performance compared with other strategies. Although there are several limitations which still need to be taken into consideration, FTIR multi-touch sensing is a promising technology in many aspects.

In short, these two paper talk about the conceptual design of an innovative office application as well as a particular implementation issue for multi-touch sensing, and give me a good introductory insight of surface computing.

-Ayden (Sep 13, 2011)

Hong Wu - 9/13/2011 23:33:39

Main Idea:

“Low-Cost Multi-Touch Sensing” proposed a multi-touch hardware. “Interacting with paper on the DigitalDesk” described a prototype to combine the function of the paper and computer by applying camera and projector.

Interpretation: “Low-Cost Multi-Touch Sensing” applied a FTIR technique for multi-touch screen. The advantages of the technique are simple, inexpensive, scalable, and high-resolution. The problems of the technique include: Contaminated surface creates background noise. Dry skin (gloved hand) generates a weaker optical signal. The screen is sensitive to the material. Size is big and it is lake of portability. It cannot determine whether the touches are from same hand, different hands, or different users.

The paper solved some of the problems but the others still bother the system.

“Interacting with paper on the DigitalDesk” ‘s prototype tried to give physical paper some electronic property such as copy and paste. It also suffers from accuracy, user fatigue, dark room environment, lag of the camera, lack of mobility, sensitive to noise and other drawbacks. However, the idea opened a new door for human to interact with computer.

Apoorva Sachdev - 9/14/2011 0:56:06

Reading Response: 13 September 2011 This week’s readings were about the Digital desk and how to achieve low-cost multi-touch sensing through frustrated total internal reflection. The first article describes how we can give computational abilities to physical desk items like paper while the other one focused on the technique of using the FTIR to sense multi-touch and how they tried different surfaces/materials to get more accurate results.

The Digital desk seems to a promising idea and seems to revolve around the same as last time‘s readings where we tried to integrate the computers in our lives rather than integrate our live around computers. It also stresses on maintaining the tactile and feel of the paper. I thought that they could have performed more extended user studies to see how natural or intuitive the interface. The idea of projecting multiple “digital desk” on top of one another might be a little distracting and cluttered. Also they seem to suggest that a feedback loop will be prevented by only projecting the “other digital desk” very lightly but I am not sure how it affects the user experience. They could also look into subtraction techniques and take out the foreground before relaying the image back. Some of the features they implemented were useful like calculator; this could be expanded to using dictionaries (highlighting a word and its meaning being projected) or drawing instant graphs by just selecting the information. Also using projectors is cumbersome due to lighting issues and shadows. So maybe this could be implemented using a multi-touch screen or a backlit projector.

The paper describes the FTIR technique and explains how it works. It is useful because using this technique one can create large touch screen displays cheaply. The described some of the problems of this interface like contamination over time due oils and sweat leading to inaccurate readings. I would have liked if they had provided some more quantitative results on how it compares with capacitive multi-touch technology. How precise are the detection points? Also, the videos for the system show that it works pretty smoothly, but it would be good to get some user-opinions on it. I skimmed through the optional reading and was a little confused with the different comparisons done and explanations provided. Particularly touch tablets vs touch screens. So it would be great if we had some time to breakdown some of his definitions and see how the class perceives the differences.

Sally Ahn - 9/14/2011 1:38:36

Han's paper presents frustrated total internal reflection (FTIR) as a promising approach to multitouch sensing technology. It describes both the implementation details as well as the pros and cons of FTIR sensing.

Han combines FTIR, an old technique in robotics, with modern image procession and computer vision techqniues to improve both the cost and resolution quality of multi-sensing technology. This offers a relatively cheap way to improve touch-sense resolution. Thus, I think this paper makes an important contribution to research in HCI for collaborative environments by offering more variety in the types of gestures that can be supported. Nevertheless, challenges remain: as Han mentions, FTIR does not offer higher-level information about the sensed touches (e.g. "hover" proximity, touch connections to same hand/user).

Wellner's "DigitalDesk" is a thorough exploration of how the user may gain the benefits of both electronic documents and traditional paper and pencil on a single device. Although some parts of the discussion such as OCR and image processing technology is now somewhat outdated, the paper nevertheless contributes a full description of interaction design details, such as handedness and occlusion, that any modern touch-sensing surface must address. Considering the year this paper was published, I was surprised by how much it resembled the Microsoft's Surface in terms of functionality.

One criticsim I have for Wellner's paper is its focus on "enhancing the paper" with computer qualities rather than bringing paper qualities to the computer. I felt such a distinction was like comparing mixing blue into red rather than red into blue to make purple. The focus should be on why users prefer certain qualities of paper over computers and how we can replicate these qualities in any form. As long as the prototype embodies those relevant qualities, it shouldn't matter whether the resulting prototype bears more resemblance towards the paper or the computer.

Moreover, Wellner argues that "since we use pens and fingers to interact directly with paper on our desks, we should be able to interact with electronic documents on the DigitalDesk in the same way." This statement hinges on the vaguely defined term, "directly," and the assumption that this "direct" interaction should be attained in the same way as before. I think this approach discourages us from exploring novel interaction techniques. It is true that we are used to using pens and paper on our desk, but that does not necessarily mean that these are the best tools and techniques for accomplishing the tasks (e.g. recording and retrieving data) we undertake with these materials. In my view, the focus should be more on analyzing existing interaction techniques for their efficacy in accomplishing specific tasks, then separating out the fundamental qualities of those interactions (which may or may not be dependent on the physical properties of the existing tools), rather than trying to preserve physical qualities of the old tools.

Suryaveer Singh Lodha - 9/14/2011 1:57:36

The digital desk - This paper talks about the aspect of Human Computer Intercation, where instead of thinking about making humans work in a computer world, the author is interested in approaches via which we can make computer work in the physical world. Here, a prototype of such kind of a system is explained . The digital desk makes the physical desk more like a workstation and supports computer based interactions with paper documents. The tasks explored in detail are - copying number from a physical paper to computer's calculater, replicating part of sketch drawn by hand on a physical paper (with a drawback that the paper must be kept motionless) and remote shared editing of paper documents. The setup requires a camera, projector, a microphone (to detect taps by user). As in the digital desk, as user will behave just as sitting on the physical desk bumps, accidental taps while thinking about something etc can totally confuse the system. Also the author couldn't have used a touch screen because by design, its expected of the user to rest his hands/elbows on the table top. In that case a touch screen might not be the best idea. One of the problems with output is presence of shadows.Another issue is that projector doesn't work well with bright lights and not all surfaces make good screen. The best combination is to have a white projection area in a room lit with normal fluorescent lights. To capture data from the desk 2 cameras are used - oneis zoomed in close to the desk to get a high quality image. This works well when we are only concerned baout small parts of a document at any particular time. The thresholding process itself is not too well developed and sometimes depends on user feedback to correct unrecognized characters. The prototype did not work well for left handed people and also had issues with self occlusion of paper on desk. It felt to me that a lot of technologies used in the system were not at their best state of development and the prototype had quite a lot of problems which would make it inconvinient to use on a daily basis. BUt based on the fact that this was done in 1993, it not looks very impressive and interesting but also talks about a very different approach to achieve better user interaction whihc has its own pros and cons.

Low cost multi-touch sensing through frustrated total internal reflection - The paper describes the technique to enable high resolution multi touch sensing on rear projected interface surfaces based on Frustrated Total Internal Reflection (FTIR). The prototype uses an acrylic sheet with polished edges as an optical waveguide, instead of glass (as glass has poor optical transmittance and is easily scratched).The sheet is edge lit by infrared LEDs and a digital camera with matching band-pass filter is mounted orthogonally. Only simple image processing operations are required, so it doesn't require huge processing resources.As this is an poptical system, it is susceptible to harsh/extreme lighting environments. The problems are that it cannot detect any of the following - hover state, if fingures were from same hand/two distinct hands or two distinct users.

Donghyuk Jung - 9/14/2011 2:01:36

  • Interacting with paper on the DigitalDesk

In this paper, the author built DigitalDesk supporting computer-based interaction with paper documents. People can utilize paper documents to perform some useful tasks (coping numbers on the paper into a calculator, replicating part of a sketch, or remote shared editing of paper documents) by using DigitalDesk. The technologies the author used are not similar to “Memex” but the general concept of using a desk for information processing is same. The author said that we interact with documents in two separate worlds: the electronic world of the workstation, and the physical world of the desk. Therefore, the author tried to design office system, which can provide the best of both types of documents.

I think it was a good try but the author could not expect the direction of technological progress. For example, new collaboration software like Google Doc or MS SharePoint enables people to cooperate the same task from a remote location due to the advancement in computer network and software technologies. In addition, the author failed to expect future office system. Cloud computing and mobile devices make people can work anywhere and anytime. In some workplaces, there is no fixed seat for one person to increase space efficiency. The bottom line is that people have become pretty used to work by using both digital and analog documents simultaneously. Especially, both digital and paper documents in office environment are not common Word documents or scratch papers. As written contracts in a company have legal force, all documents have their own level of authority and accountability. These properties of documents reinforce trade-off relationship between digital and paper documents. In my opinion, we need to build tools that can strengthen the strength of each document rather than making up for the weakness.

  • Low-cost multi-touch sensing through frustrated total internal reflection

From the non-experts’ point of view, this touch device is useless due to many drawbacks. First of all, the system cannot provide proximity information and don’t have any ability to label or classify each point of contact. In order to detect and process multiple simultaneous points of contact, these defects must be solved. Secondly, installing video camera and projector behind the screen require a huge amount of space so that this architecture must encounter space restraints. I think only tabletop structure can support the whole system. Nonetheless, this study has a significant discovery for future researchers because they might avoid the same problems with this approach or design new architecture that can overcome some drawbacks.

Ali Sinan Koksal - 9/14/2011 2:25:20

The first of this week's readings, the Digital Desk, aims to provide a work environment that brings together electronic resources with more conventional and easy to use materials like paper and pen, by enhancing physical resources with properties of workstations.

This paper seemed to follow Weiser's vision: integrating computing into our everyday life instead of simulating conventional concepts virtually on a workstation. I liked the possibilities offered by such a system, such as smoothly automating tedious tasks of digitizing physical information, and especially offering a collaborative work environment using desks.

However, we can see that such a system is still not widely adopted. One reason could be that these are rather bulky systems that we cannot carry around. Moving around and working with Weiser's pads offers a greater freedom and flexibility. Furthermore, a more crucial problem seems to be the difficulty of working with real paper in an environment that must be rather dark for projection purposes, and as we have already discussed when reading LightSpace, the problem of working in dark places altogether. The authors deem mouses as the opposite of direct manipulation interfaces, meanwhile these devices have an advantage when it comes to not occluding the display area. Overall, this was a useful investigation on merging the two kinds of media: it is still the case that people extensively print material for reading convenience, we are still far from "paperless" offices.

The second paper, a technical one, revisits the Frustrated Total Internal Reflection (FTIR) to build large, inexpensive, highly precise multi-touch sensing surfaces.

The idea is to sense when and at which point precisely a contact that disrupts total internal reflection occurs on a surface. The requirement of having instrumented surfaces may not be a great disadvantage: combined with rear-projection, these surfaces avoid the problem of users getting in a projector's way (as it is the case for systems using un-instrumented surfaces). However, this technology lacks the capacity to associate touches with users, as can be done using depth cameras, and therefore limits certain possibilities of interaction. Today, capacitive sensing seems to have overtaken with the use of mobile devices, but for larger surfaces, FTIR may be preferable (if we need touch sensing at that scale, which is another question).

Peggy Chi - 9/14/2011 2:30:14

This week we will discuss surface computing and its applications: The classic paper in 1993 introduced DigitalDesk, a system that aimed to combine the advantages of physical and digital worlds to create "augmented reality." Its author Wellner deployed a projector and a camera to enhance a traditional desktop, and proposed three examples that enabled users to interact with physical paper and pen but enjoy the computation power supported by recognition. Han in 2005 then shortly summarized the state-of-the-art of multi-touch sensing techniques and applied the existing FTIR (Frustrated Total Internal Reflection) method to this domain using LED light, acrylic panels, and a video camera.

So far we've explored topics including input devices (mice, keyboards, depth cameras), design principles (direct manipulation), and design space (AR, projected surfaces, augmented rooms, ubicomp). It is interesting to see how these ideas are connected to improve user experiences. Maybe we can align these concepts into one dimension, from the left as a purely digital world (VR), moving toward the right with more and more physical attributes - interfaces accessed by physical input devices (PCs, phones, tablets), adding some physical feedback (screen/mouse/pen with tactile reaction), somewhere in between (AR, augmented surfaces), then digitalized physical props (e-ink, computing fabric), and finally invisible or ambient interfaces that blend into our daily lives (ubiquitous computing). It seems that computer scientists and HCI designers have been moving between the two extremes (digital world vs. physical world) with long history. The concept of DigialDesk (considering both physical and digital attributes), for example, was brought up to somewhat oppose to direct manipulation (bringing physical features/metaphors to the digital world). What pushes people forward? Due to the fact that there is no one perfect solution to fit all the situations. Based on our developed skills for lifetimes, we are intuitive in manipulating physical objects and using our body/five-sense as input and output. We prefer those features that already become natural to us, but when interacting with a digital world, to what degree should we preserve? It is still an open question, but by accessing more and more related topics, I believe we are getting closer to the core discussion.

Galen Panger - 9/14/2011 2:33:02

The DigitalDesk project blends real and virtual documents on one large desktop surface using a projector and cameras mounted above. The ideal is to meld the benefits of a paper-based desk with our virtual workstations. Computer functions, like copy-and-paste and mathematical calculations, can be performed on and with paper documents, and shared desktops can be maintained with remote colleagues. There are some obvious limitations to the state-of-the-art in 1993 when the article was published, including one that would feel pretty painful to me—you can’t move your paper documents once they’re placed. But overall, this approach intrigues me, and reminds me of Weiser’s piece from last week (as well as LightSpace). Rather than bringing users constantly into the virtual environment, the DigitalDesk brings computing to the physical environment. Personally, I would love to be able to point to a list of numbers and have the total projected below; any improvement in the interface between paper and electronic documents, I think, would be a huge boon to society. Wellner is right—the barriers are too high. (By the way, I do appreciate the dig the authors took at the way “direct manipulation” is conceived, on page 92.) I would have preferred to see Wellner present more data from user studies, because in fact, I think the dual-use mode could be confusing. We’re used to interacting with paper objects and computer objects, and their circles of functionality overlap significantly (though not entirely obviously). What factors seemed to determine whether users used a pen or marker to directly mark the physical paper, versus when they used fingers or a stylus to interact with the information? And unfortunately, even though the DigitalDesk helps meld the two interaction modes, there are still big barriers, for example when one wishes (as Wellner notes) to transfer a digital image onto paper (you can’t just do it at the desk, you have to print), or undo a physical marking made to paper, which of course is possible on the digital representation but—oops!—not on the physical representation. It’s fascinating to think about, though. Our other piece, on frustrated internal reflection, doesn’t make complete sense to me because of my ignorance of the physical properties of light. Does touch disrupt the light wavelengths because of the touching itself, or does it have something to do with the way the finger blocks the light being transmitted through the transparent medium? Not sure. Also, not knowing very much about touch techniques (for example, multi-finger capacitive), I’m not sure how to interpret the significance of this article. But there are some cool alternative ideas presented, such as the tactile sensors used with robot grippers that use air between surfaces to detect force. Makes me think we could do more with force—though our smartphones now mostly all include accelerometers, which can detect force.

Yun Jin - 9/14/2011 3:03:32

Interacting with paper on the DigitalDesk: The first paper discusses about DigitalDesk. DigitaldDesk is a real physical desk and it is enhanced to provide some characteristics of an electronic workstation. By using this desk, paper gain electronic properties and electronic objects gain physical properties. To better demonstrate this new technology, this article also introduces implementation and some applications towards it. The DigitalDesk provides an environment that papers gain electronic properties that allow it to overcome some of the physical limits. Take the calculator on DigitalDesk as an example, it is more rapid and direct to do some calculating based on interaction with selected paper documents than using common calculators. And paper print is a unique interaction technique that is different from standard “select and paste” and it takes advantages such as using both hands for manipulating and pointing as well as the superimposition of paper and electronic objects. Moreover, by using double digital desk, people could share paper documents from different continents, which makes the DigitalDesk convenient and human to people far away. Thus, I think DigitalDesk has the following virtues: fast, direct, convenient, and human. Despite of so many advantages of DigitalDesk, there are also some limits and disadvantages of it. With overhead projection, one potential problem is shadow, which would make the projection obscure at sometimes. And the brightness of the room can also be an issue with projection. That is, a bright desk lamp or direct sunlight can make the display unreadable and limit the desk’s usability in some settings. An additional problem is that not all surfaces make good screens. Taken these limits and problems into account, we know that what the next step to do is to diminish or avoid such problems. In conclusion, DigitalDesk is an innovation of making desk more like workstation and it supports computer-based interaction with paper documents. In the past, people focused on the integration of paper document into the electrical world of personal workstations. DigitalDesk is another angle of view to human computer interaction and it really contributes. Thus, this article helps us to think more about some unusual ways to the human computer interaction.

Low cost multi-touch sensing through frustrated total internal reflection: This paper introduces a technique for multi-touch sensing of high-resolution on rear-projected interactive surfaces based on frustrated total internal reflection, which is simple, low-cost and scalable. Also, it discusses the previous applications of sensing, provides implementation details and talks about the results from initial prototype and describes future work. This technique provides full imaging touch information without occlusion or ambiguity issues. By using this technology, the touch sense is zero-force and true. It can distinguish touch from a slight hover accurately. And it samples at both high temporal and spatial resolutions. More importantly, it is inexpensive to construct, and is scalable to much larger surfaces. However, there are also some limits to the technology for multi-touch sensing. First, the surface of the touch should be planar. Another drawback of the approach is that it requires quite a lot of space behind the interaction surface. Moreover, it remains be influenced by harsh lighting environments. Finally, the greatest drawback to the approach is how little other information it would provide about the tactile image.

Cheng Lu - 9/14/2011 7:40:48

The first paper, “Interacting with Paper on the Digital Desk”, describes the way of interacting with documents. Basically, there are two ways of interacting techniques: the electronic one and the physical one. Each one has their own advantages, but choosing in one world means forgoing the advantages of the other. Therefore, a great challenge to office system designers is to provide the best types of documents. Although electronic documents provide valuable functions such as spell checking, people must sacrifice highly developed tactile skills with paper in order to access these functions. With these differences, we then have two desks: one for “paper pushing”, and the other for “pixel pushing”. Instead of replacing paper with computers, which is more typically, the author of this paper states that they could enhance paper with computation. The aim of Digital Desk is to make the desk more like a workstation. On this desk, papers gain electronic properties, and electronic objects gain physical properties. There are many possible ways to enhance ordinary paper documents on the desk, especially those interesting applications. After talking about the applications built on the Digital Desk, the author further discussed the implementation issues. At last, user experience experiments found out that generally people are comfortable and natural to work with Digital Desk.

The second paper, “Low-Cost Multi-Touch Sensing Through Frustrated Total Internal Reflection”, describes a simple, inexpensive, and scalable techniques for enabling high-resolution multi-touch sensing on rear-projected interactive surfaces based on the frustrated total internal reflection, a phenomenon familiar to both the biometric and robot sensing communities. While touch sensing is commonplace for single points of contact, it is still difficult and/or expensive to construct a touch sensor that can register multiple simultaneous points of contact. Then the author discussed in detail about the FTIR sensing techniques and the related implementation. The greatest drawback of this approach to be how little other information it provides about the tactile image. The system provides no proximity information, nor any ability to label or classify each point of contact. These drawbacks may become the future breakthroughs and need more works to be done.

Vinson Chuong - 9/14/2011 8:09:47

Wellner's DigitalDesk is an attempt to provide a direct interface for editing documents on the computer, moving this type of interaction closer to that of writing on physical paper. Han's FTIR touch sensor offers a low-cost way to sense multi-touch input over potentially large surfaces.

In a world where paper and physical writing are still part of our daily lives, the DigitalDesk offers a compelling way to unify the management of digital documents with the management of physical documents, while preserving many of the advantages of both. Wellner focuses mainly on enhancing the act of writing on physical paper with features typically found in today's productivity suites: data and text extraction, communication and collaboration, annotations, and versioning--a physical copy backed by a digital representation superimposed on that physical copy. We see some products today which offer a subset of these features, in particular digital pens, which create a digital copy of whatever is written on paper.

Wellner glossed over an unintended use of the DigitalDesk during user testing, that is, editing purely digital documents without the presence of a physical copy. This is seen in graphics tablets and tablet computers today, where instead of writing on physical paper, people write directly on a touch screen. Instead of having any physical copy at all, tablets yield all the advantages of document management on computers through an interface that mimics that of traditional writing. Wellner's ideas have been mostly realized, although, perhaps not in the form he envisioned.

Han's FTIR touch sensor provides features that are very similar to those provided by depth cameras. The main difference is that depth cameras have to infer a touch event by comparing the depth of a finger with that of the surface, whereas a FTIR sensor relies on data transmitted by the surface itself at the point of contact; so, it seems that a FTIR sensor would be able to offer better accuracy. Another difference, beyond just touch interaction, is that depth cameras also offer context as they can track a user's entire body, whereas FTIR sensors only offer touch and possibly hover sensing. Overall, I think that depth cameras are superior to FTIR sensors.

Jason Toy - 9/14/2011 8:56:05

Interacting with Paper on the Digital Desk

In this paper, Weller describes building the DigitalDesk which makes the desk more like an electronic workstation to draw upon the advantages of both the electronic and physical worlds. The DigitalDesk uses projection, touch-interaction, and cameras to create documents that bridge the gap between the virtual and physical workspaces.

This paper presents an interesting new system to take advantage of positive characteristics of both electronic and physical documents. Physical documents, for example, are easy to manipulate with pens and other tools. On the other hand, electronic documents have advantages such as searches, spell checks, and ease of sharing. One strength of the DigitalDesk is the ease in which you can use values from physical documents in the virtual world, taking advantage of applications such as the calculator. The other is the ability to take these physical documents and write on them, and share them with others in a collaborative manner. This paper presents a system similar to Microsoft Research's LightSpace system, with use of a projector and camera system to allow for interactions with your workspace.

The system has issues similar to that of the LightSpace system. While the motivation for the DigitalDesk appears to be use in the workplace, it seems that the setup for a desk would not only be expensive, but space-consuming. From a technological standpoint there are problems as well. The desk suffers from the problem of obscuring selections, in which the hand or another object covers a part of the document you are working on. In addition, the projection of changes people make onto the physical paper itself means that the paper cannot be moved for the duration of its use. I question why a virtual copy of the paper was not made and projected, such that the paper can be removed or moved around. Limitations like these make the system while well-motivated, unfeasible as presented. However, the paper's direction is a good one. More research needs to be done on OCR or other methods to make the conversion between physical and virtual documents plausible. Another approach, as this paper describes, would be to bridge the gap between the two types of documents to achieve the benefits of both. An interesting idea comes from the paper's discussion on user response, which could lead to an avenue of research: maybe this type of input device may be suitable for some more than others. Take for example a computer programmer, who would likely be happier coding with a keyboard vs an architect or a cad designer who deal with both paper and electronic documents all the time.

Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection

Jefferson Han describes using frustrated total internal reflection to detect multiple touches to a screen at a time. The system works by placing a projector and a video camera to both display the computer image on a screen and capture the refracted light which indicates a touch at the same time.

This paper presents a new system to take advantage of multi-touch sensing. This is similar to the depth camera touch sensor built by Andrew Wilson of Microsoft Research. The disadvantage of this technique in comparison to the depth camera, is the limited set of materials that can be used as a reflection surface to achieve results. The advantage of this technique is that it relies a lot less on the guess-work and approximations that the depth sensor uses to determine what constitutes a touch. Future research can be done on determining combinations of light and materials that may work better and attaching this system to an lcd display. Having a single portable system that can be sold could be an advantage over the depth camera which needs to be attached and set up on a surface.

The argument for this system is sound: total internal reflection is a well documented phenomenon. In addition the system could be useful if combined with lcd screens. The current multi-touch systems are capacitive screens which are not only more expensive than resistive screens but limited in that you must use a finger to activate it. FTIR screens on the other hand may be activated with gloves or stylus, though it is not guaranteed so: this is something that could be researched and improved. A disadvantage of this system is that there is a limited set of materials that could be used to manufacture screens with. I actually think it would have been nice if the paper did go into a discussion of interaction techniques that the system enables. Is it possible this technique provides not only a cheaper solution, but better performance in comparison to other techniques? Does performance vary depending on the task at hand?

Rohan Nagesh - 9/14/2011 8:59:43

The first paper discusses a novel approach to surface computing, known as the Digital Desk, which aims to make the desk more like a workstation rather than the other way around. The second paper discusses the use of FTIR (Frustrated Total Internal Reflection) to create multi-touch sensing.

I found the first paper's topic to be quite interesting. The idea of surface computing is definitely appealing, considering the paradigm shift towards mobile, the success of touch-based tablet devices, and the need for everywhere, anywhere data and applications. I had a few points of contention though. First, one of the interesting points the author brought up was that the learning curve for some computer-based applications is immense compared to real-world applications (such as holding a pen or pointing). While this is true, I do believe that recent generations have grown up with computers and have mastered tasks even previous generations haven't yet in certain cases. In fact, I could imagine that for this group of users, the Digital Desk may actually have a steeper learning curve than the familiar computer applications such as Excel and Powerpoint, which everyone nowadays is growing up with. Second, the use cases mentioned by the paper (calculator, paint, etc.) are great, but I believe to really achieve mass adoption, the common use cases must be accounted for. There is currently not enough value added for me as a user to go out and purchase this device. The paper doesn't even delve into the cost aspects of this desk, but one must consider this in figuring out if the desk has any real-world potential.

Overall, the paper presents a novel approach to a futuristic world of surface computing, but more experimentation is needed to see if the technology will get adopted.

With regards to the second paper, I was a bit disappointed that the paper focused mostly on the technical details as opposed to potential applications of the technology. I believe that the limitations of the system in that it cannot detect two users, where the touch points are coming from, etc. are pretty cripling for multi-touch displays, and traditional tablets/lower-scale devices are more comfortable and reassuring to current users.

Shiry Ginosar - 9/14/2011 9:11:26

These two papers describe two different approaches to constructing input interfaces in the form of "multi-touch" surfaces. While the DigitalDesk uses a regular desk with an overhead video camera and thus allows the user and the system to interact with physical objects like paper, the FTIR system focuses on simplifying multi user and multi finger capabilities through the use of novel materials and techniques.

It is interesting to compare the work of Wellner to that of Wilson's depth camera interfaces. While the approaches are similar, except for type of camera used, there are two interesting differences between the two. The first difference is a technological one. Depth cameras allow for easier detection of user body parts and a simpler emulation of touch without the need for the additional microphone listening for a tapping sound. The second difference is a contextual one. While Wellner was primarily concerned with bringing computation into the physical world of paper, the focus has shifted in Wilson's time. While some of Wellner's suggested applications remain to be desired (like auto-copying numbers from a sheet of paper), others have become obsolete due to the much higher resolution available in computation and displays today (such as grabbing doodles from paper instead of just drawing them with a stylus on a touch screen to begin with). Perhaps that means that we are nearing that time when paper becomes obsolete as Wellner puts it? I write this, of course, while holding a fully marked printed copy of the paper in my hand...

Compared to the involved computer vision algorithms required for both regular and depth camera desk setups, the processing required for the FTIR technology is much simpler and seems like a promising avenue for large multi touch displays given that the perfect material for this application is found. However, in order to be useful in practice it seems like there would need to be a more compact solution than placing a camera far behind the display. While the cameras in the previous setups are overhead and therefore do not deem part of the room unusable, this setup literally requires users to have dead space in their working environment.