- 1 Lecture Slides
- 2 Extra Materials
- 3 Discussant's Slides and Materials
- 4 Reading Responses
- 4.1 Thomas Schluchter - 11/16/2010 8:31:04
- 4.2 Siamak Faridani - 11/16/2010 9:30:31
- 4.3 Charlie Hsu - 11/16/2010 11:31:02
- 4.4 Krishna - 11/16/2010 12:41:58
- 4.5 Airi Lampinen - 11/16/2010 16:50:30
- 4.6 Aaron Hong - 11/16/2010 17:25:08
- 4.7 Arpad Kovacs - 11/16/2010 17:35:58
- 4.8 Luke Segars - 11/16/2010 17:37:01
- 4.9 Luke Segars - 11/16/2010 17:57:43
- 4.10 Dan Lynch - 11/16/2010 18:03:07
- 4.11 David Wong - 11/16/2010 18:21:28
- 4.12 Thejo Kote - 11/16/2010 18:26:51
- 4.13 Bryan Trinh - 11/16/2010 18:40:25
- 4.14 Matthew CHan - 11/16/2010 18:42:26
- 4.15 Anand Kulkarni - 11/16/2010 18:45:30
- 4.16 Shaon Barman - 11/16/2010 18:48:39
- 4.17 Richard Shin - 11/16/2010 18:57:32
- 4.18 Matthew Can - 11/16/2010 18:58:55
- 4.19 Linsey Hansen - 11/16/2010 19:02:02
- 4.20 Kenzan boo - 11/16/2010 19:08:00
- 4.21 Aditi Muralidharan - 11/16/2010 19:12:01
- 4.22 Drew Fisher - 11/16/2010 19:57:14
- 4.23 Brandon Liu - 11/16/2010 19:58:16
Discussant's Slides and Materials
Thomas Schluchter - 11/16/2010 8:31:04
The paper presents the evaluation of a system that integrates the search for code examples directly with the development environment and utilizes the code's context to formulate richer search queries.
This is a very incremental innovation. Nevertheless interesting as it takes into account changing practices that the ubiquity of search has brought about. To those that look at IDEs as hand-holding it will be just another distracting gimmick. To me, it is useful especially for people who frequently deal with different languages and different frameworks and don't have time to develop deep expertise in either. This would be the case for anyone who is mainly programming for prototyping purposes and takes an opportunistic approach to choosing tools.
I imagine this to be a great way of learning and exploring a new language or framework. In this particular case though, the potential efficiency gains would be less salient because the user would not be 'coding for speed'. The fact that 3 out of 4 usages do not involve copy-and-paste hints at people using it as either a syntax memory jog or a cookbook of programming concepts.
I liked their combination of qualitative and quantitative evaluation in combination with a longitudinal study.
What would other programmers do?
HelpMeOut is a tool that attempts to make error messages more useful by difftracking bug fixes and abstracting them into solutions that are presented to programmers when they encounter the same problem.
Very interesting concept, and an excellent example of how to make the crowd work for a system without intruding on them. It stands to reason that only very few errors will be unique, and the vast majority are common coding mistakes made by many. Therefore, I agree with the idea that a longer deployment to a broader user base would improve the ratio of useful to useless suggestions.
An interesting question is whether this kind of error correction has any effect on learning. Is it a crutch for those who couldn't walk otherwise or is it a crutch that lazy programmers rest on? For novice programmers I see a definite advantage in just moving forward by removing obvious errors. Novices' tolerance for frustration is more likely to be low, and one of the more frustrating aspects of programming is hunting down mistakes that turn out to be trivial. At the same time, one could argue that diligence and good programming practices will not emerge unless one faces one's own mistakes and learns from them.
For experienced programmers (at least if they are familiar with a language), the default cryptic error messages will give enough information to quickly glance at the problem area and fix a typo. Since the errors caught by HelpMeOut are on the grammatical level, not on the semantic level, the problems that this user will more often encounter remain unaddressed: problems with program logic and the modeling of relationships.
HelpMeOut is a valuable tool for beginners, but I'd like to see more work on how it influences their learning process.
Siamak Faridani - 11/16/2010 9:30:31
Example-centric programming: integrating web search into the development environment
Authors start by pointing out that online code examples play a crucial part in code developments. Especially in language families like Flex and .Net that have huge libraries and namespaces examples can save developers‚Äô time. The paper is about a Flex builder plug-in called blue print that enables developers to look for examples and load them into their codes as templates. Authors show that by integrating the search tool into the IDE they could decrease the cost of searching by programmers. By interviewing top users of the plug-in authors find 3 themes for using this plug in (1) example-centric results outweigh the drawbacks of missing context. (2) Blueprint is symbiotic with existing IDE features. (3) Blueprint can clarify existing knowledge and remind users of forgotten details. The paper also contains very useful details of the implementation and how authors developed Blue-print. I feel the evaluation is not fair though, a developer uses different sets of tools, like IRC, stackoverflow, and websearch. Additionally the controll group was not allowed to use the google customized search that was used in the blue-print experiment. I personally really like the exploration that they performed on the user logs. This kind of experiment gives a lot of insight into how people use the system and how one can improve it.
What would other programmers do: suggesting solutions to error messages
The first article was about an SE tool for the coding phase and this article is about an smart SE tool for debugging. Authors describe the development and evaluation of HelpMeOut which is a tool for quickly pinpointing and fixing the problems in the code by looking into a database of errors and fixes. HelpMeOut tokenizes the code by passing it through a lexical analyzer and looks for similar patterns and fixes. I personally liked the discussion section. Authors have addressed many of the concerns that I also had during reading the paper for example with tools like this what will happen to the privacy, what about plagiarism? Specially commercial software development companies have strict rules for their developers in many cases developers cannot contribute to open source projects to avoid the risk of inserting similar code structures. Finally I am wondering if anyone has thought about building a Soylent like interface for code development and if that is even possible at all.
Charlie Hsu - 11/16/2010 11:31:02
This paper introduced Blueprint, an user interface for accessing online code examples from within the development environment, integrating web search and filtering and presenting results in an example-centric manner. The paper hypothesized that Blueprint would require less click-throughs, return correctly formatted code more often, and induce more repeated queries than standard search engines. The authors did some small-scale evaluation in the lab, and some large-scale evaluation deployed publicly.
I felt that the Blueprint server was particularly well done. Caching entries and re-crawling them is a great optimization, and if the authors were to implement the suggestion of rating example code, the database of example code snippets could become extremely powerful. However, the problem of dirty data was extremely apparent in the Web, the source of all code snippet data. As the authors noted, code in forum and blog post comments have low reliability compared to API pages, and formatting issues such as line numbers or typos affect the code parser. Furthermore, once a segment of HTML is identified as example code, extracting it is still difficult due to a lack of a standard "code->description" HTML element (or even a code element). Hacking through these difficulties using classification via heuristics and looking for DIVs/HRs separating subsequent segments less than 250 words may work, but in the future, huge gains could be made with the use of microformats and better Web architecture. Research into making Web content more machine-readable certainly poses an HCI question I hadn't though of in this context before: how do humans have to alter the way they produce content to make it optimal for the computer?
Though integrated web search for example code snippets is a great idea and eases a seemingly common but unsupported task for programmers (searching for code on the Web), the nature of Web content leaves me slightly skeptical about the quality of retrieved data. The suggestion on rating snippets seemed to me like an essential feature to implement later, as an extra filtering mechanism. Though example code is often available and the point of focus on many official documentation websites (Python standard library, JQuery, as examples from personal experience), documentation generated by tools such as Javadoc often do not have example code (certainly not written within function headers). Example-centric programming needs to be supported on both sides, being accessible to the user and being generated from official sources.
What Would Other Programmers Do?
This paper presented HelpMeOut, an interface that helps programmers debug code by suggesting crowdsourced solutions that have been applied in the past. HelpMeOut is integrated into the IDE, allows users to dynamically change their source code according to the suggestions, and offers an interface for giving feedback on suggested solutions. HelpMeOut was evaluated in two workshops with graduate students from an Art & Design school, with most testers being novice programmers.
Ultimately, HelpMeOut is attempting to ease and integrate the process of searching for example-based code solutions on the Web. Certainly the social aspect of code and the Web here become valuable; many problems regularly tackled by programmers have been solved before, evidenced by the software engineering philosophy of reuse. Using both error message and source code context is one of the primary technical contributions of the paper, and this method effectively filters and pinpoints a small set of correct solutions. Using string edit distance with lexical analysis on fixes and stack trace similarity on runtime exceptions are logical and innovative ways to infer relevance.
On the surface, HelpMeOut sounds great for novice programmers; automatically detecting syntax errors, offering solutions for simple logic flaws in code, and giving a crowd-backed, intuitive interface for making changes. Even for expert programmers, a quick reminder from HelpMeOut on a minor syntax error will reduce cognitive load and make programming a more painless task. HelpMeOut might even be enhanced in a few ways: a general syntax problem repeated against many lines of code could be detected and handled all at once (i.e. checking to see if the syntactically alike erroring line of code shows up in multiple places in the file, and offering to edit all of them simultaneously).
Krishna - 11/16/2010 12:41:58
The authors describe HelpMeOut, a crowd source based framework that suggests corrections and solutions for programming errors within an IDE. The architecture is pretty straightforward, upon hitting a compile time or run time error, the framework tracks source code changes before and after the error is fixed. The changes, information on the error context are stored in a database and can be possibly annotated and enriched by comments, explanations and ratings.
When an users encounter errors in their program, they can query the framework for possible suggestions. The framework first finds a set of candidate error matches by using a string matching procedure. It then employs a set of heuristics to rank the matches based on how similar user code segment or error stack trace is with the corresponding ones in the database. Matches are further ordered based on relevance scores provided by other users.
Given that error messages are more or less standard makes this framework robust. However, I have doubts on how the framework handles runtime errors - especially when fixes have semantic underpinnings. As an example, a programmer might change some semantics in his code to avoid a 'divide-by-zero' error that involves a completely different code execution path. Such fixes do not seem to be captured by the suggested heuristic.
I particularly liked the evaluation section - a well thought out evaluation plan and honest discussion of results. An important question is whether novice programmers can make sense of the suggestions - something they themselves mention at the end of the paper.
The authors argue that searching for code is different from generic search - user query should be augmented with the context which may include the semantics of the code, the development environment, etc. They propose a framework that extends existing set of search engines with features that promote search via example code and features that integrate code from search results into existing code.
Although interesting, many questions remain, their search seems to work based on a keyword based approach. No code semantics is taken into account and I thought they promised some contributions in this space (?). They seem to have restricted their notion of context to just information on the development environment. The only contribution in my opinion is their idea that parses code segments, examples and explanations from web pages and an interface that integrates the results into an IDE - which is nice.
Airi Lampinen - 11/16/2010 16:50:30
Brandt. et al‚Äôs paper on example-centric programming revolves starts by stating that programming practices have changes fundamentally as online source-code example have become readily available. The authors discuss integrating web search into the development environment by presenting a Web search interface that is integrated into one development. The interface, Blueprint, is a task-specific search engine that aims at helping users locate example code. The authors position it in the design space of tools for developers.
The paper explains how Blueprint works and how it has been implemented. Furthermore, the authors report results from evaluating the system and from a three-month-long study that looked at how Blueprint would affect the workflow of real-world programmers. The evaluation showed that Blueprint enabled users to search for and select example code significantly faster than with traditional Web search tools. Based on the user study, the authors claim, furthermore, that task-specific search interfaces may cause a fundamental shift in how and when individuals search the Web.
Hartmann et al.'s paper is motivated by the belief that there is significant value to leverage in enhancing communal information exchange around debugging by means of integrating it directly into authoring tools. The authors introduce HelpMeOut which is a social recommendation system that facilitates debugging of error messages by suggesting peer-provided solutions. Technically, the insight enabling HelpMeOut is to use both error messages and source code context in the capture and search for relevant fixes.
The system is designed mainly for non-expert (novice) users for whom the originality of produced code is not a core value but who rather write code as part of their work, in order to get done the things that are important to them. The evaluation of this initial version of the system gave promising results. For instance, the technique was able to suggest useful fixes for almost half of the errors the novice participants encountered.
It is easy to see the value of such a system for the novice user, as debugging can be both frustrating and difficult. Furthermore, I think HelpMeOut could be very useful also in terms of a deeper level of learning, as instead of giving mere examples of how to fix the code, the system is providing also explanations. In this way, the novice users may learn to understand why something works instead of coping with a process of trial-and-error enabled by using pre-existing functional code. While many of the common bugs can be easily explained and understood, it is a core question to understand what the motivations for participation could be for the expert users. Their knowledge is needed to provide the explanations concerning more complex problems.
Aaron Hong - 11/16/2010 17:25:08
In "What Would Other Programmers Do?" by Hartmann et al. talks about "a new strategy of collecting and presenting crowdsourced suggestions for programming errors inside an IDE." The insight of HelpMeOut is to "use both error messages and source code context in the capture and search for relevant fixes."
The system is an interesting that is a set in the right direction, to use the source code context for searching and integrating into the IDE. However, there are several limitations that might prevent it for being particularly useful as long as the novice programmer knows how to use Google. Some limitations mentioned where that it "does not deal with type systems of object-oriented languages." Or more importantly with runtime exceptions with significant success. However the idea to apply it to dynamic languages seems like where it would be most useful, since low level languages (especially those that run on devices) are much harder to implement such a "meta" system for.
In "Example-Centric Programming: Integrating WEb Search into the Development Environment" by Brandt et al. talks about a "user interface for accessing online example code from within the development environment." It's funny how both the themes in both papers are that people usually program by example. I find that to be true myself and see why research to assist this can have a large impact. I'm impressed that they did a longitudinal study and great they have partnered with Adobe to try to get as many users on it as possible.
Arpad Kovacs - 11/16/2010 17:35:58
Blueprint is a code example search tool which is directly embedded into the Flex Builder IDE. The use of a task-specific search engine which is sensitive to the current coding context significantly reduces the cost of search, compared to the standard workflow of firing up a browser and searching for code examples on general search engines. The main insights of the paper are that integration into an IDE allows the search engine to learn the user's context of programming language and framework, and thus automatically construct more targeted queries. In addition, providing a consistent search results interface allows users to evaluate each result more quickly, and thus reduces the time and effort required to find an adequate solution.
I found it interesting that people use Blueprint for search differently than traditional search engines. Since the cost of performing a search is much lower, users often use Blueprint for quick clarifications and reminders. I was also surprised that users were not bothered by the lack of context.
One drawback of the system is that it treats each code block as independent. The disadvantage of this is that it may inadvertently hide useful information from users in case of a step-by-step tutorial, or line-by-line breakdown of a section of code. Another issue is that blueprint completely ignores all pages that include "group" or forum" in the url in an attempt to block buggy code. I think this is a mistake, since sites such as stackoverflow are quickly becoming the premier source of answers to obscure requests that are not addressed in the standard documentation. A better solution would have been to ignore the initial question post (which probably contains incorrect code), and then extract code from the highest voted reply, since upvotes imply that the solution works.
I thought that the feature which records the url of the source file and checks if updates are available was clever. This would be especially relevant for example code obtained from community sites such as stackoverflow, where it is very likely that the first reply that appears is often not the best answer, and will probably be replaced by a more efficient solution in the future.
The HelpMeOut social recommender system assists programmers in debugging process by suggesting successful solutions to compiler error messages and runtime exceptions that other programmers have encountered. The system is directly integrated into the development environment and avoids the latency and lack of answers that plague traditional Internet coding forums. The main contribution of the paper is the insight that much information is generated during the process of programming, and by correctly instrumenting an IDE and recording the state of each user's code, this can become the basis for a powerful crowdsourcing platform.
I thought that the process for collecting relevant fixes for runtime exceptions was quite clever. This involves a Processing/Java interpreter that tracks the number of times that a line was called before encountering an exception, and thus can detect whether forward progress can be made after the modification of source code. It would be interesting to expand this system to automatically profile the user's code, and suggest ways to improve performance by optimizing redundant/bottleneck lines of code.
I think that HelpMeOut would be a great timesaver for novice programmers who are using limited languages that have a finite number of error conditions that are encountered with high frequency. Given this target audience, I think that the decision to initially deploy HelpMeOut on Processing and Arduino was wise. Everyone using those two systems are using the same IDE and compatible versions of the language, therefore they encounter the same compiler errors and runtime exceptions. Thus the probability that someone else has encountered the same error as you is much higher, and their solution is more likely to work for your problem.
In contrast, I do not think that HelpMeOut would be as effective when applied to a more complex language such as C++, which uses a wider variety of compilers (each with different error messages), and whose users encounter a much wider variety of often obscure errors. It is less likely that someone else has encountered the same problem you have on the same platform you are running, and that their solution will work for you. This is unfortunate, since HelpMeOut would arguably be more valuable for C++ users who often have to contend with cryptic and uninformative compiler errors, as well as nondeterministic runtime errors that are extremely difficult to debug. Perhaps this is really a network effect problem; once the system reaches a critical mass of users, the situation would improve.
Luke Segars - 11/16/2010 17:37:01
This paper describes a new system called HelpMeOut that provides example-based suggestions to fix common programming problems. The system's database of errors is centralized and shared between users, meaning that the mistakes of others (and the matching solutions) will hopefully be available to someone the first time they make the error.
I think that this is actually a really good idea for hunting down minor bugs. For example, students in programming classes could benefit tremendously from essentially having an online virtual "tutor" who could point out likely problems as they made them. The authors have gone through a good bit of trouble to catch a variety of errors that can be quite difficult to hunt down. If this were available, I would install it today.
That being said, there is a question of pedagogy in here as well. People have argued that the spell checker built into modern word processors is leading to a decreased ability of individuals' ability to spell without a machine's help. It is usually unnecessary for us to know how a particular word is spelled -- as long as we can get close enough, the computer can figure out the rest. In a way, this system provides the same potential pitfall for programmers. There is certainly a learning experience buried in many debugging sessions (despite their short-term productive inefficiency), and I could see this potentially getting in the way of that experience.
It would be interesting to see how the expert community would be able to handle the "comments / explanation" section of HelpMeOut. I suspect that there would be many cases where it would be difficult to comment on the particular cause of an error due to loss of context -- if a particular chunk of code is taken from a function or larger arrangement, it may be difficult to see exactly why it is failing. Nevertheless, I think that the majority of (simple) problems that would be submitted to a system like this wouldn't suffer from this fate.
Luke Segars - 11/16/2010 17:57:43
Example-centric Programming w/ Search
This paper introduced a code search feature called Blueprint into the Adobe Flex environment that can be used to search for example snippets of Flex code. The search box could be activated with a hotkey and attempted to determine the context of either the search term or the particular line of code that the search bar was invoked on.
This, like the HelpMeOut system, seems like a pretty handy thing to have around. I spend a decent amount of time searching for code examples, although I have found that they have become generally easier to find as time has passed without the use of this tool and I suspect that many programmers may not consider finding samples to be a significant time sink (as suggested by the "would you download this" question in the smaller user study). Nevertheless, the smooth integration into the IDE and keyboard shortcuts make this seem pretty darn cool.
The Flex environment is, in a sense, an ideal case for this sort of project due to its huge maintained repository of example code. Other popular languages such as Python and Java may not have a high-quality database of this sort to draw on. It would be interesting to see if this could be turned into a social service as well, similar to HelpMeOut. HMO populated its database with information from its users -- it would be interesting to see whether a similar model (accompanied with ratings) could make the in-line code search feasible with any language.
Dan Lynch - 11/16/2010 18:03:07
What would other programmers do?
A very useful and important paper that explored the social aspects of debugging code! Super important and super useful. It was also very thoughtful to have privacy for parts of code that developers may be unsure about sharing. This encourages people to share contributing to the system, thus making it better. I also like the fact that the authors thought about both the source code and the error message, so it can really provide a big enough picture to help developers.
Although it would suit novices more, I still think there are benefits to experts being a part of the target user: I think a system like this could have a major impact in software technology as a whole and everyones contribution would be important in building such a system.
This paper presented a system for sharing and searching of code, in an IDE while maintaining links between shared code and its source. Basically while you code you can query and search for code and insert it into your system. This takes so much time out of the current pre-existing system of use, especially since its all in the same system.
One aspect I though they could have looked at more was the quality. It was hard from their results to understand while they only took quantity of queries into account. I would like to know how the results were used.
David Wong - 11/16/2010 18:21:28
1) The "Blueprint" paper discussed a plugin system to Adobe Flex Builder that helps users find example code quickly. The paper discusses the implementation, studies, and the results of the system. The "HelpMeOut" paper described a system that employs a new strategy of collecting and presenting crowdsourced suggestions for programming errors within an IDE. The paper describes the system implementation in two languages as well as a user study and its results.
2) To me, the Blueprint system seems to add value only in a limited domain: the search for code examples. While this may help in certain situations, the benefit of having this system may not outweigh the time and energy needed to develop it. Nevertheless, I think it is a useful tool that can be naturally integrated into an IDE. Having used Adobe Flex Builder extensively, the tool is certainly appealing. I think that it would be better if it took a crowdsourcing approach in addition to utilizing commercial search engines since we're dealing with code examples. This could add more material to the system. It would be like stackoverflow but for code examples and with a search interface.
The HelpMeOut system is a novel approach to aggregating a knowledge base on how to solve programming errors. While crowdsourcing solutions exist, I think HelpMeOut's primary benefit is that the process is largely automated with a social component. This removes the step of needing to interact with a separate, third-party site, like stackoverflow. While the paper was more of a proof-of-concept, I think that this type of tool would have a lot of added value for programmers whose main goal is to write programs that work. I'd like to see more systems like this integrated into IDE's.
3) The "Blueprint" system argued its viability well. The user study and longitudinal tests both conveyed its potential and the paper pointed out the limitations of the system. I think the problem is well-motivated, but it seems to be a problem that could be resolved--partially--by smarter searching on search engines. The HelpMeOut system also argued its viability well. The user study illustrated its effectiveness in a small group and extended those results to future work on larger groups. The paper also pointed out its limitations and clearly noted where the innovation was and where there could be more improvement.
Thejo Kote - 11/16/2010 18:26:51
Example centric programming:
Brandt and co-authors present Blueprint, an IDE plugin that allows task specific searches and embedding of code examples from results. The goal of Blueprint is to prevent the need to context switch between IDEs and browsers during development to look up code samples.
While the concept makes a lot of sense in theory, the implementation was underwhelming. The heavy lifting, which is the search, is performed through simple text searches via Google. What they call an augmented search query is just the addition of the programming language. The main value addition of the Blueprint server is the extraction of code samples and checking for updates to code, so that users can be updated. The code extractor did not appear to be robust with rules like the exclusion of URLs with "group" or "forum" in them. What would have been particularly interesting is if the query augmentation and result retrieval included a better semantic understanding of the code in the IDE and the web page. This might have prevented the problem of major edits needed in the copied code which the authors note too. Also, I think the utility of the looking for changes to code is minimal. When code is posted to blogs, forums etc., the window of time in which it will see edits is small. After a few days, there is usually no more activity related to the code. The authors argue that open source code repositories are not a good resource because they lack comments and contextual descriptions, but it may have been a better choice for code that is "living" if that was a design goal.
Overall, I thought Blueprint was interesting but took too many shortcuts.
What would programmers do:
In this paper, Hartmann and co-authors present HelpMeOut, a system that aids novice programmers debug compile and run-time errors in their programs. HelpMeOut instruments IDEs to collect data about error messages and corresponding fixes made by the programmer, which are then uploaded to a centralized database. This database of errors and fixes can then be queried by other programmers who encounter similar errors in their code. The suggestions for fixes are available in context in the IDE and in some cases HelpMeOut can even auto replace errorneous code for the programmer. The system also provides a web interface where explanations can be added to specific errors which are then displayed as part of the suggested fix in the IDE.
I really liked the idea of using distributed learning to help novice programmers, but I question the fundamental premise that it is possible to accurately capture compile time errors and corresponding fixes in code. As Hartmann notes, the implementation presented in the paper depends on a naive single error and fix scenario, and not the more common, multiple errors leading to multiple fixes scenario. What makes the collection of reliable data even harder is the wide availablity of automatic compilation in most popular IDEs. Modern IDEs have pretty much done away with the compile step since they provide active feedback when code is being written. It's common to get even large programs to run without any compile errors on the first try. I don't know if it is possible to collect the information that HelpMeOut needs in such a system.
Bryan Trinh - 11/16/2010 18:40:25
This paper presents a well engineered tool that guides the programmer along in producing code by implementing a search function inside the IDE. The search function finds code chunks on the internet and displays them so that programmers can code by example.
Even beyond helping novice users, the system acted as a reminder to seasoned programmers on how to implement something. This would be extremely useful for programmers who often work with many systems at once. Even more, since the system queries example code on the web, the database of code chunks will continually expand.
Although the authors mentioned that this system is better suited for programmers who have the ability to gage whether or not a particular code chunk is appropriate, I would've liked to see how successful novice users were in filtering out appropriate vs inappropriate code. In particular those who are trying to learning the language through these queried examples. One potential problem is that hobbling together chunks of code just simply won't always work and would cause more confusion. This is unfortunately the mode of learning that many end user programmers take though.
What would Other Programmers Do?
This paper presents a tool that helps the programmer search for solutions to common debugging error messages by using crowd sourced solutions. When the programmer receives error messages, the system gives a list of recommended solutions by presenting example code generated by other users who have fixed the problem.
The authors mentioned that HelpMeOut might decrease the learning of the novice programmer because solutions are presented and tested through guess and check. This would be especially true for processing because they have one of the worst debugging systems ever. The common tools available in other IDE are just not available. By creating an interface that lets the user perform normal debugging activities as well as present example code, I think that the novice programmer will learn much quicker and more importantly, will be more inclined to learn.
Overall the paper is comprehensive in approach and it was hard to find obvious holes that were not addressed. One thing that came to mind though is that HelpMeOut heavily relies on the value of the error messages produced by the system. Thus, programming environments that already have poor debugging won't get the granularity needed to match the problem with solution. Sometimes error messages are just as confusing as not having them.
Matthew CHan - 11/16/2010 18:42:26
In this paper, the authors tackle one of the most notorious problems in computer programming: extremely bad error messages. The authors introduce HelpMeOut, a system that uses crowdsourced fixes to address errors, and a database that stores them all. The other component is the UI to view before/after fixes that might be appropriate for the individual. In particular, this is mostly aimed at novice users. This paper is pretty important because of its novel approach to vague/unclear error messages by crowdsourcing the fixes to other users. The current habit is to search online at places like StackOverFlow, but HelpMeOut integrates this style into its own system, also building on the analogy that it is easier to fix something when given an example.
HelpMeOut was tested at a programming workshop, each two hours long. From the participants in this crowd, students queried HelpMeOut 274 times, approximately 7 queries per person, per hour. The system was very helpful in common errors such as misspellings, missed smicolon, and other common errors that novice usually encounter. More importantly, what's unique about HelpMeOut is its lexical tokenizer. This addresses the difficulty with web searches for code. The methdolodies offered is the architecture used when they augmented the Processing IDE and the lexical tokenizer.
This paper relates to today's technologies because it lowers the learning threshold for novice programmers and other newbies such as hobbyists. In some ways, think Arduino and it's popularity due to a huge online community that is ready to help each other. HelpMeOut also integrates the use of crowdsourcing; this could save the user time so they don't have to pull up a web browser to search for solutions on forums.
This always relates to my field of work because I'm always expected to pick up a new language sometimes. As usual, error messages are unclear, and HelpMeOut would make the experience easier and more receptive. Blindspots were well covered, such as addressing plagiarism, or people who just want to fix their code instead of learning the principle.
Anand Kulkarni - 11/16/2010 18:45:30
Example-Centric Programming: Integrating Web Search into the Development Environment
This paper presents an IDE extension, Blueprint, that incorporates searching for code examples on the web and incorporating it into code.
The main contribution here is a tool that supports a practice that even experienced programmers resort to -- searching for code samples on the web that can be incorporated into code. This is a phenomenal idea, since it has the potential to dramatically speed up coding. I see the future development of this technology as supporting a more precise level of abstraction for coding -- it seems likely that such technology would inspire the design of specialized code databases tailored around exactly this kind of searching, much as this tool has emerged to support access to existing code sample repositories. A natural development could be a shift in example-based programming from being principally a novice's way of understanding and learning to a primary programming paradigm for even advanced users; after all, much code functionality is reused. The potential for increasing abstraction in coding is good. I would have liked to see a bit more focus on automatic integration of the code into the user's program with hotswapping of variables (for instance) or a greater ability to divine the user's intent in searching for a specific piece of code, but this may be too technically ambitious.
The evaluation metric focused on three tasks, which were directed, exploratory, and tutorial. These are appropriate examples of the kinds of tasks that programmers are likely to search for code on. Measuring the speed of task completion is a great metric since it validates both cost concerns in hiring programmers and developing technology and gives a proxy for measuring a user's frustration. Asking users how well they liked the system on a Likert scale gave a reasonably good response, though perhaps purely qualitative evaluations might have been better here. The fact that experts rated the blueprint-produced code as higher quality is important and a very interesting finding; it means that users using Blueprint are able to objectively improve their programming ability, or at least their ability to adhere to programming standards, while still producing code themselves. I certainly would love to use this software!
What Would Other Programmers Do? Suggesting Solutions to Error Messages
The authors introduce HelpMeOut, a recommender system that suggests solutions to error messages that other users have found in the past.
The main contribution of HelpMeOut is the notion that IDEs can collect and intelligently present the solutions obtained by other users in resolving programming errors experienced by the present user. This has the potential to make programming substantially faster, since a substantial amount of time is spent on resolving (for example) compile-time errors. However, I suspect the utility of the program may diminish with time; a user will quickly learn to avoid common errors (or the program will be taught to catch them), but rare errors will still crop up. It would be interesting if the interface could simply make these changes themselves over time without the need for user confirmation; this might be an extension of the authors' automatic patching feature. I'm a little concerned that the suggestions provided could at times be misleading if it's based purely on socially-obtained suggestions, and would have loved to see more discussion of this idea.
The evaluation is a little unconvincing, since it used somewhat poorly described The result of 47% correctness is actually reasonably good, but it would be better if the authors defined better standards for when a suggestion was useful. More broadly, the system circumvents issues that are more common, whereas less common issues are likely to take up a much larger amount of time. I would have liked to see a comparison of how well the system works against a baseline alternative such as web searching. I also would have liked to see a little more evaluation of cases where the system made poor suggestions that misused a user's time -- how can the system be modified to avoid this?
Shaon Barman - 11/16/2010 18:48:39
What Would Other Programmers Do?
This tool crowdsources bug fixes to explain compile time errors to programmers.
This tool differs from prior work in that it does not mine information from forums or documentation, but rather silently views the actions of the programmer to learn potential bug fixes. Although the paper talks about fixing dynamic errors, the actual instrumentation to detect a dynamic error and how that error was fixed is too big a challenge. Instead, the paper focuses on correcting compile errors, whose solution can easily be detected (ie the program now compiles). The tool uses both the context (source code) along with the error code to query the database. The tool is novel in that it uses crowdsourcing combined with context specific information to tackle a fairly difficult task.
The evaluation is interesting because it shows fairly few queries are needed before the database is properly seeded. This is encouraging that a new language can be "learned" by the database in a short amount of time. I would have liked to see user study which compared how effective users were with and without the tool. The study only shows how the users used the tool, and does not explore how the tool compares with other sources of information (such as autocomplete, blueprint, etc).
The tool seems to be limited to beginning programmers and tackles mostly syntactic bugs. The syntactical nature of the tool also affects the results of the paper since the Processing language has a much simpler language than most programming languages. More advanced users also tend to have less syntax bugs and more logical or api bugs which manifest only during execution. Also, it is noted that HelpMeOut was implemented for the Arduino, but no evaluation is given for that platform.
One possible extension of this tool is to combine it with current fault localization tools. Fault localization tools usually provide a list lines which may contain the bug, but usually this list has many false positives. The HelpMeOut tool could then inspect each line to see if it matches any of the known fixes. I think the use of automatic programming languages tools alongside crowdsourced information would yield a lower false-positive rate while providing more fixes.
Example-centric Programming: Integrating Web Search into the Development Environment
The Blueprint tool allows programmers to embed their current context into a code search query. The results are then shown in the IDE in code-centric view.
The main aspect I liked was that the tool allows the programmer to write a query without breaking the programming flow. This minimal interruption is also supported by showing a modified version of the page, with the code example highlighted. The tool targets mid to expert programmers who the basics of programming, but are unfamiliar with or forgot some of the specific APIs they need. Learning a new API can be a daunting task. Blueprint attempts to make this task easier by lowering the costs for executing a query.
The tool reminds me of the API documentation hover boxes which appears in Eclipse. The user can quickly read the API in the IDE without having access a separate browser. Blueprint is novel in that it provides code snippets instead of documentation. I liked their evaluation, which allowed them to see how a user community interacted with the tool. By integrating with the Adobe Community Help search, they got high quality search engine and a way to realistically compare their results. One point that should be considered is that Flex seems to be a fairly targeted language, which probably has a limited set of libraries and uses. Such a targeted language would show more useful results than if the tool was developed for Java or Python, since the search query would probably need more information to target specific application.
Richard Shin - 11/16/2010 18:57:32
Example-Centric Programming: Integrating Web Search into the Development Environment
This paper describes a system for embedding web search into the software development environment, called Blueprint. The authors note that since the web's advent, developers have used it to, in particular, search for code samples that they can integrate into their own work. With Blueprint, the authors aim to decrease the costs associated with web searching by deeply integrating it into the IDE. The user can invoke Blueprint while programming by pressing a simple hotkey, and quickly select example code snippets, and paste them into the editor.
Blueprint is available as an experimental end-user product, and while I don't program in Adobe Flex, I certainly often look for examples and references on the web when I program, and it seems certainly something I would love to have. Many examples of programmer-helping user interfaces, such as code completion, already exist, and Blueprint along with those others seem like a good example of intelligent user interfaces as we read about a few weeks ago. While the paper tilts more towards studying a specific end-user system rather than examining general ideas for aiding programming-by-example, nor study the importance of aiding programming-by-example, the core idea of the product still seems like one that can be built into and extended by future efforts to aid programming.
For the system to work well, it must present useful and relevant results and accurately find code examples and descriptions for them in the results. This process, however, seemed rather arbitrary and ad-hoc as described in the paper. Despite the evaluation presented in the paper, it's hard to tell how useful or helpful the system really would have been in practice from the vague descriptions of how it produces results. I think it would have been helpful if their approach had been more principled, or better described in the paper to be convincing of its efficacy.
What Would Other Programmers Do? Suggesting Solutions to Error Messages
This paper presents HelpMeOut, a system which helps programmers find the cause behind the error messages that they encounter, by analyzing previous instances of the error from other users and the fixes they applied, and suggesting potential changes by using these results. The authors describe the details of the syntax-aware architecture and implementation of their system, and explain results from the evaluation of the system in workshops with novice programmers.
Broadly, this paper seems very similar to the other one assigned today, as they both describe fully-implemented systems which aid users with editing tools as they program. Like the other paper, while I don't typically use the IDEs augmented in this paper, I still thought I would find a system like this helpful; I certainly end up often searching for compiler error messages on the web in order to figure out what they mean, a process which I imagine would become far more expedient with a special-purpose tool such as this one that is fully aware of the context of the error I am experiencing and can find results from others who experienced the same error in a similar situation. The use of syntax-aware source code analysis along with comparing the error messages themselves seemed the core novel idea of this paper, and one that could certainly see a lot of improvement in the future to move beyond the 50% helpfulness ratings measured in the evaluation.
I wonder, though, whether crowdsourcing fixes to error messages is really the best way to solve errors. It seems that the system could easily and disproportionately promote solutions which don't necessarily address the core of the problem; for example, with the example shown throughout the paper of the array index out-of-bounds exception, the suggested fix was to simply check for the array bounds every time, rather than limit the range of the original iteration to avoid the error. Also, it seems much helpful to, and I don't see why it is impossible to, simply have better and more informationve error messagse in the first place. If compilers with such features become commonplace, it seems a system like this could become largely become obsolete or irrelevant.
Matthew Can - 11/16/2010 18:58:55
This paper introduces Blueprint, an interface that integrates web search into the development environment. The results of a lab study show that adding a task-specific search engine to the programming environment helps programmers adapt existing code to their projects faster, and that it helps them write better quality code.
The concept of hooking a browser into the IDE is not this paper‚Äôs most significant contribution. That is straightforward to do with a plug-in. What is novel is the way in which Blueprint integrates a task-specific search engine into the programming environment. In particular, I liked the system‚Äôs context-free, compact format for the search results. It removes a lot of extraneous details, and combined with the keyword highlighting, it focuses the programmer‚Äôs attention on the example code. This detail, combined with the ease of copying code to the editor, makes it fast for programmers to adapt existing code to their projects.
The paper provides strong evidence that Blueprint facilitates example-centric programming better than standalone development environments, but there are likely other situations in which this interface (or some of its elements) can provide benefits. For example, in education this could be used to help students learn new programming languages. However, it might not be a good teaching tool for first-time programmers; the participants in the paper‚Äôs longitudinal study found Blueprint most useful when they already had some idea of how to complete the task.
Furthermore, even just the ability to create a link between one‚Äôs source code and some target (web page, local document, image, etc.) is a benefit to the programmer. Programmers can index their code with relevant references. For example, if I am writing a program that calculates users‚Äô taxes, I might link bits of my code to references in that domain.
What Would Other Programmers Do?
This paper presents HelpMeOut, a system that helps novice programmers debug their programs by suggesting fixes that other programmers developed for similar errors.
The paper‚Äôs contribution is the concept of facilitating the task of debugging by collecting solutions to programming errors and presenting those in the IDE. In the study done in the paper, about half of the suggestions provided by HelpMeOut were deemed useful. This is an indication that the concept is viable, at least in certain situations.
One potential limitation is that HelpMeOut either will not work or will be altogether irrelevant for professional programmers. My suspicion is that the study in the paper had the novice design students work on a tutorial or some other kind of ‚Äútoy program‚Äù. Given this context, I can see how HelpMeOut can be beneficial. But professional programmers are unlikely to benefit from the syntax errors or variable misspellings for which HelpMeOut will be most useful (besides, a good IDE will already take care of these problems). On top of this, the kinds of errors generated by complex programs might not have existing solutions in the HelpMeOut database. The authors acknowledge this long tail, but suggest no way around it besides gathering more data (but gathering more fixes also means gathering more queries, so I think the long tail will always exist).
Linsey Hansen - 11/16/2010 19:02:02
In the first article, the authors dices a program which gives users access to a task-specific search engine in their respective development environments. This is intended to allow users to search for code examples meant to accomplish various tasks a user might be coding for.
The idea itself seems nice, because it can often be difficult to find code examples if there is no definite way of describing it. I know that the iOS has an example section that allows users to search for specific projects performing various tasks, which is awesome, but sadly it is not all-encompassing, and just covers iOS and Mac OSX. If I could find code examples like this all over the internet, it would be pretty sweet. One thing I didn't notice that would be nice is if the description for search results could include images, or maybe the page they link to would, because images are often handy for more interface-oriented code.
What would Other Programmers Do?
In this article, the authors introduce their HelpMeOut system, which recommends ways to debug error messages in IDEs. It does this using solutions that others using the system have used in the past.
The concept itself, of adding a tool that can automatically communicate with the project you are working on and then find code and solutions for the exact error you are receiving is pretty awesome. It is nice because it not only cuts out the need to search through google, but it can quickly provide you with some suggestion using actual code examples. Sometimes for certain errors, I will google the error and never find physical code examples, instead I find some question thread where people are just trying to explain everything verbally and that is a pain to follow, but with this, there will almost always be a code example (or a direct suggestion using my code) assuming that people are using the tool. Another great thing is that this also includes explanations, which can hopefully be iterated many times through by more professional people, as opposed to those rather annoying 3 word answer I might find on the internet.
I also found the privacy settings rather satisfactory, because I am pretty sure that there are definitely companies out there that would have a fit if their programmers were sharing certain parts of code via "fixes." This would probably lead to a lot of silly legal issues, but luckily this system allows user to opt not to post their code for others to see, and just reap the benefits. At the same time, this could lead to the problem of professionals never sharing with the database, but I am sure that the issues many students face will be plenty for that.
Kenzan boo - 11/16/2010 19:08:00
Example-Centric Programming: Integrating Web Search into the Development Environment
The article is about Adobe's implemtation of a system to help developers in Flex find examples of how to code something. It is a web interface that is built into flex that helps users locate possible example code that would do something like load an image, and then imbeds that code into there allowing them to edit it.
What Would Other Programmers Do? Suggesting Solutions to Error Messages
This article describes a system that is integrated into an IDE that helps users find help for given error situations. It captures a critical need of new developers and students trying to learn programming. "Programmers¬ó‚Äîespecially amateurs¬ó‚Äîoften create software by opportunistically modifying found examples , and they regularly use online forums and blogs to seek help." It has 4 main points: it tracks the changes that make code go from error free to an error situation, has an online database for storing fixes, a suggestion that pulls from that DB of possible fixes, and a web interface to get text explanations from experts.
Both of these articles are based on examples. I find that most of my time programming is actually reading other code to see how something is done, then sometimes copy pasting it in and altering it to fit the need. This is particularly true when learning a new language or new development methodology. Its much easier and almost always cleaner to copy well written code and alter it to fit your needs. This same process of learning by example of well written code is done by lots of other developers new to a language and many students as well.
Aditi Muralidharan - 11/16/2010 19:12:01
In the first paper, Jonathan Edwards presents a plug-in for Eclipse that illustrates the concept of "example-centric" programming. A piece of code is given as an example, and the IDE "works through" a couple of runs of the example code automatically, showing the parameter values as they change, and function calls. A trace is produced, for view next to the example call.
This seems really similar to the debugger in eclipse, which allows you to see state, but even more automatic. Instead of the programmer having to create example code, it automatically creates a framework, picks values, and initializes them. When exceptions occur, it highlights them in the trace.
This idea seems like it could be useful in a classroom environment - one thing I struggle to teach beginning computer science students is the trick of actually working through a program to understand how it works. Other than that, however, I am not sure how useful it would be. A large code base consisting of many functions would actually ot be possible to work through - but it is not an example, but I can't imagine anyone other than beginners needing this.
In the second paper, Hartmann et. al. introduce HelpMeOut, which recommends ways in which a programmer can deal with an exception based on what other programmers have done in the past. It is targeted at novice programmers, who often do not understand what compiler errors actually mean. In these situations, it helps to have a more experienced programmer around to point out what the problem actually is and how to fix it. HelpMeOut is integrated into an IDE. The IDE, like other Hartmann productions, can be thought of as having a crowd inside. Bugs and suggested fixes are stored in a remote database, and queried, or added to as they come up. When an error occurs in the code, the programmer is shown a before-and-after view for each of the suggested fixes in a separate pane, showing the lines that need changing.
HelpMeOut seemed to provide useful suggestions in user studies, but there are issues of privacy, intellectual property, and plagiarism surrounding this idea that the authors discussed. I personally would not want my beginner students using this interface. I think it would make it too easy for them to get by without really understanding those "cryptic" error messages.Learning to debug effectively is an important skill - I don't think it can be learned by copying examples blindly.
Drew Fisher - 11/16/2010 19:57:14
What Would Other Programmers Do? Suggesting Solutions to Error Messages
Core idea: since most novice programmers learn better from modifying other people's code, and they also look to experienced programmers for guidance when faced with errors, create an IDE that assists this sort of workflow. The trick: collect tons of source code in a more generic
One has to wonder: if types of errors can in fact be classified and the appropriate corrections made automatically, why don't our compilers already leverage this knowledge? This paper is neat in that it helps counteract the weaknesses of the compiler, but the parallel problem of poor compiler diagnostics is embarrassingly undersolved.
I'd be interested in seeing more applications of using a generic syntax tree to implement powerful code search and model how code changes over time. While HelpMeOut was targeted at novices, a more engineered solution to the "logically, how did the code change?" problem with knowledge of the impacts of such changes could ease maintenance and bugfixes in a code base, as well as help enforce different style rules. This would be an interesting avenue for industry to follow.
Example-Centric Programming: Integrating Web Search into the Development Environment
The first things I think of when I see tools that make it easier to deploy snippets of code from the web are:
- possible security implications
- licensing issues - used a GPL'd snippet? your product must now be GPL'd!
- poorly-parsed code snippets getting pasted
I like the work toward extracting code snippets from webpages - it's a hard problem, and the easiest approach (which the authors took) seems to be to produce a lot of false negatives, to make sure that you don't wind up showing the user something unhelpful.
This tool seems like it would do (does) well at satisfying the need for a better API explorer and autocomplete functionality, but I'd hesitate to use it for code production, for the reasons noted above. Based on the usage statistics in which 76% of users NEVER copied anything off the web, I'd say the testers feel the same.
I also offer counterpoint on the concept of offering both interface modalities (popup and docked window) - the fact that the users switched a lot suggests that the tool didn't know where it belongs in the interface, and the user had to tell it where to go more often.
Brandon Liu - 11/16/2010 19:58:16
It would be nice to characterize recall vs learning new functionality. For example, in my own use of Processing, I frequently reference the online site to see how basic functions with many arguments (for example, color methods) work. I already have a good understanding of what the methods do; i just need to be reminded of their syntax. This is sort of like the distinction between incidental and fundamental complexity.
One thing that I suspect (going back to the Emacs/Vi vs. IDE war) is that an experienced programmer has a model of her own code that is as "close to the metal" as possible. What I mean by this is that the programmer is more comfortable typing code herself than copy-pasting something, since she is able to verify the code's correctness at each step. This would explain why relatively few uses of Blueprint resulted in a copy-paste event.
Another interesting idea is that a link is retained between copied code and the actual source code. The opposite of this idea is to match source code with example code, which is kind of the approach the next paper takes.
"What would other programmers do?"
The major idea behind the paper is that the system is by-default opt-in. The 'knowledge base' that is queried is generated by users who do not need to put in any effort to adding data. This presents a chicken-and-egg problem where for the system to be really useful, the database needs to be at first seeded with useful examples.
There were two areas of further research I was interested in after reading this paper.
One is the nature of search tools for programming. Searching on Google or Bing is an iterative process: one makes a query, looks at how fine grained the results are, and then edits their original query. It would be interesting to find the part of a search query error message that is the most significant.
Another factor that I noticed in the research is the design of program language error messages. It seems that interpreters that provide unique error messages are better than those that provide detailed error messages. One could try to create the "most Googleable" language by designing error messages in a certain way.