Experiment Design and System Architecture
We developed our bubble cursor interface such that each user goes through the same set of clicks. The order that users click on objects are shown in the diagram below. Additionally for each cursor type and each pair of objects we collect 3 timing values (each participant goes through the experiment 3 times for each cursor. 60 clicks per participant)
This experiment setting enabled us to collect information for 10 different amplitude/target size pairs.
For the client application, we used the Processing Network library. As the user engaged in the experiment, the client would send logs of the user's activity to the server if the form of GET HTTP Requests with all information sent in the url.
We use the MVC model in Python/Django on the server to store values in a MySQL database. The following figure shows our data model. (Some more variables are later calculated in R from the data.). The online database enabled us to collect information from our friends by posting the link to our facebook pages.
and example of the data that is logged in the database is shown bellow (from the phpMyAdmin panel)
Analysis and Results
38 of our friends participated in our study. We used R and Microsoft Excel to perform the statistical significance tests on our dataset.
Number of Missed Clicks
We collected 2322 clicks, 22 of which were missed. The following diagram shows the percentage of clicks missed for each target for each cursor type. We ran a Welch two sample t-test in R and the resulting p-value was 0.047 which rejects the null hypothesis and suggests that the bubble cursor has lower missed rates and it is statistically significant (in general Welch's t-test is more conservative than the student t-test so we expect to have a smaller p-value if we do it with the student t-test)
Average time for each target
The most important question in this section is: "Is bubble cursor a significant factor for different amplitude/width groups. Since we have ten different groups of amplitude/widths we ran a two factor ANOVA on the dataset (the factors that are used are avergageTime~(cursortype and amplitude/width group). From the ANOVA analysis we get the p-value of 0.0056066 which suggests that the bubble type and the amplitude and width are in fact significant factors in this experiment. See below for R output.
In order to find the cases that the bubble cursor is a significant case we run Welch t-test between pairs of groups for example the following compares group 5 and 6. And the p-value of 0.0003063 suggests that for those two cases the cursor type is a significant factor in the difference between means
The average time for each target is shown in the following graph
For group 7 we observe that the average time for the normal cursor is lower than the bubble cursor but the t-test analysis showed that this difference was not statistically significant. (p-value was .85 which is very high and shows that there is 85% chance that this difference has happened randomly)
Interestingly group 8 also showed higher average time for the bubble cursor and the Welch t-test p-value was about 0.09 which is not significant but is suggesting that the bubble cursor has no advantage when the two object are very close.
We ran the experiment for a large number of participants (38 people), Most of our participants used their own computers but a portion of our participants (all our classmates in CS260) participated by using the same laptop (A Lenovo laptop, with Windows operating system and using the trackpad). Data was logged automatically into a MySQL database and was exported to an R data table for analysis.
We collected 60 mouse clicks for each user (30 for each cursor type) and compared 10 groups of objects/amplitudes.
We used ANOVA on ranks and Welch t-test (both from R main library) to analyze our results. All of our R scripts are included in the zip file.
The two-sided t-test suggested that number of misses is significantly higher for the normal cursor than the bubble cursor (p-value= 0.04).
The ANOVA method suggested that the cursor type and amplitudes are significant factors for differences between groups (p=0.0056). As a follow up test we studied the effect of bubble cursor for each group. Interestingly for those cases that the bubble cursor was faster, p-value suggested that cursor type was the factor and for those groups in which normal cursor was faster t-test failed to reject the null hypothesis meaning that cursor type was not a factor. Although in group 8 the p-value was relatively low p=0.09 while the normal cursor was faster and we inferred that the bubble cursor is not advantages when the amplitude is very small and the target is relatively large.