SITbench 1.0: A Novel Switch-based Interaction Technique Benchmark
Abstract
Evaluation process of a switch-based interaction technique (hereafter: SIT) requires an interdisciplinary team effort and takes a considerable amount of time. Collecting subjective evaluation data from the users is a very common approach, but the subjective evaluation data alone might be manipulated and unreliable for comparing performances in many cases. Thus, therapists generally cannot succeed in determining the optimum SIT setup (i.e., determining the most appropriate combination of setup variables such as switch type or switch site) at first attempts, since it is hard to evaluate the measurable performance by collecting subjective data instead of objective data. Inevitably, each unsuccessful attempt to reach the optimum SIT setup results in a loss of serious time and effort. On the other hand, a benchmark application is also required to make performance evaluation of SITs by using a number of standard tests and empirical attributes. It is obvious that a quicker and more accurate SIT evaluation process provides a better cost and schedule management considering the increasing number of SIT users in the world. Therefore, we propose a novel benchmark for performance evaluation called SITbench that provides a quicker and more accurate switch evaluation process by collecting and saving the objective data automatically. We conducted a user study with eight participants and demonstrated that the objective data collected via SITbench helped to determine the optimum SIT setup accurately. Result of a questionnaire applied to evaluate the SITbench itself was also satisfactory. SITbench is expected to help researchers and therapists to make a better evaluation according to any change done in SIT setup variables (switch type, activation method, etc.) with the aim of reaching optimum SIT setup, which leads to a better cost and schedule management. As the first benchmark application compatible with all SITs, which can emulate keyboard characters or mouse-clicks, it can be utilized by assistive technology professionals to make comparisons and evaluations automatically via standardized tests.