Blog
About

117
views
0
recommends
+1 Recommend
1 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Conference Proceedings: found
      Is Open Access

      Heterogeneity in the Usability Evaluation Process

      People and Computers XXII Culture, Creativity, Interaction (HCI)

      Computers XXII Culture, Creativity, Interaction

      1 - 5 September 2008

      Usability Evaluation, Five Users Debate, Evaluation, Stochastic Models, Overdispersion, Process Prediction

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Current prediction models for usability evaluations are based on stochastic distributions derived from series of Bernoulli processes. The underlying assumption of these models is a homogeneous detection probability despite of it being intuitively unrealistic. This paper contributes a simple statistical test for existence of heterogeneity in the process. The compound beta-binomial model is proposed to incorporate sources of heterogeneity and compared to the binomial model. Analysis of several data sets from the literature illustrates the methods and reveals that heterogeneity occurs in most situations. Finally, it is demonstrated how heterogeneity biases the prediction of evaluation processes. Open research questions are discussed and preliminary advice for practitioners for controlling their processes is given.

          Related collections

          Most cited references 11

          • Record: found
          • Abstract: not found
          • Conference Proceedings: not found

          Heuristic evaluation of user interfaces

            Bookmark
            • Record: found
            • Abstract: not found
            • Conference Proceedings: not found

            A mathematical model of the finding of usability problems

              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              Sample sizes for usability studies: additional considerations.

               J R Lewis (1994)
              Recently, Virzi (1992) presented data that support three claims regarding sample sizes for usability studies: (1) observing four or five participants will allow a usability practitioner to discover 80% of a product's usability problems, (2) observing additional participants will reveal fewer and fewer new usability problems, and (3) more severe usability problems are easier to detect with the first few participants. Results from an independent usability study clearly support the second claim, partially support the first, but fail to support the third. Problem discovery shows diminishing returns as a function of sample size. Observing four to five participants will uncover about 80% of a product's usability problems as long as the average likelihood of problem detection ranges between 0.32 and 0.42, as in Virzi. If the average likelihood of problem detection is lower, then a practitioner will need to observe more than five participants to discover 80% of the problems. Using behavioral categories for problem severity (or impact), these data showed no correlation between problem severity (impact) and rate of discovery. The data provided evidence that the binomial probability formula may provide a good model for predicting problem discovery curves, given an estimate of the average likelihood of problem detection. Finally, data from economic simulations that estimated return on investment (ROI) under a variety of settings showed that only the average likelihood of problem detection strongly influenced the range of sample sizes for maximum ROI.
                Bookmark

                Author and article information

                Contributors
                Conference
                September 2008
                September 2008
                : 89-98
                Affiliations
                University of Passau

                Information Systems II

                Innstr. 43

                94032 Passau, Germany
                Article
                10.14236/ewic/HCI2008.9
                © Martin Schmettow. Published by BCS Learning and Development Ltd. People and Computers XXII Culture, Creativity, Interaction

                This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

                People and Computers XXII Culture, Creativity, Interaction
                HCI
                22
                Conference Liverpool John Moores University, UK.
                1 - 5 September 2008
                Electronic Workshops in Computing (eWiC)
                Computers XXII Culture, Creativity, Interaction
                Product
                Product Information: 1477-9358BCS Learning & Development
                Self URI (journal page): https://ewic.bcs.org/
                Categories
                Electronic Workshops in Computing

                Comments

                Comment on this article