Blog
About

4
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Equivalence Test in Multi-dimensional Space with Applications in A/B Testing

      Preprint

      , ,

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          In this paper, we provide a statistical testing framework to check whether a random sample splitting in a multi-dimensional space is carried out in a valid way, which could be directly applied to A/B testing and multivariate testing to ensure the online traffic split is truly random with respect to the covariates. We believe this is an important step of quality control that is missing in many real world online experiments. Here, we propose a randomized chi-square test method, compared with propensity score and distance components (DISCO) test methods, to test the hypothesis that the post-split categorical data sets have the same multi-dimensional distribution. The methods can be easily generalized to continuous data. We also propose a resampling procedure to adjust for multiplicity which in practice often has higher power than some existing method such as Holm's procedure. We try the three methods on both simulated and real data sets from Adobe Experience Cloud and show that each method has its own advantage while all of them establish promising power. To our knowledge, we are among the first ones to formulate the validity of A/B testing into a post-experiments statistical testing problem. Our methodology is non-parametric and requires minimum assumption on the data, so it can also have a wide range of application in other areas such as clinical trials, medicine, and recommendation system where random data splitting is needed.

          Related collections

          Most cited references 3

          • Record: found
          • Abstract: not found
          • Article: not found

          Controlled experiments on the web: survey and practical guide

            Bookmark
            • Record: found
            • Abstract: not found
            • Article: not found

            Overlapping experiment infrastructure

              Bookmark
              • Record: found
              • Abstract: not found
              • Article: not found

              Network A/B Testing

                Bookmark

                Author and article information

                Journal
                24 September 2018
                1810.04630

                http://arxiv.org/licenses/nonexclusive-distrib/1.0/

                Custom metadata
                9 pages; double-column
                stat.ME

                Methodology

                Comments

                Comment on this article