Dielectric strength testing of ceramics can be performed with various setups and parameters. Comparisons of results from different sources are often not meaningful, because the results are strongly dependent on the actual testing procedure. The aim of this study is to quantify the influence of voltage ramp rate, electrode size, electrode conditioning, and sample thickness on the measured AC dielectric strength of a commercial alumina. Mean values, Weibull moduli, and failure probabilities determined in standardized short time tests are evaluated and related to withstand voltage tests. Dielectric strength values in the range from 21.6 to 33.2 kV·mm −1 were obtained for the same material using different testing procedures. Short time tests resulted in small standard deviations (< 2 kV·mm −1) and high Weibull moduli around 30, while withstand tests at voltage levels with low and virtual zero failure probability in short time tests resulted in large scatter of withstand time and Weibull moduli < 1. The strong decrease in Weibull moduli is attributed to progressive damage from partial discharge and depolarization during AC testing. These findings emphasize the necessity of a thorough documentation of testing procedure and highlight the importance of withstand voltage tests for a comprehensive material characterization.