This paper studies the reliability and validity of naturalistic speech errors as a tool for language production research. Possible biases when collecting naturalistic speech errors are identified and specific predictions derived. These patterns are then contrasted with published reports from Germanic languages (English, German and Dutch) and one Romance language (Spanish). Unlike findings in the Germanic languages, Spanish speech errors show many patterns which run contrary to those expected from bias: (1) more phonological errors occur between words than within word; (2) word-initial consonants are less likely to participate in errors than word-medial consonants, (3) errors are equally likely in stressed and in unstressed syllables, (4) perseverations are more frequent than anticipations, and (5) there is no trace of a lexical bias. We present a new corpus of Spanish speech errors collected by many theoretically naïve observers (whereas the only corpus available so far was collected by two highly trained theoretically informed observers), give a general overview of it, and use it to replicate previous reports. In spite of the different susceptibility of these methods to bias, results were remarkably similar in both corpora and again contrary to predictions from bias. As a result, collecting speech errors "in the wild" seems to be free of bias to a reasonable extent even when using a multiple-collector method. The observed contrasting patterns between Spanish and Germanic languages arise as true cross-linguistic differences.