Traditional Chinese medicine (TCM) is commonly considered to operate due to the synergistic effects of all the major and minor components in the medicines. Hence sensitive and comprehensive analytical techniques are needed to acquire a better understanding of the pharmacological basis of the herb and to enhance the product quality control. The present review mainly focuses on the phytochemical analysis of TCMs using high-performance liquid chromatography coupled with mass spectrometry (HPLC-MS). Atmospheric pressure chemical ionization (APCI) and electrospray ionization (ESI) are the two commonly used ion sources. Triple quadrupole, ion trap (IT), Fourier transform ion cyclotron resonance (FTICR) and time-of-flight (TOF) mass spectrometers are used as on-line analyzer. The relationship between structural features and fragmentation patterns should be investigated as thoroughly as possible and hence be applied in the on-line analysis to deduce the structures of detected peaks. Characteristic fragmentation behaviors of the reference standards, as well as information regarding polarity obtained from retention time data, on-line UV spectra, data from the literature and bio-sources of the compounds allowed the identification of the phytochemical constituents in the crude extracts. Although a mass spectrometer is not a universal detector, high-performance liquid chromatography coupled with multistage mass spectrometry (HPLC-MS(n)) technique was still proved to be a rapid and sensitive method to analyze the majority of the many constituents in herbal medicines, particularly for the detection of those present in minor or trace amounts. The methods established using HPLC-MS techniques facilitate the convenient and rapid quality control of traditional medicines and their pharmaceutical preparations. However, the quantitative analysis is not the topic of this review.