Concrete is the main material in building. Since its poor structural integrity may cause accidents, it is significant to detect defects in concrete. However, it is a challenging topic as the unevenness of concrete would lead to the complex dynamics with uncertainties in the ultrasonic diagnosis of defects. Note that the detection results mainly depend on the direct parameters, e.g., the time of travel through the concrete. The current diagnosis accuracy and intelligence level are difficult to meet the design requirement for automatic and increasingly high-performance demands. To solve the mentioned problems, our contribution of this paper can be summarized as establishing a diagnosis model based on the GA-BPNN method and ultrasonic information extracted that helps engineers identify concrete defects. Potentially, the application of this model helps to improve the working efficiency, diagnostic accuracy and automation level of ultrasonic testing instruments. In particular, we propose a simple and effective signal recognition method for small-size concrete hole defects. This method can be divided into two parts: (1) signal effective information extraction based on wavelet packet transform (WPT), where mean value, standard deviation, kurtosis coefficient, skewness coefficient and energy ratio are utilized as features to characterize the detection signals based on the analysis of the main frequency node of the signals, and (2) defect signal recognition based on GA optimized back propagation neural network (GA-BPNN), where the cross-validation method has been used for the stochastic division of the signal dataset and it leads to the BPNN recognition model with small bias. Finally, we implement this method on 150 detection signal data which are obtained by the ultrasonic testing system with 50 kHz working frequency. The experimental test block is a C30 class concrete block with 5, 7, and 9 mm penetrating holes. The information of the experimental environment, algorithmic parameters setting and signal processing procedure are described in detail. The average recognition accuracy is 91.33% for the identification of small size concrete defects according to experimental results, which verifies the feasibility and efficiency.