The Normalized Least-Mean-Squares (NLMS) algorithm is widely used to cancel the direct and multiple path interferences in Passive Coherent Location (PCL) radar systems. This study proposes that the interference cancelation using the NLMS algorithm and the calculation of the radar Cross Ambiguity Function (CAF) can be modeled as a notch filter, with the notch located at zero Doppler frequency in the surface of the radar CAF. The analysis shows that the notch’s width and depth are closely related to the step size of the NLMS algorithm. Subsequently, the effect of the notch in PCL radar target detection is analyzed. The results suggest that the detection performance of the PCL radar deteriorates because of the wide notch. Furthermore, the Nonuniform NLMS (NNLMS) algorithm is proposed for removing the clutter with the Doppler frequency by using notch filtering. A step-size matrix is adopted to mitigate the low Doppler frequency clutter and lower the floor of the radar CAF. With the step-size matrix, can be obtained notches of different depths and widths in different range units of the CAF, which can filter the low Doppler frequency clutter. In addition, the convergence rate of the NNLMS algorithm is better than that of the traditional NLMS algorithm. The validity of the NNLMS algorithm is verified by experimental results.