Adiabatic regularization is a useful method to remove UV divergence in quantum fields in curved spacetime. For relic gravitational wave generated during inflation, regularization on all \(k\)-modes of the power spectrum to 2nd adiabatic order, and of the energy density and pressure to 4th order, respectively, causes low-frequency distortion in the spectra, and even leads to IR divergences. The general covariance and energy conservation are respected at 4th order but is violated at 6th and higher orders. For de Sitter inflation, all three spectra are regularized to zero. To avoid these, we regularize only the short modes whose wavelengths are inside the horizon during inflation (corresponding to the present frequencies \(f \gtrsim 10^{9}\)Hz), and keep the long modes intact. Doing this avoids low-frequency distortion and IR divergence, and respects the energy conservation for \(k\)-modes outside the horizon. The inside-horizon scheme is legitimate since the \(k\)-modes of RGW are independent of each other, and can apply to the scalar curvature perturbation during inflation. The resulting spectra of RGW are UV convergent and simultaneously free of low-frequency distortion, and are nonvanishing for de Sitter inflation. With these regularized spectra as the initial condition, by evolution, we obtain the spectra at the present, which remain UV convergent and free of low-frequency distortion. We also analyze the structure of RGW at the present, and show that the spectra exhibit quick oscillations in frequency domain, even if the initial spectra during inflation have no oscillations. This characteristic pattern is due to the interference between the positive and negative frequency modes developed during cosmic expansion, and can be probed by future RGW detections.