This study discusses the problem of direction-of-arrival estimation (DOA) estimation for a monostatic multiple-input multiple-output (MIMO) radar system, and a novel sparse Bayesian learning (SBL) framework is presented. To lower the computational load, the matched array data is firstly compressed via reduced-dimension transformation. Then the problem of DOA estimation is linked to a sparse inverse problem. Finally, a forgotten factor-based root SBL algorithm is derived from hyperparameters learning, which can solve the off-grid problem by finding the roots of a polynomial. The proposed algorithm does not require the prior of the source number, and it can apply to the scenario with a small snapshot as well as coarse grid, thus it has a blind and robust characteristic. Numerical simulations verify the effectiveness of the proposed algorithm.