In order to gain insight into the nature of self-organized criticality (SOC), we present a minimal model exhibiting this phenomenon. In this analytically solvable model, the state of the system is fully described by a single-integer variable. The system organizes in its critical state without external tuning. We derive analytically the probability distribution of durations of disturbances propagating through the system. As required by SOC, this distribution is scale invariant and follows a power law over several orders of magnitude. Our solution also reproduces the exponential tail of the distribution due to finite size effects. Moreover, we show that large avalanches are suppressed when stabilizing the system in its critical state. Interestingly, avalanches are affected in a similar way when driving the system away from the critical state. With this model, we have reduced SOC dynamics to a leveling process as described by Ehrenfest's famous flea model.