We study the moduli space of handlebodies diffeomorphic to \((D^{n+1}\times S^{n})^{\natural g}\), i.e. the classifying space \(BDiff((D^{n+1}\times S^n)^{\natural g}, D^{2n})\) of the group of diffeomorphisms that restrict to the identity near a \(2n\)-dimensional disk embedded in the boundary, \(\partial(D^{n+1}\times S^n)^{\natural g}\). We construct a map \(colim_{g\to\infty}BDiff((D^{n+1}\times S^n)^{\natural g}, D^{2n}) \longrightarrow Q_{0}BO(2n+1)\langle n \rangle_{+}\) and prove that it induces an isomorphism on integral homology in the case that \(2n+1 \geq 9\). Above, \(BO(2n+1)\langle n \rangle\) denotes the \(n\)-connective cover of \(BO(2n+1)\). The (co)homology of the space \(Q_{0}BO(2n+1)\langle n \rangle_{+}\) is well understood and so our results enable one to compute the homology groups \(H_{k}(BDiff((D^{n+1}\times S^n)^{\natural g}, D^{2n}))\) in a range of degrees when \(k << g\). Our main theorem can be viewed as an analogue of the Madsen-Weiss theorem for the moduli spaces of surfaces and the recent theorem of Galatius and Randal-Williams for the moduli spaces of manifolds of dimension \(2n \geq 6\).