This paper is concerned with the numerical solution of the unified first order hyperbolic formulation of continuum mechanics proposed by Peshkov & Romenski (HPR model), which is based on the theory of nonlinear hyperelasticity of Godunov & Romenski . Notably, the governing PDE system is symmetric hyperbolic and fully consistent with the first and the second principle of thermodynamics. The nonlinear system of governing equations of the HPR model is large and includes stiff source terms as well as non-conservative products. In this paper we solve this model for the first time on moving unstructured meshes in multiple space dimensions by employing high order accurate one-step ADER-WENO finite volume schemes in the context of cell-centered direct Arbitrary-Lagrangian-Eulerian (ALE) algorithms. The numerical method is based on a WENO polynomial reconstruction operator on moving unstructured meshes, a fully-discrete one-step ADER scheme that is able to deal with stiff sources, a nodal solver with relaxation to determine the mesh motion, and a path-conservative technique of Castro & Par\`es for the treatment of non-conservative products. We present numerical results obtained by solving the HPR model with ADER-WENO-ALE schemes in the stiff relaxation limit, showing that fluids (Euler or Navier-Stokes limit), as well as purely elastic or elasto-plastic solids can be simulated in the framework of nonlinear hyperelasticity with the same system of governing PDE. The obtained results are in good agreement when compared to exact or numerical reference solutions available in the literature.