We consider inverse problems for the coupled Einstein equations and the matter field equations on a 4-dimensional globally hyperbolic Lorentzian manifold \((M,g)\). We give a positive answer to the question: Do the active measurements, done in a neighborhood \(U\subset M\) of a freely falling observed \(\mu=\mu([s_-,s_+])\), determine the conformal structure of the spacetime in the minimal causal diamond-type set \(V_g=J_g^+(\mu(s_-))\cap J_g^-(\mu(s_+))\subset M\) containing \(\mu\)? More precisely, we consider the Einstein equations coupled with the scalar field equations and study the system \(Ein(g)=T\), \(T=T(g,\phi)+F_1\), and \(\square_g\phi-\mathcal V^\prime(\phi)=F_2\), where the sources \(F=(F_1,F_2)\) correspond to perturbations of the physical fields which we control. The sources \(F\) need to be such that the fields \((g,\phi,F)\) are solutions of this system and satisfy the conservation law \(\nabla_jT^{jk}=0\). Let \((\hat g,\hat \phi)\) be the background fields corresponding to the vanishing source \(F\). We prove that the observation of the solutions \((g,\phi)\) in the set \(U\) corresponding to sufficiently small sources \(F\) supported in \(U\) determine \(V_{\hat g}\) as a differentiable manifold and the conformal structure of the metric \(\hat g\) in the domain \(V_{\hat g}\). The methods developed here have potential to be applied to a large class of inverse problems for non-linear hyperbolic equations encountered e.g. in various practical imaging problems.