The development of novel double-hybrid density functionals offers new levels of accuracy and is leading to fresh insights into the fundamental properties of matter. Hartree–Fock exact exchange and correlated wave function methods, such as second-order Møller–Plesset (MP2) and direct random phase approximation (dRPA), are usually required to build such functionals. Their high computational cost is a concern, and their application to large and periodic systems is, therefore, limited. In this work, low-scaling methods for Hartree–Fock exchange (HFX), SOS-MP2, and direct RPA energy gradients are developed and implemented in the CP2K software package. The use of the resolution-of-the-identity approximation with a short range metric and atom-centered basis functions leads to sparsity, allowing for sparse tensor contractions to take place. These operations are efficiently performed with the newly developed Distributed Block-sparse Tensors (DBT) and Distributed Block-sparse Matrices (DBM) libraries, which scale to hundreds of graphics processing unit (GPU) nodes. The resulting methods, resolution-of-the-identity (RI)-HFX, SOS-MP2, and dRPA, were benchmarked on large supercomputers. They exhibit favorable sub-cubic scaling with system size, good strong scaling performance, and GPU acceleration up to a factor of 3. These developments will allow for double-hybrid level calculations of large and periodic condensed phase systems to take place on a more regular basis.