Non-Markovian time-evolution of quantum systems is a challenging problem, often mitigated by employing numerical methods or making simplifying assumptions. In this work, we address this problem in waveguide QED by developing a diagrammatic approach, which performs fully analytical non-Markovian time evolution of single-photon states. By revisiting Fermi's two atom problem, we tackle the impeding question of whether rotating-wave approximation violates causality in single-photon waveguide QED. Afterward, we introduce and prove the \emph{no upper half-plane poles (no-UHP) theorem}, which connects the poles of scattering parameters to the causality principle. Our diagrammatic approach is the first method to perform exact and analytical non-Markovian time evolution of multi-emitter systems in waveguide QED.