This paper presents isolated photon-hadron correlations using pp and p-Pb data collected by the ALICE detector at the LHC. For photons with |\(\eta\)| < 0.67 and 12 < \(p_{\rm{T}}\) < 40 GeV/\(c\), the associated yield of charged particles in the range |\(\eta\)| < 0.80 and 0.5 < \(p_{\rm{T}}\) < 10 GeV/\(c\) is presented. These momenta are much lower than previous measurements at the LHC. No significant difference between pp and p-Pb is observed, with PYTHIA 8.2 describing both data sets within uncertainties. This measurement constrains nuclear effects on the parton fragmentation in p-Pb collisions, and provides a benchmark for future studies of Pb-Pb collisions.