The spectral signature of an exoplanet can be separated from the spectrum of its host star using high-resolution spectroscopy. During such observations, the radial component of the planet's orbital velocity changes, resulting in a significant Doppler shift which allows its spectral features to be extracted. Aims: In this work, we aim to detect TiO in the optical transmission spectrum of HD 209458b. Gaseous TiO has been suggested as the cause of the thermal inversion layer invoked to explain the dayside spectrum of this planet. Method: We used archival data from the 8.2m Subaru Telescope taken with the High Dispersion Spectrograph of a transit of HD209458b in 2002. We created model transmission spectra which include absorption by TiO, and cross-correlated them with the residual spectral data after removal of the dominating stellar absorption features. We subsequently co-added the correlation signal in time, taking into account the change in Doppler shift due to the orbit of the planet. Results: We detect no significant cross-correlation signal due to TiO, though artificial injection of our template spectra into the data indicates a sensitivity down to a volume mixing ratio of ~10E-10. However, cross-correlating the template spectra with a HARPS spectrum of Barnard's star yields only a weak wavelength-dependent correlation, even though Barnard's star is an M4V dwarf which exhibits clear TiO absorption. We infer that the TiO line list poorly match the real positions of TiO lines at spectral resolutions of ~100,000. Similar line lists are also used in the PHOENIX and Kurucz stellar atmosphere suites and we show that their synthetic M-dwarf spectra also correlate poorly with the HARPS spectra of Barnard's star and five other M-dwarfs. We conclude that the lack of an accurate TiO line list is currently critically hampering this high-resolution retrieval technique.