Background: Multinucleon transfer (MNT) and quasifission (QF) processes are dominant processes in low-energy collisions of two heavy nuclei. They are expected to be useful to produce neutron-rich unstable nuclei. Nuclear dynamics leading to these processes depends sensitively on nuclear properties such as deformation and shell structure. Purpose: We elucidate reaction mechanisms of MNT and QF processes involving heavy deformed nuclei, making detailed comparisons between microscopic TDHF calculations and measurements for \(^{64}\)Ni+\(^{238}\)U reaction. Methods: Three-dimensional Skyrme-TDHF calculations are performed. Particle-number projection method is used to evaluate MNT cross sections from the TDHF wave function after collision. Results: Fragment masses, total kinetic energy (TKE), scattering angle, contact time, and MNT cross sections are investigated for \(^{64}\)Ni+\(^{238}\)U reaction. They show reasonable agreements with measurements. At small impact parameters, collision dynamics depends sensitively on the orientation of deformed \(^{238}\)U. In tip (side) collisions, we find a larger (smaller) TKE and a shorter (longer) contact time. In tip collisions, we find a strong influence of quantum shells around \(^{208}\)Pb. Conclusions: It is confirmed that the TDHF calculations reasonably describe both MNT and QF processes in \(^{64}\)Ni+\(^{238}\)U reaction. Analyses of this system indicates the significance of the nuclear structure effects such as deformation and quantum shells in nuclear reaction dynamics at low energies.