Organic-inorganic layered perovskites are two-dimensional quantum wells with layers of lead-halide octahedra stacked between organic ligand barriers. The combination of their dielectric confinement and ionic sublattice results in excitonic excitations with substantial binding energies that are strongly coupled to the surrounding soft, polar lattice. However, the ligand environment in layered perovskites can significantly alter their optical properties due to the complex dynamic disorder of soft perovskite lattice. Here, we observe the dynamic disorder through phonon dephasing lifetimes initiated by ultrafast photoexcitation employing high-resolution resonant impulsive stimulated Raman spectroscopy of a variety of ligand substitutions. We demonstrate that vibrational relaxation in layered perovskite formed from flexible alkyl-amines as organic barriers is fast and relatively independent of the lattice temperature. Relaxation in aromatic amine based layered perovskite is slower, though still fast relative to pure inorganic lead bromide lattices, with a rate that is temperature dependent. Using molecular dynamics simulations, we explain the fast rates of relaxation by quantifying the large anharmonic coupling of the optical modes with the ligand layers and rationalize the temperature independence due to their amorphous packing. This work provides a molecular and time-domain depiction of the relaxation of nascent optical excitations and opens opportunities to understand how they couple to the complex layered perovskite lattice, elucidating design principles for optoelectronic devices.