Cationic polymers have been regarded as promising non-viral gene carriers because of their advantages over viral gene vectors, such as low cost, a high level of safety and easy manipulation. However, their poor transfection efficiency in the presence of serum and high toxicity are still limiting issues for clinical applications. In addition, the lack of adequate understanding of the gene delivery mechanism hinders their development to some extent. In this study, new polycations (PAPEs) consisting of a low generation polyamidoamine (PAMAM) core and branched polyethyleneimine (PEI-1.8k) outer layers were synthesized and their transfection activity and mechanism were studied. PAPEs were characterized by FTIR, (1)H NMR and gel permeation chromatography. PAPEs were able to self-assemble with pDNA and form spherical nanoparticles with sizes of 70-204 nm and zeta potentials of 13-33 mV. Importantly, the PAPE-pDNA complexes displayed lower cytotoxicity and higher transfection activity than PEI 25k in various cell lines, specifically in the presence of serum. The transfection mechanism was evaluated by endocytosis inhibition with specific inhibitors, time-dependent transfection, and intracellular trafficking inspection by CLSM. The high levels of transgene expression mediated by PAPEs were attributed to caveolae-mediated cellular uptake, the reduced entry into lysosomes and the entry into the nucleus through mitosis.