Genome duplication and repeat multiplication contribute to genome evolution in plants. Our previous work identified a recent allotetraploidization event and five high-copy LTR retrotransposon (LTR-RT) families PgDel, PgTat, PgAthila, PgTork, and PgOryco in Panax ginseng. Here, using whole-genome sequences, we quantified major repeats in five Panax species and investigated their role in genome evolution. The diploids P. japonicus, P. vietnamensis, and P. notoginseng and the tetraploids P. ginseng and P. quinquefolius were analyzed alongside their relative Aralia elata. These species possess 0.8–4.9 Gb haploid genomes. The PgDel, PgTat, PgAthila, and PgTork LTR-RT superfamilies accounted for 39–52% of the Panax species genomes and 17% of the A. elata genome. PgDel included six subfamily members, each with a distinct genome distribution. In particular, the PgDel1 subfamily occupied 23–35% of the Panax genomes and accounted for much of their genome size variation. PgDel1 occupied 22.6% (0.8 Gb of 3.6 Gb) and 34.5% (1.7 Gb of 4.9 Gb) of the P. ginseng and P. quinquefolius genomes, respectively. Our findings indicate that the P. quinquefolius genome may have expanded due to rapid PgDel1 amplification over the last million years as a result of environmental adaptation following migration from Asia to North America.