A balanced fertilization can increase crop yields partly due to stimulated microbial activities and growths. In this study, we investigated arbuscular mycorrhizal fungi (AMF) in arable soils to determine the optimal practices for an effective fertilization. We used pyrosequencing-based approach to study AMF diversity, as well as their responses to different long-term (>20 years) fertilizations, including OM (organic manure) and mix chemical fertilizers of NP (nitrogen-phosphorus), NK (nitrogen-potassium), and NPK (nitrogen-phosphorus-potassium). Results revealed that 124,998 of 18S rRNA gene fragments were dominated by Glomeromycota with 59,611 sequences, generating 70 operational taxonomic units (OTUs), of which the three largest families were Glomeraceae, Gigasporaceae and Acaulosporaceae. In Control and NK plots, AMF diversity and richness significantly decreased under long-term P fertilizations, such as NP, NPK, and OM. Concomitantly, the AMF community structure shifted. Supported by canonical correspondence analysis, we hereby propose that long-term balanced fertilization, especially P fertilizer with additional N fertilizer, helps the build-up of soil nutrients. Consequently, some AMF community constituents are sacrificed, propelled by the self-regulation of plant-AMF-microbes system, resulting in an agro-ecosystem with a better sustainability. This knowledge would be valuable toward better understandings of AMF community in agro-ecosystem, and long-term ecosystem benefits of the balanced fertilization.