In this study, based on the classical Archard adhesion wear theory, a three-dimensional finite element model was established, with the aim of simulating the failure process of self-lubricating spherical plain bearings in the swinging wear condition. The results show that the self-lubricating spherical plain bearings go through two different stages during the wear process, namely, initial wear stage and stable wear stage. Because the large contact points wear out during the initial wear stage, the maximum contact pressure decreases as the test period increases. The relatively larger wear depth region shows elliptical distribution, and the maximum distribution appears in the central contact area. The wear depth reaches 0.974 mm after swinging 25,000 times. PTFE fibers, which possess a good friction performance but poor abrasion resistance, abundantly exist on the friction surfaces of the fabric liner. Consequently, the friction torque during the initial wear stage is slightly smaller than the friction torque during the stable wear stage; however, the wear rate during the initial wear stage is high. The reliability and effectiveness of the finite element model are verified by experiment. The developed finite element model can be used for the analysis of the wear mechanisms of bearings and the prediction of the service life of bearings.