Due to increasing cost and decreasing labor, prefabricated buildings have developed rapidly. With the prolongation of prefabricated buildings’ supply chain (PBSC) and an increase in risk factors, project delays and even interruptions occur occasionally. The difficulty of supply chain management is increasing. Supply chain resilience (SCR) as a risk management tool has gradually attracted the attention of scholars. This paper uses the grounded theory to identify the influencing factors of prefabricated buildings’ supply chain resilience (PBSCR) based on the dynamic capacity theory. By collecting questionnaires from relevant stakeholders in Shenyang and Hangzhou, a structural equation model (SEM) was used to test the research hypothesis. The capacity effect relationship of the PBSC was constructed. The results show that resilient capability has the highest direct effect on the improvement in PBSCR, and collaborative capability has the highest total and indirect effect on the improvement in PBSCR. The critical paths to improving PBSCR were then identified. Suggestions were made based on the calculated effect relationships. This paper is expected to improve PBSCR, enrich the research on supply chains in the construction field, and help better realize the stable development of prefabricated buildings.