In the present study, a coarse grained AZ31 plate was refined by twin boundaries (TBs) and grain boundaries (GBs), respectively. A comparative study about the different effects of grain refinements by GBs and by TBs on tension-compression yield asymmetry was performed. Our results show that both the refinements by GBs and by TBs increase the tensile and compressive yield strengths, but to a different degree. TBs are more effective to harden twinning, but yield a lower strengthening against prismatic <a> slip, and a much lower tension-compression yield asymmetry is thus obtained. Both the differences in boundary coherence and misorientation between GBs and TBs affect the hardening. The misorientation of TBs provides a lower geometric compatibility factor (a higher hardening) for both prismatic <a> slip and twinning than that of GBs, which in detail is the result of the much higher angle between c-axes of the two sides of TBs (about 86°) than GBs (0–50°). It is found that, for hardening of prismatic <a> slip, boundary coherence plays a more important role than misorientation. With regard to twinning, the different misorientation of TBs from GBs mainly accounts for their different hardening effects.