Ductile-to-brittle transition (DBT) temperature and brittle fracture stress, σ F , are important toughness criteria for structural materials. In this paper, low-carbon steels with an ultrafine elongated grain (UFEG) structure (transverse grain size 1.2 μm) and with two ferrite ( α)-pearlite structure with grain sizes 10 µm and 18 µm were prepared. The UFEG steel was fabricated using multipass warm biaxial rolling. The tensile tests with a cylindrical specimen and three-point bending tests with a single-edge-notched specimen were performed at −196 °C. The local stress near the notch was quantitatively calculated via finite element analysis (FEA). The σ F for each sample was quantified based on the experimental results and FEA. The relationship between σ F and d α in the wide range of 1.0 μm to 138 μm was plotted, including data from past literature. Finally, the conditions of grain size and temperature that cause DBT fracture in low-carbon steel were shown via the stress− d −1/2 map. The results quantitatively showed the superiority of α grain size for brittle fracture.