Using Grad's method, we calculate the entropy production and derive a formula for the second-order shear viscosity coefficient in a one-dimensionally expanding particle system, which can also be considered out of chemical equilibrium. For a one-dimensional expansion of gluon matter with Bjorken boost invariance, the shear tensor and the shear viscosity to entropy density ratio \(\eta/s\) are numerically calculated by an iterative and self-consistent prescription within the second-order Israel-Stewart hydrodynamics and by a microscopic parton cascade transport theory. Compared with \(\eta/s\) obtained using the Navier-Stokes approximation, the present result is about 20% larger at a QCD coupling \(\alpha_s \sim 0.3\)(with \(\eta/s\approx 0.18\)) and is a factor of 2-3 larger at a small coupling \(\alpha_s \sim 0.01\). We demonstrate an agreement between the viscous hydrodynamic calculations and the microscopic transport results on \(\eta/s\), except when employing a small \(\alpha_s\). On the other hand, we demonstrate that for such small \(\alpha_s\), the gluon system is far from kinetic and chemical equilibrium, which indicates the break down of second-order hydrodynamics because of the strong noneqilibrium evolution. In addition, for large \(\alpha_s\) (\(0.3-0.6\)), the Israel-Stewart hydrodynamics formally breaks down at large momentum \(p_T\gtrsim 3\) GeV but is still a reasonably good approximation.