Multi-material decomposition (MMD) decomposes CT images into basis material images, and is a promising technique in clinical diagnostic CT to identify material compositions within the human body. MMD could be implemented on measurements obtained from spectral CT protocol, although spectral CT data acquisition is not readily available in most clinical environments. MMD methods using single energy CT (SECT), broadly applied in radiological departments of most hospitals, have been proposed in the literature while challenged by the inferior decomposition accuracy and the limited number of material bases due to the constrained material information in the SECT measurement. In this paper, we propose an image-domain SECT MMD method using material sparsity as an assistance under the condition that each voxel of the CT image contains at most two different elemental materials. L0 norm represents the material sparsity constraint (MSC) and is integrated into the decomposition objective function with a least-square data fidelity term, total variation term, and a sum-to-one constraint of material volume fractions. An accelerated primal-dual (APD) algorithm with line-search scheme is applied to solve the problem. The pixelwise direct inversion method with the two-material assumption (TMA) is applied to estimate the initials. We validate the proposed method on phantom and patient data. Compared with the TMA method, the proposed MSC method increases the volume fraction accuracy (VFA) from 92.0% to 98.5% in the phantom study. In the patient study, the calcification area can be clearly visualized in the virtual non-contrast image generated by the proposed method, and has a similar shape to that in the ground-truth contrast-free CT image. The high decomposition image quality from the proposed method substantially facilitates the SECT-based MMD clinical applications.