The amounts of inbreeding depression upon selfing and of heterosis upon outcrossing determine the strength of selection on the selfing rate in a population when this evolves polygenically by small steps. Genetic models are constructed which allow inbreeding depression to change with the mean selfing rate in a population by incorporating both mutation to recessive and partially dominant lethal and sublethal alleles at many loci and mutation in quantitative characters under stabilizing selection. The models help to explain observations of high inbreeding depression (> 50%) upon selfing in primarily outcrossing populations, as well as considerable heterosis upon outcrossing in primarily selfing populations. Predominant selfing and predominant outcrossing are found to be alternative stable states of the mating system in most plant populations. Which of these stable states a species approaches depends on the history of its population structure and the magnitude of effect of genes influencing the selfing rate.