To investigate the effect of cholesterol on the oxytocin receptor function in myometrial membranes, we developed a new method to alter the membrane cholesterol content. Using a methyl-substituted beta-cyclodextrin, we were able to selectively deplete the myometrial plasma membrane of cholesterol. Vice versa, incubating cholesterol-depleted membranes with a preformed soluble cholesterol-methyl-beta-cyclodextrin complex restored the cholesterol content of the plasma membrane. Binding experiments showed that, with the removal of cholesterol from the membrane, the dissociation constant for [3H]oxytocin is enhanced 87-fold (from Kd = 1.5 nM to Kd = 131 nM), therefore shifting the oxytocin receptor from high to low affinity. Increasing the cholesterol content of the cholesterol-depleted membrane again restored the high-affinity binding (Kd = 1.2 nM). The presence of 0.1 mM GTP gamma S did not significantly change the number of high-affinity binding sites for [3H]oxytocin in native plasma membranes, in membranes depleted of cholesterol, and in plasma membranes with restored cholesterol content. The number of high-affinity binding sites for the oxytocin antagonist [3H]PrOTA was dependent in the same way on the cholesterol content as for [3H]oxytocin. Substitution of the membrane cholesterol with other steroids showed a strong dependence of the oxytocin receptor function on the structure of the cholesterol molecule. The detergent-solubilized oxytocin receptor was not saturable with [3H]oxytocin even at concentrations up to 10(-6) M of radioligand. Addition of the cholesterol-methyl-beta-cyclodextrin complex to the detergent-solubilized oxytocin receptor induced a saturation of the solubilized binding sites (Bmax = 0.98 pmol/mg) for oxytocin (Kd = 16 nM).(ABSTRACT TRUNCATED AT 250 WORDS)