This work develops a new method for rapid and efficient detection of sex hormones in cosmetics. Gold nanoparticles (AuNPs) with high dissipation coefficient was used as fluorescence quenchers in this detection system, rhodamine B(RB) was considered as fluorescence probe and β-cyclodextrin(β-CD) was provided as recognition element. Inclusion of RB into β-CD made fluorescence resonance energy transfer (FRET) occur through the donor (RB) and quencher (AuNPs) nearby. And then, the progesterone-induced release of RB from β-CD cavity, leads to the recovery of the fluorescence of RB. Thus the corresponding fluorescence recovery rate had a good linear relationship with concentration of progesterone in the range of 0 μmol/L-25 μmol/L,and the limit of detection is 0.15 μmol/L. The system was used for the determination of progesterone in toner, and the recoveries are from 95.3% to 101.9%, and the relative standard deviations were less than 4.2%. The fluorescence probe also response to other sex hormones such as estradiol and testosterone. It could be concluded that the system can be applied to the detection of sex hormones in cosmetics.