The star has opened up about unwanted male attention