The former president has a long history of controversial remarks about the opposite sex