Actor said he would often ‘be wearing a skirt or stuff like that’ during his teen years