The actor broke his silence on becoming the ninth celebrity to leave the BBC One dance contest over the weekend