The star said the decision to exit the show was "really upsetting for me"