‘There are some people I won’t work for again,’ the presenter said