The TV presenter has revealed a heartbreaking reason for her refusal