‘Time together is limited,’ the TV personality said