Tom originally said he planned to keep the house for a while