The Bargain Hunt star has been telling his side of the story