He said he will come down from the clock tower 'on his own terms'