Plans to convert a prominent town centre building into apartments have been thrown out due to inactivity.