One objector said the proposal would 'completely destroy the town centre'