The town has been 'let down' according to experts