Planning officials said it could have a 'significantly detrimental' impact on residents