Plans to turn a vacant office into a Darwen town centre flat have been rejected because the surrounding area is too busy and noisy at night.