Fears have been raised over the future of a charity that helps vulnerable young people if 300 flats are built nearby.