More than 100 new homes will be built on a controversial site