Permission has been given to build over 100 homes on the historic Wadworth brewery despite local opposition.