The pastor is accused of groping individuals who worked at his cleaning business