Didcot and Wantage MP Olly Glover has stepped down from his town council role.