Councillor Paul Stuart hit back at calls from the Green Party that he should stand aside