A planning inspector has decided that a controversial fence put up to tackle antisocial behaviour in Colchester has to come down.