A couple have lost their battle to retain a 1.5m high boundary fence that was built without planning permission.