'There is a danger people will go overboard and target us physically'