'The security advice is that I can't go to the stands,' he says