A small town on the Victorian-NSW border has seen a group of white supremacists enter their community on the weekend of their annual show.