Brown is one of the early frontrunners and Danny Swanson 'can't see past' him