'There were a number of my colleagues who were using some of the same language and some of the same ideas as Pierre Poilievre,' Gould said