Immigration Minister Marc Miller said efforts of Liberals to get the prime minister to step aside is "the most passive-aggressive" display of questioning leadership he has ever seen.