SNP Westminster leader Stephen Flynn said patience is running out.