The Prime Minister could be in trouble as early as next year if things do not improve for his party