Cabinet ministers say they must not overreact to Trump’s threats of annexation