'I would be prepared to impoverish the country and not be annexed, if that was the option we’re facing,' the former prime minister reportedly said