Talks on forming a new three-party government in Austria have collapsed as the smallest of the prospective coalition partners pulled the plug on negotiations.