The leader of an Austrian anti-immigration party says talks on forming a coalition government with a conservative party have ended in failure.