Trump said he did not "talk too much about tariffs" when he spoke with Chinese President Xi Jinping last week