Tories reject Bank of England governor Andrew Bailey's suggestion that Brexit had harmed trade as growth flatlines