"When we see fear in the market, we see people go to safe havens," one gold dealer says.