While many traders have brushed off concerns, the potential impact is clear