Too often, horror films confuse surprise with fear. A new class of directors is exploring darker places