Actor Willem Dafoe has said he has never worked with controversial director Woody Allen because he is “not right” for the director’s films.