‘It was not a very harmonious place,’ Spider-Man actor recalled