‘That was the idea back then,’ said the actor