‘It was the hardest thing in the world,’ actor addmitted