‘I think for Patrick it was really difficult,’ Charlotte Le Bon revealed