Inside Father Brown star's life away from series from famous spouse to reason for leaving show