The latest series of the period drama wraps up on Sunday