Father Brown has proved to be incredibly popular and one member of cast knows why