Set within a 400-year-old converted mansion, the last three decades has seen the classy pub restaurant move from strength to strength