The pair were spotted at Jeremy Clarkson's pub, The Farmer's Dog, last week