The popular spot was known for its afternoon teas and dog friendly cafe