The comment offended some fans of the property show