"Anyone who made such a gesture in ordinary working life would no longer be in their position," he said