Insiders have reportedly grown tired of the world's richest man's unpredictability, now seeing him as a liability.