The Vivienne's ex-husband said his "heart sank"