As consumer spending falls, the group behind struggling brands such as Michael Kors looks out of style