Consumers have been holding back on discretionary spending as prices remain relatively high despite inflationary trends declining, hurting demand for higher-end brands.