Only 31% of those in highest-income group think economy is moving in right direction