Attacked for being 'far right' as their economic prospects slip away, voters are turning conservative