A swastika has been scrawled onto the city centre building.