Rushanara Ali, who remains minister for homelessness, said she had relinquished her building safety responsibilities, saying "perception matters"