Families and campaigners are deeply unhappy after a summit was scrapped with the Prime Minister