Four former prime ministers have said they are opposed to the legislation although none will have a vote on Friday.