The Prime Minister accused the Conservatives of using the historic decision to quit the bloc as an "open borders experiment".