Fred Thomas has nevertheless backed the Labour manifesto pledge not to rejoin the EU