The club said Potter bringing in his own head of recruitment was behind the decision