The former X Factor winner's role on the show caused a massive setback for him and fiancee Sophie Austin