The Wanted star was alleged to have 'kept his distance' from Kelsey and her two children