'Karyakartas felt we should have fought independently'