Contentious plans for a fishing and holiday lodge complex near Banbury have been unanimously refused.