A Married At First Sight groom has hit back at the backlash he has faced for leaving his 'marriage'