Scott's sibling Ryan and Adam Thomas were keen to have their brother home after he quit the villa