Martin Frizzell said he was 'unable to do both' as he spoke about needing to spend more time at home