Luke Grimes said that the fifth season felt ‘different’ without the Dutton family patriarch