The actor said ‘it was both too little and not enough to be anything more than it was’