‘The weekends are always hard,’ actor said