The actor shares four children with his wife Blake Lively