The 84-year-old said the show’s filming schedule had become too intense