The five remaining living US presidents and their spouses were seated together at the event