Before the family moved into Pataudi Palace, they lived in a historic residence built during the reign of Nawab Muhammad Akbar Ali, where the Pataudi family resided for several generations until 1941.