Concerns over the territories were raised after the Government announced its decision to relinquish sovereignty over the Chagos Islands.